Everything Fails, FCGI::ProcManager::Dynamic Edition

I have been reading a lot of rachelbythebay, which has led me to thinking about the reliability of my own company’s architecture.

It’s top of my mind right now, because an inscrutable race condition caused a half-hour outage of our primary site.  It was a new, never-before-seen condition that slipped right past all our existing defenses.  Using Site as a placeholder for our local namespace, it looked like this:

use Try::Tiny qw(try catch);
try {
  require Site::Response;
  require Site::Utils;
  ...;
} catch {
  # EX_PRELOAD => exit status 1
  exit_manager(EX_PRELOAD, $_);
};
...;
$res = Site::Response->new();

Well, this time—this one time—on both running web servers… it started throwing an error that Method "new" wasn't found in package Site::Response (did you forget to load Site::Response?).  Huh?  Of course I loaded it; I would’ve exited if that had failed.

In response, I added a lot more try/catch, exit_manager() has been improved, and there is a separate site-monitoring service that will issue systemctl restart on the site, if it starts rapidly cycling through workers.

Replacing `mod_fcgid` with `mod_proxy_fcgi`

This is using somewhat outdated technology, and specific to my environment, but I’ve found nothing of value on the internet other than the actual documentation included with FCGI::ProcManager, so I wanted to share what I did to Perl and FastCGI this week.

Recap: mod_fcgid loses its connection table during a “graceful reload” with Apache 2.4 (we have the version included with Ubuntu 17.04, so it’s Apache 2.4.25.) The current connections get broken, and we’ve reached a size where that can interrupt around 20 connections during one reload event.  Which we invoke when deploying new code.

Therefore, I wanted something more reliable, so I built something to use mod_proxy_fcgi to talk to a listening daemon.

Nonlocality

Three years ago, I was porting a Perl CGI-based system to FastCGI, one URL at a time, using mod_rewrite to do the dispatching.  (If the handler.pm exists where the FCGI dispatcher will look for it, invoke this request via FastCGI.)  A consequence of this is that the core library code needs to run on both paradigms, since both front-ends use it.  That runs straight into problems when the CGI access checking functions blithely print and call exit when the access is denied.

Instead of updating 60+ scripts to read e.g. $session = get_session() or Site::CGI->go_login; I decided to get hackier.

SNS Deployment update

As an update to this old post from 2013…

We have moved to using php-fpm, so naturally suEXEC has been replaced by php-fpm’s configuration. That allows for running multiple pools, each running PHP scripts under their own user.

We have the “main” pool still using the same unprivileged user as Apache, and then there’s a “privileged” pool that uses the privileged user. Only the traffic on the port receiving SNS notifications is directed to this privileged pool. The main pool still has the same permission it would if it were running under mod_php, to ease the transition.

The transition was relatively painless, but only because I’d already converted per-dir .htaccess files into rules in the server config. It was part micro-optimization, part “I will probably want nginx someday.” Although Apache is still serving our needs admirably.

Apache, FastCGI, and the Authorization header

I couldn’t find much about why Apache (up to and including 2.4.x) doesn’t pass the HTTP Authorization header to FastCGI by searching the Internet, so I fished through their source code.

There’s a function that sets up the default variables to pass to CGI programs, called ap_add_common_vars. This function expects that other users on the server can see these variables with ps -e, so to hide usernames and passwords from appearing in there, it specifically avoids passing Authorization through to CGI.

FastCGI uses the same ap_add_common_vars function, and mod_proxy_fcgi doesn’t take any special care to put the Authorization back in for requests that are sent over a socket. Thus, by default, no FastCGI requests will receive usernames and passwords.

The standard workaround is to use mod_rewrite to set an environment flag if the request has authorization. Because nothing unsets the Authorization—ap_add_common_vars simply skips setting a default—that explicitly-set environment variable will be included in the FastCGI request.

RewriteCond %{HTTP:Authorization} .
RewriteRule / - [E:HTTP_AUTHORIZATION=%{HTTP:Authorization}]

That’s fine if you’re already using mod_rewrite. Can it be done without that module? I haven’t benchmarked the difference, but mod_setenvif can do it via the rule:

SetEnvIf Authorization (.+) HTTP_AUTHORIZATION=$1

tl;dr: Authorization isn’t passed to FastCGI because of the default protection from exposing passwords with slow CGI. It can safely be passed to a FastCGI server (requests are sent over a socket, not through world-readable process attributes) but it must be explicitly passed.

Updated 28 Jul 2015: I finally tested the SetEnvIf rule. I also mentioned what Apache line I'm working with.

PHP-FPM error logging (where FastCGI errors go)

Let's start with the catchy summary graphic:

There are a lot of streams in the worker because historically, CGI directly connected a worker's standard output and error streams (file descriptors 1 and 2 on Unix) to the web server.  FastCGI kept the notions, but established new, separate FastCGI output and error streams for FastCGI-aware applications to use.  It also specified that the standard output/error streams should be discarded.

So by default, that's exactly what the FPM master does: it arranges for workers' standard streams to be discarded.  That's actually fine with PHP because the error logs go through the SAPI layer, and that layer writes them to the FastCGI error stream when FastCGI is active.

By default, that stream goes through to the web server, where Apache logs it to whatever error log is in effect.  (Typically, I've worked on sites that have per-virtualhost logs.  Especially for error logs, it's a lot easier to find the relevant messages when it's known that Host X is down.)

PHP-FPM, however, also has the option of setting per-pool php.ini values in the pool configuration.  If a value is provided there (e.g. /etc/php-fpm-5.5.d/www.conf) for the error_log setting, then that pool will log to the specified destination instead of the Web server's error log.  In other words, the php_value[error_log] set in the pool configuration overrides the default behavior of logging to the FastCGI error stream.

These options turn out to be entirely independent of the master FastCGI configuration setting named catch_workers_output.  This option, in the php-fpm configuration (e.g. /etc/php-fpm-5.5.conf) controls whether the standard streams are discarded as specified, or if they will appear as lines in the master process's error log.  Unless the workers are dying unexpectedly, this probably doesn't contain anything interesting.

More FastCGI: Apache 2.4, PHP-FPM, PSGI, and hot deployment

Driven by mod_fcgid's failings at graceful restart and preforking, I've been looking hard for alternatives.

tl;dr

External FCGI is best done via mod_fastcgi on Apache prior to 2.4, or mod_proxy_fcgi on 2.4.  mod_proxy in general is super awesome on 2.4.

php-fpm ships with an init script and reloads gracefully via SIGUSR2, so that's all you really need there.

For Perl, gracefully restarting an FCGI app is difficult, so the better approach for PSGI apps is to run them in an HTTP server with Server::Starter support (e.g. Starman or others).

Notes on FastCGI and webservers

This post is a distillation of what I've learned over the past couple of months.  There's both new information here, and links to everything else on the FastCGI topic that I've written so far.

Contents

1. mod_fcgid
2. php-fpm (nginx)
3. other things I tried to use
4. concluding remarks

Minimal, Working Perl FastCGI Example, version 2

This is an update to a previous post.  File layout remains the same: "site" is a placeholder for the actual site name, and /home/site/web is the actual repository of the project.  Static files then appear under public, and Perl modules specific to the site in lib/Site (i.e. visible in Perl as Site::Modname when lib is put in @INC).  I am still using mod_fcgid as the FastCGI process manager.

The major improvement: This version handles FCGI-only scripts which have no corresponding CGI URL.  I discovered that limitation of the previous version when I tried to write some new code, where Apache or mod_fcgid realized that the CGI version didn't exist, and returned a 404 instead of passing it through the wrapper.  As a consequence of solving that problem, FcgidWrapper is no longer necessary, which gives the dispatch.fcgi code a much cleaner environment to work in.

Everything I liked about the previous version is preserved here: I can create Site/Entry/login.pm to transparently handle /login.pl as FastCGI, without requiring every other URL to be available in FastCGI form.  It also stacks properly with earlier RewriteRules that turn pretty URLs into ones ending in ".pl".

Apache configuration:

# Values set via SetEnv will be passed in the request;
# to affect Perl startup, it must be FcgidInitialEnv
FcgidInitialEnv PERL5LIB /home/site/web/lib
RewriteCond /home/site/web/lib/Site/Entry/$1.pm -f
RewriteRule ^/+(.+)\.pl$ /home/site/web/dispatch.fcgi [L,QSA,H=fcgid-script,E=SITE_HANDLER:$1]
<directory /home/site/web/fcgi>
    Options ExecCGI FollowSymLinks
    # ...
</directory>

Again, the regular expression of the RewriteRule is matched before RewriteCond is evaluated, so the backreference $1 is available to test whether the file exists.  This time, I also use the environment flag of the RewriteRule to pass the handler to the dispatch.fcgi script.  Since I paid to capture it and strip the leading slashes and extension already, I may as well use it.

That means the new dispatch.fcgi script doesn't have to do as much cleanup to produce the module name:

#!/home/site/bin/perl
use warnings;
use strict;
use FindBin qw($Bin);
use Site::Response;
use Site::Preloader ();
while (my $q = CGI::Fast->new) {
    my ($base, $mod) = ($ENV{SITE_HANDLER});
    $base =~ s#/+#::#g;
    $base =~ s#[^\w:]##g;
    $base ||= 'index';
    $mod = "Site::Entry::$base";
    my $r = Site::Response->new($base, "$Bin/templates");
    eval {
        eval "require $mod;"
            and $mod->invoke($q, $r);
    } or warn "$mod => $@";
    $r->send($q);
}

I remembered to include the $r->send call this time.  I pass the CGI query object so the response can call $q->header.  That's not strictly necessary—FCGI children process one request at a time and copy $q to the default CGI object, meaning header should work fine alone, but I didn't know that yet.

I also remove non-{word characters or colons} from the inbound request for security, since my site uses URLs like /path/somereport.pl.  You may need to carefully adjust that for your site.

Site::Response is initialized as a generic error so that if the module dies, the response written to the client is a complete generic error.  Otherwise, the template is selected and data set, so the send call ships the completed page instead.

The only thing left that I'd like to do is make this configuration more portable between web servers instead of dependent on Apache's mod_rewrite and mod_fcgid, but since Apache isn't killing us at work, it probably won't happen very soon.

mod_fcgid and graceful restarts

I see plenty of this in my logs when the server needs reloaded to pick up fresh Perl:

(43)Identifier removed: mod_fcgid: can't lock process table in pid 3218

tl;dr: this appears to be harmless in practice.

The leading portion corresponds to EIDRM (see errno(3)) which comes back out of pthread_mutex_lock and cheerfully gets logged as the failure code of proctable_lock_internal.  The proctable is in turn locked during request handling.

My best guess for the order of events is that the Apache parent receives a graceful restart, unloads and reloads mod_fcgid, which destroys the mutex as a side effect.  After old-generation children tie up their requests, they try to notify their parent that they're available again, only to discover that the mutex is gone.  The child then exits, but it doesn't hurt any clients because they've already been served at this point.

This problem is not fixable in Apache 2.2 because there aren't any hooks for graceful-restart.  It just unloads DSOs without warning, and their first clue anything happened is that they start receiving config events.  By then, the mutex and process table are gone, so the newly-loaded master can't communicate with old-generation children.  Someone did make an attempt to fix this for 2.4 (along with modifying mod_cgid to test their infrastructure) but AFAICT nobody has made this available in mod_fcgid for 2.4 yet.

A Minimal, Working Perl FastCGI Example

Updated 2013-01-30

Please see the next version, which allows for newly-written FastCGI scripts that do not have a CGI equivalent.

Original post follows.

I couldn't really find one of these on the Internet, so I'm going to document what I have working so far. In the following examples, assume an application named "site" rooted at /home/site/web, with static files served out of public/, app-specific modules contained under lib/, and other perl module paths (vendor, local::lib, etc.) compiled into Perl itself.  Site::* references an app-specific module, naturally.

Apache configuration snippet, using mod_fcgid as the FastCGI process manager:

DocumentRoot /home/site/web/public
FcgidInitialEnv PERL5LIB /home/site/web/lib
FcgidWrapper /home/site/web/handler.fcgi
RewriteEngine on
RewriteCond /home/site/web/lib/Site/Entry/$1.pm -f
RewriteRule ^/(.*)\.pl$ - [QSA,L,H=fcgid-script]

Note that the left of RewriteRule is matched, and the $1 reference assigned, prior to evaluating RewriteCond's.  The rule means, "if the requested filename exists where the FastCGI wrapper will look for it, force it to be handled by FastCGI."  This interacts perfectly with DirectoryIndex: requesting / with a DirectoryIndex index.pl in effect invokes the handler.fcgi for index.pl as long as Site::Entry::index exists.

Now, my handler.fcgi named in the Apache configuration for the FcgidWrapper looks like this:

#!/home/site/bin/perl
use warnings;
use strict;
use CGI::Fast;
use FindBin;
use Site::Preloader ();
while (my $q = CGI::Fast->new) {
    my ($base, $mod) = ($ENV{SCRIPT_FILENAME});
    $base = substr($base, length $ENV{DOCUMENT_ROOT});
    $base =~ s/\.pl$//;
    $base =~ s#^/+##;
    $base =~ s#/+#::#g;
    $base ||= 'index';
    $mod = "Site::Entry::$base";
    my $r = eval {
        eval "require $mod;"
            and $mod->invoke($q);
    };
    warn "$mod => $@" unless defined $r;
}

This means that I can have models named like Site::Login and view-controllers for them under Site::Entry::login (handling /login.pl, naturally).  I still have to rewrite the site from vanilla CGI scripts into module form, but the RewriteRule work above means that the FastCGI versions can be picked up URL-by-URL.  It doesn't require a full-site conversion to a framework to gain any benefits.

There's one additional feature this wrapper has: by using SCRIPT_FILENAME and removing DOCUMENT_ROOT off the front, I can rewrite a pretty URL (prior to the last RewriteRule shown above) to one ending .pl and still have the wrapper work.  SCRIPT_NAME keeps the name of the script as it was in the original request, and does not receive the rewritten value.  (Only SCRIPT_FILENAME does, on my exact setup.)  So I did it this way, rather than re-applying all my rewrites in Perl.

mod_fcgid and PHP_FCGI_CHILDREN

While attempting to validate a configuration for some server benchmarking, I happened upon some curious pstree output which read in part:

httpd─┬─httpd───10*[php-cgi───8*[php-cgi]]
      └─23*[httpd]

I had expected at most 8 php-cgi children, having configured PHP_FCGI_CHILDREN=8 in the wrapper script, but mod_fcgid was clearly starting multiple instances of the wrapper, so I was getting an absurd number of children.

I discovered FcgidMaxProcesses and further benchmarks ensued:

MaxProc	CHILDREN	req/sec	MiB/sec
4	5	1279	4.0
4	2	1221	3.8
10	2	2488	7.8

The first two rows, in spite of the second only providing 40% of the available PHP children, are within error of each other in actual performance.  The third line produces a total of 20 children again, but now 10 of them instead of 4 are visible in mod_fcgid for handling requests.

Clearly, there are too many cooks in the kitchen: both mod_fcgid and php-cgi (which, I should mention, was built with --disable-fpm) are trying to act as master, so PHP ends up wasting $PHP_FCGI_CHILDREN - 1 processes because mod_fcgid won't pass concurrent requests for them to handle.

What happens if you don't configure PHP_FCGI_CHILDREN at all?  php-cgi handles the request directly, as if it were a child to begin with, and pstree looks more like this:

httpd─┬─httpd───12*[php-cgi]
      └─25*[httpd]

That looks rational.  How does it perform?

MaxProc	CHILDREN	req/sec	MiB/sec
10	unset	2580	8.2
20	unset	2617	8.3

Concurrency was held the same at 16 throughout all tests today, so it looks like we've run into the limits of the 100 mbit network.  (Serious benchmarking will happen once my gigabit gear arrives.)  Still, the fact that performance keeps up with 10×2 children portends well; it means there's no penalty for unsetting PHP_FCGI_CHILDREN and mod_fcgid isn't broken without it.