mod_fcgid and graceful restarts

I see plenty of this in my logs when the server needs reloaded to pick up fresh Perl:

(43)Identifier removed: mod_fcgid: can't lock process table in pid 3218

tl;dr: this appears to be harmless in practice.

The leading portion corresponds to EIDRM (see errno(3)) which comes back out of pthread_mutex_lock and cheerfully gets logged as the failure code of proctable_lock_internal.  The proctable is in turn locked during request handling.

My best guess for the order of events is that the Apache parent receives a graceful restart, unloads and reloads mod_fcgid, which destroys the mutex as a side effect.  After old-generation children tie up their requests, they try to notify their parent that they're available again, only to discover that the mutex is gone.  The child then exits, but it doesn't hurt any clients because they've already been served at this point.

This problem is not fixable in Apache 2.2 because there aren't any hooks for graceful-restart.  It just unloads DSOs without warning, and their first clue anything happened is that they start receiving config events.  By then, the mutex and process table are gone, so the newly-loaded master can't communicate with old-generation children.  Someone did make an attempt to fix this for 2.4 (along with modifying mod_cgid to test their infrastructure) but AFAICT nobody has made this available in mod_fcgid for 2.4 yet.

Fun, Work, Puzzles, and Programming

Some programming tasks are just more fun than others. The same thing extends to languages—why are Perl and Ruby so much more fun to work with than Python?

I suspect that the answer lies in the scope of the solution space, in a sweet spot between “too straightforward” and “too complex.”

PHP's debug_backtrace: a compact guide

Every time I need to use this function, I can't remember how it works.

1. The array includes all call sites leading up to the current stack frame, but not actually the current one.  (Everything in the current frame is still in scope to you, so you can use __FILE__ and __LINE__ or your current variables directly.)
2. The array is indexed with 0=innermost to N=outermost frame.
3. Each array index gives you information related to the call site of the next frame inward / earlier in the array.  That is, $bt[0] gives you the immediate caller of your current point of execution.  $bt[0]['function'] refers to the function or method invocation that called you, e.g. if the main code executes foo(1), then inside function foo, $bt[0]['function'] is foo.  The file and line point to the file/line containing the call.
4. When a 'class' key is present, it is the class of the line of code actually executing the call, i.e. what __CLASS__ is at the 'file' and 'line'.
5. When an 'object' key is present, it has the actual object being used for dispatch; i.e. get_class($bt[$i]['object']) may return either the same value as 'class', or any descendant of that class.
6. The 'type' key, when present, is either -> or :: for dynamic or static calls, respectively.  The latter means that the 'object' key won't be set.
7. There is no way in my PHP (5.3.3-14_el6.3 from CentOS updates) to view the invoked class of a static call, e.g. if SubThing::foo is called but Thing::foo is executed because SubThing didn't override foo.  Per the rules above, 'class' will still report Thing.

I needed to know this (this time) because I wanted to write a rough equivalent to Perl's carp in PHP:

<?php
function carp () {
  $msg = func_get_args();
  if (empty($msg)) $msg = array('warned');
  $bt = debug_backtrace();

  // find nearest site not in our caller's file
  $first_file = $bt[0]['file'];
  $end = count($bt);
  for ($i = 1; $i < $end; ++$i) {
    if ($bt[$i]['file'] != $first_file)
      break;
  }

  if ($i == $end) {
    // not found; try the caller's caller.
    // otherwise we're stuck with our caller.
    $i = ($end > 1 ? 1 : 0);
  }

  error_log(implode(' ', $msg) .
    " at {$bt[$i]['file']}:{$bt[$i]['line']}");
}

Obviously this is a bare-bones approach, and could be adapted to pick different (or report more) stack frames, etc.  But, it Works For Me.™

A Minimal, Working Perl FastCGI Example

Updated 2013-01-30

Please see the next version, which allows for newly-written FastCGI scripts that do not have a CGI equivalent.

Original post follows.

I couldn't really find one of these on the Internet, so I'm going to document what I have working so far. In the following examples, assume an application named "site" rooted at /home/site/web, with static files served out of public/, app-specific modules contained under lib/, and other perl module paths (vendor, local::lib, etc.) compiled into Perl itself.  Site::* references an app-specific module, naturally.

Apache configuration snippet, using mod_fcgid as the FastCGI process manager:

DocumentRoot /home/site/web/public
FcgidInitialEnv PERL5LIB /home/site/web/lib
FcgidWrapper /home/site/web/handler.fcgi
RewriteEngine on
RewriteCond /home/site/web/lib/Site/Entry/$1.pm -f
RewriteRule ^/(.*)\.pl$ - [QSA,L,H=fcgid-script]

Note that the left of RewriteRule is matched, and the $1 reference assigned, prior to evaluating RewriteCond's.  The rule means, "if the requested filename exists where the FastCGI wrapper will look for it, force it to be handled by FastCGI."  This interacts perfectly with DirectoryIndex: requesting / with a DirectoryIndex index.pl in effect invokes the handler.fcgi for index.pl as long as Site::Entry::index exists.

Now, my handler.fcgi named in the Apache configuration for the FcgidWrapper looks like this:

#!/home/site/bin/perl
use warnings;
use strict;
use CGI::Fast;
use FindBin;
use Site::Preloader ();
while (my $q = CGI::Fast->new) {
    my ($base, $mod) = ($ENV{SCRIPT_FILENAME});
    $base = substr($base, length $ENV{DOCUMENT_ROOT});
    $base =~ s/\.pl$//;
    $base =~ s#^/+##;
    $base =~ s#/+#::#g;
    $base ||= 'index';
    $mod = "Site::Entry::$base";
    my $r = eval {
        eval "require $mod;"
            and $mod->invoke($q);
    };
    warn "$mod => $@" unless defined $r;
}

This means that I can have models named like Site::Login and view-controllers for them under Site::Entry::login (handling /login.pl, naturally).  I still have to rewrite the site from vanilla CGI scripts into module form, but the RewriteRule work above means that the FastCGI versions can be picked up URL-by-URL.  It doesn't require a full-site conversion to a framework to gain any benefits.

There's one additional feature this wrapper has: by using SCRIPT_FILENAME and removing DOCUMENT_ROOT off the front, I can rewrite a pretty URL (prior to the last RewriteRule shown above) to one ending .pl and still have the wrapper work.  SCRIPT_NAME keeps the name of the script as it was in the original request, and does not receive the rewritten value.  (Only SCRIPT_FILENAME does, on my exact setup.)  So I did it this way, rather than re-applying all my rewrites in Perl.

mod_fcgid and PHP_FCGI_CHILDREN

While attempting to validate a configuration for some server benchmarking, I happened upon some curious pstree output which read in part:

httpd─┬─httpd───10*[php-cgi───8*[php-cgi]]
      └─23*[httpd]

I had expected at most 8 php-cgi children, having configured PHP_FCGI_CHILDREN=8 in the wrapper script, but mod_fcgid was clearly starting multiple instances of the wrapper, so I was getting an absurd number of children.

I discovered FcgidMaxProcesses and further benchmarks ensued:

MaxProc	CHILDREN	req/sec	MiB/sec
4	5	1279	4.0
4	2	1221	3.8
10	2	2488	7.8

The first two rows, in spite of the second only providing 40% of the available PHP children, are within error of each other in actual performance.  The third line produces a total of 20 children again, but now 10 of them instead of 4 are visible in mod_fcgid for handling requests.

Clearly, there are too many cooks in the kitchen: both mod_fcgid and php-cgi (which, I should mention, was built with --disable-fpm) are trying to act as master, so PHP ends up wasting $PHP_FCGI_CHILDREN - 1 processes because mod_fcgid won't pass concurrent requests for them to handle.

What happens if you don't configure PHP_FCGI_CHILDREN at all?  php-cgi handles the request directly, as if it were a child to begin with, and pstree looks more like this:

httpd─┬─httpd───12*[php-cgi]
      └─25*[httpd]

That looks rational.  How does it perform?

MaxProc	CHILDREN	req/sec	MiB/sec
10	unset	2580	8.2
20	unset	2617	8.3

Concurrency was held the same at 16 throughout all tests today, so it looks like we've run into the limits of the 100 mbit network.  (Serious benchmarking will happen once my gigabit gear arrives.)  Still, the fact that performance keeps up with 10×2 children portends well; it means there's no penalty for unsetting PHP_FCGI_CHILDREN and mod_fcgid isn't broken without it.

The Router Trinity

There are technically two separate companies housed in the suite, with separate subnets set up for themselves, but they wanted to be able to exchange data at some point in the past.  The link between the two networks is accomplished by no less than three consumer-grade "cable/dsl routers."

Let's start with a picture so you have something to boggle at:

Hairy Escaping Problems (Keep the Pieces 2)

I was just settling in to hack out a Smarty-like template system (or at least an interpreter for it) in a non-PHP language, when my brain went all meta on me. ‘How can I never, ever have to deal with careful manual control over output encoding, ever again?’