A Minimal, Working Perl FastCGI Example

Updated 2013-01-30

Please see the next version, which allows for newly-written FastCGI scripts that do not have a CGI equivalent.

Original post follows.

I couldn't really find one of these on the Internet, so I'm going to document what I have working so far. In the following examples, assume an application named "site" rooted at /home/site/web, with static files served out of public/, app-specific modules contained under lib/, and other perl module paths (vendor, local::lib, etc.) compiled into Perl itself.  Site::* references an app-specific module, naturally.

Apache configuration snippet, using mod_fcgid as the FastCGI process manager:

DocumentRoot /home/site/web/public
FcgidInitialEnv PERL5LIB /home/site/web/lib
FcgidWrapper /home/site/web/handler.fcgi
RewriteEngine on
RewriteCond /home/site/web/lib/Site/Entry/$1.pm -f
RewriteRule ^/(.*)\.pl$ - [QSA,L,H=fcgid-script]

Note that the left of RewriteRule is matched, and the $1 reference assigned, prior to evaluating RewriteCond's.  The rule means, "if the requested filename exists where the FastCGI wrapper will look for it, force it to be handled by FastCGI."  This interacts perfectly with DirectoryIndex: requesting / with a DirectoryIndex index.pl in effect invokes the handler.fcgi for index.pl as long as Site::Entry::index exists.

Now, my handler.fcgi named in the Apache configuration for the FcgidWrapper looks like this:

#!/home/site/bin/perl
use warnings;
use strict;
use CGI::Fast;
use FindBin;
use Site::Preloader ();
while (my $q = CGI::Fast->new) {
    my ($base, $mod) = ($ENV{SCRIPT_FILENAME});
    $base = substr($base, length $ENV{DOCUMENT_ROOT});
    $base =~ s/\.pl$//;
    $base =~ s#^/+##;
    $base =~ s#/+#::#g;
    $base ||= 'index';
    $mod = "Site::Entry::$base";
    my $r = eval {
        eval "require $mod;"
            and $mod->invoke($q);
    };
    warn "$mod => $@" unless defined $r;
}

This means that I can have models named like Site::Login and view-controllers for them under Site::Entry::login (handling /login.pl, naturally).  I still have to rewrite the site from vanilla CGI scripts into module form, but the RewriteRule work above means that the FastCGI versions can be picked up URL-by-URL.  It doesn't require a full-site conversion to a framework to gain any benefits.

There's one additional feature this wrapper has: by using SCRIPT_FILENAME and removing DOCUMENT_ROOT off the front, I can rewrite a pretty URL (prior to the last RewriteRule shown above) to one ending .pl and still have the wrapper work.  SCRIPT_NAME keeps the name of the script as it was in the original request, and does not receive the rewritten value.  (Only SCRIPT_FILENAME does, on my exact setup.)  So I did it this way, rather than re-applying all my rewrites in Perl.

mod_fcgid and PHP_FCGI_CHILDREN

While attempting to validate a configuration for some server benchmarking, I happened upon some curious pstree output which read in part:

httpd─┬─httpd───10*[php-cgi───8*[php-cgi]]
      └─23*[httpd]

I had expected at most 8 php-cgi children, having configured PHP_FCGI_CHILDREN=8 in the wrapper script, but mod_fcgid was clearly starting multiple instances of the wrapper, so I was getting an absurd number of children.

I discovered FcgidMaxProcesses and further benchmarks ensued:

MaxProc	CHILDREN	req/sec	MiB/sec
4	5	1279	4.0
4	2	1221	3.8
10	2	2488	7.8

The first two rows, in spite of the second only providing 40% of the available PHP children, are within error of each other in actual performance.  The third line produces a total of 20 children again, but now 10 of them instead of 4 are visible in mod_fcgid for handling requests.

Clearly, there are too many cooks in the kitchen: both mod_fcgid and php-cgi (which, I should mention, was built with --disable-fpm) are trying to act as master, so PHP ends up wasting $PHP_FCGI_CHILDREN - 1 processes because mod_fcgid won't pass concurrent requests for them to handle.

What happens if you don't configure PHP_FCGI_CHILDREN at all?  php-cgi handles the request directly, as if it were a child to begin with, and pstree looks more like this:

httpd─┬─httpd───12*[php-cgi]
      └─25*[httpd]

That looks rational.  How does it perform?

MaxProc	CHILDREN	req/sec	MiB/sec
10	unset	2580	8.2
20	unset	2617	8.3

Concurrency was held the same at 16 throughout all tests today, so it looks like we've run into the limits of the 100 mbit network.  (Serious benchmarking will happen once my gigabit gear arrives.)  Still, the fact that performance keeps up with 10×2 children portends well; it means there's no penalty for unsetting PHP_FCGI_CHILDREN and mod_fcgid isn't broken without it.

The Router Trinity

There are technically two separate companies housed in the suite, with separate subnets set up for themselves, but they wanted to be able to exchange data at some point in the past.  The link between the two networks is accomplished by no less than three consumer-grade "cable/dsl routers."

Let's start with a picture so you have something to boggle at:

Hairy Escaping Problems (Keep the Pieces 2)

I was just settling in to hack out a Smarty-like template system (or at least an interpreter for it) in a non-PHP language, when my brain went all meta on me. ‘How can I never, ever have to deal with careful manual control over output encoding, ever again?’

DynamoDB in the Trenches

Amazon's DynamoDB, as they're happy to tell you, is an SSD-backed NoSQL storage service with provisioned throughput.  Data consists of three basic types (number, string, and binary) in either scalar or set forms.  (A set contains any number of unique values of its parent type, in no particular order.)  All lookups are done by hash keys, optionally with a range as sub-key; the former effectively defines a root type, and the latter is something like a 1:Many relation.  The hash key is the foreign key to the parent object, and the range key defines an ordering of the collection.

But you knew all that; there are two additional points I want to add to the documentation.

1. Update is Upsert

DynamoDB's update operation actually behaves as upsert—if you update a nonexistent item, the attributes you updated will be created as the only attributes on the item.  If this would result in an invalid item as a whole, then you want to use the expected-value mechanism to make sure the item is really there before the update applies.

2. No Attribute Indexing or Querying

NoSQL is at once the greatest draw to non-relational storage, and also its biggest drawback.  On DynamoDB, there's no query language.  You can get items by key cheaply, or scan the whole table expensively, and there's nothing in between.  There's no API for indexing any attributes, so there's no API for querying by index, either.  You can't even query on the range of a range key independently of the hash (e.g. "find all the posts today, regardless of topic" on a table keyed by a topic-hash and date-range.)

If you need lookup by attribute-equals more than you need consistent query performance, then you can use SimpleDB.  RDS could be a decent option, especially if you want ordered lookups (as in DELETE FROM sessions WHERE expires < NOW();—when the primary key is "id".)

A not-so-good option would be to add another DynamoDB table keyed by attribute-values and containing sets of your main table's hash keys—but you can't update multiple DynamoDB tables transactionally, so this is more prone to corruption than other methods.

And if you want to pull together two sources of asynchronous events, and act on the result once both events have occurred, then the Simple Workflow service might work.  (I realized this about 98% of the way through a different solution, so I stuck with that.  I might have been able to store the half-complete event into the workflow state instead, no DynamoDB needed, but since I didn't walk that path, I can't vouch for it.)

Perl non-CGI: The Missing Overview

After living in mod_php and Perl CGI for far too long, it was time to look at reworking our application to fit something else.  Although we had mod_perl installed and doing little more than wasting resources, I didn’t want to bind the app tightly to mod_perl in the way it was already tightly bound to CGI.  That meant surveying the landscape and trying to understand what modern Perl web development actually looks like.

But first, some history!

Because I Can: SMTPS + MIME::Lite monkeypatch

I had an app that calls to MIME::Lite->send() to send an email, which until recently was using an SMTP server slated for decommissioning Real Soon Now.  It was my job to convert it to Amazon SES, and I figured it would be easier to tell MIME::Lite to use SES's SMTP interface instead of importing the web side's full Perl library tree just for one module out of it.

Ha ha!  SES requires SSL, and neither MIME::Lite nor Net::SMTP have any idea about that.  They were both written before the days of dependency injection, so I had to go to some length to achieve it.  And now, I golfed it a bit for you:

package MyApp::Monkey::SMTPS;
use warnings;
use strict;
use parent 'IO::Socket::SSL';

# Substitute us for the vanilla INET socket
require Net::SMTP;
@Net::SMTP::ISA = map {
  s/IO::Socket::INET/MyApp::Monkey::SMTPS/; $_
}  @Net::SMTP::ISA;

our %INET_OPTS = qw(
  PeerPort smtps(465)
  SSL_version TLSv1
); # and more options, probably

# Override new() to provide SSL etc. parameters
sub new {
  my ($cls, %opts) = @_;
  $opts{$_} = $INET_OPTS{$_} foreach keys %INET_OPTS;
  $cls->SUPER::new(%opts);
}

PeerPort overrides the default of smtp(25) built in to Net::SMTP; I needed a port where the whole connection is wrapped in SSL instead of using STARTTLS, and 465 is the one suitable choice of the three that SES-SMTP supports.

The main caveat about this is that it breaks Net::SMTP for anyone else in-process who wants to send mail to a server that lacks a functional port 465.  But as you may have guessed, that's not a problem for my script, today.