Why we have a memcached-dynamo proxy

Previously, I mentioned that we have a proxy that speaks the memcached protocol to clients and stores data in dynamodb, “for reasons.”  Today, it’s time to talk about the architecture, and those reasons.

Minimal, Working Perl FastCGI Example, version 2

This is an update to a previous post.  File layout remains the same: "site" is a placeholder for the actual site name, and /home/site/web is the actual repository of the project.  Static files then appear under public, and Perl modules specific to the site in lib/Site (i.e. visible in Perl as Site::Modname when lib is put in @INC).  I am still using mod_fcgid as the FastCGI process manager.

The major improvement: This version handles FCGI-only scripts which have no corresponding CGI URL.  I discovered that limitation of the previous version when I tried to write some new code, where Apache or mod_fcgid realized that the CGI version didn't exist, and returned a 404 instead of passing it through the wrapper.  As a consequence of solving that problem, FcgidWrapper is no longer necessary, which gives the dispatch.fcgi code a much cleaner environment to work in.

Everything I liked about the previous version is preserved here: I can create Site/Entry/login.pm to transparently handle /login.pl as FastCGI, without requiring every other URL to be available in FastCGI form.  It also stacks properly with earlier RewriteRules that turn pretty URLs into ones ending in ".pl".

Apache configuration:

# Values set via SetEnv will be passed in the request;
# to affect Perl startup, it must be FcgidInitialEnv
FcgidInitialEnv PERL5LIB /home/site/web/lib
RewriteCond /home/site/web/lib/Site/Entry/$1.pm -f
RewriteRule ^/+(.+)\.pl$ /home/site/web/dispatch.fcgi [L,QSA,H=fcgid-script,E=SITE_HANDLER:$1]
<directory /home/site/web/fcgi>
    Options ExecCGI FollowSymLinks
    # ...
</directory>

Again, the regular expression of the RewriteRule is matched before RewriteCond is evaluated, so the backreference $1 is available to test whether the file exists.  This time, I also use the environment flag of the RewriteRule to pass the handler to the dispatch.fcgi script.  Since I paid to capture it and strip the leading slashes and extension already, I may as well use it.

That means the new dispatch.fcgi script doesn't have to do as much cleanup to produce the module name:

#!/home/site/bin/perl
use warnings;
use strict;
use FindBin qw($Bin);
use Site::Response;
use Site::Preloader ();
while (my $q = CGI::Fast->new) {
    my ($base, $mod) = ($ENV{SITE_HANDLER});
    $base =~ s#/+#::#g;
    $base =~ s#[^\w:]##g;
    $base ||= 'index';
    $mod = "Site::Entry::$base";
    my $r = Site::Response->new($base, "$Bin/templates");
    eval {
        eval "require $mod;"
            and $mod->invoke($q, $r);
    } or warn "$mod => $@";
    $r->send($q);
}

I remembered to include the $r->send call this time.  I pass the CGI query object so the response can call $q->header.  That's not strictly necessary—FCGI children process one request at a time and copy $q to the default CGI object, meaning header should work fine alone, but I didn't know that yet.

I also remove non-{word characters or colons} from the inbound request for security, since my site uses URLs like /path/somereport.pl.  You may need to carefully adjust that for your site.

Site::Response is initialized as a generic error so that if the module dies, the response written to the client is a complete generic error.  Otherwise, the template is selected and data set, so the send call ships the completed page instead.

The only thing left that I'd like to do is make this configuration more portable between web servers instead of dependent on Apache's mod_rewrite and mod_fcgid, but since Apache isn't killing us at work, it probably won't happen very soon.

A Minimal, Working Perl FastCGI Example

Updated 2013-01-30

Please see the next version, which allows for newly-written FastCGI scripts that do not have a CGI equivalent.

Original post follows.

I couldn't really find one of these on the Internet, so I'm going to document what I have working so far. In the following examples, assume an application named "site" rooted at /home/site/web, with static files served out of public/, app-specific modules contained under lib/, and other perl module paths (vendor, local::lib, etc.) compiled into Perl itself.  Site::* references an app-specific module, naturally.

Apache configuration snippet, using mod_fcgid as the FastCGI process manager:

DocumentRoot /home/site/web/public
FcgidInitialEnv PERL5LIB /home/site/web/lib
FcgidWrapper /home/site/web/handler.fcgi
RewriteEngine on
RewriteCond /home/site/web/lib/Site/Entry/$1.pm -f
RewriteRule ^/(.*)\.pl$ - [QSA,L,H=fcgid-script]

Note that the left of RewriteRule is matched, and the $1 reference assigned, prior to evaluating RewriteCond's.  The rule means, "if the requested filename exists where the FastCGI wrapper will look for it, force it to be handled by FastCGI."  This interacts perfectly with DirectoryIndex: requesting / with a DirectoryIndex index.pl in effect invokes the handler.fcgi for index.pl as long as Site::Entry::index exists.

Now, my handler.fcgi named in the Apache configuration for the FcgidWrapper looks like this:

#!/home/site/bin/perl
use warnings;
use strict;
use CGI::Fast;
use FindBin;
use Site::Preloader ();
while (my $q = CGI::Fast->new) {
    my ($base, $mod) = ($ENV{SCRIPT_FILENAME});
    $base = substr($base, length $ENV{DOCUMENT_ROOT});
    $base =~ s/\.pl$//;
    $base =~ s#^/+##;
    $base =~ s#/+#::#g;
    $base ||= 'index';
    $mod = "Site::Entry::$base";
    my $r = eval {
        eval "require $mod;"
            and $mod->invoke($q);
    };
    warn "$mod => $@" unless defined $r;
}

This means that I can have models named like Site::Login and view-controllers for them under Site::Entry::login (handling /login.pl, naturally).  I still have to rewrite the site from vanilla CGI scripts into module form, but the RewriteRule work above means that the FastCGI versions can be picked up URL-by-URL.  It doesn't require a full-site conversion to a framework to gain any benefits.

There's one additional feature this wrapper has: by using SCRIPT_FILENAME and removing DOCUMENT_ROOT off the front, I can rewrite a pretty URL (prior to the last RewriteRule shown above) to one ending .pl and still have the wrapper work.  SCRIPT_NAME keeps the name of the script as it was in the original request, and does not receive the rewritten value.  (Only SCRIPT_FILENAME does, on my exact setup.)  So I did it this way, rather than re-applying all my rewrites in Perl.

Perl non-CGI: The Missing Overview

After living in mod_php and Perl CGI for far too long, it was time to look at reworking our application to fit something else.  Although we had mod_perl installed and doing little more than wasting resources, I didn’t want to bind the app tightly to mod_perl in the way it was already tightly bound to CGI.  That meant surveying the landscape and trying to understand what modern Perl web development actually looks like.

But first, some history!

Autoflush vs. Scope (CGI::Session)

CGI::Session writes its session data to disk when DESTROY is called, possibly during global destruction, but the order of global destruction is non-deterministic.  It will generally work when CGI::Session is writing to files, since it doesn't depend on anything else to do that, but using other storage like memcached or a database, the connection to storage may have been cleaned up before CGI::Session can use it.  Then your session data is mysteriously lost, because it was never saved to begin with.

Another possible interaction between object lifetime occurs when there are multiple CGI::Session objects: unless both have identical data, whichever one gets destroyed last, wins.  At one point, I added an END {} block to a file which had my $session declared.  All of a sudden, that END block kept $session alive until global destruction and the other CGI::Session instance, into which I recorded that a user was in fact logged in, now flushed first.  Because the logged-in state was then overwritten by the session visible from the END block (even though the block itself never used it), nobody could log in!

Yet another problem happened when I pulled $session out of that code and stored it in a package.  The END block had finished its purpose and been deleted, so moving $session to a package once again extended its life to global destruction: a package variable stays around forever, because the package itself is a global resource.  However, since the login path had flush() calls carefully placed on it, what broke this time was logout.  The delete() call couldn't take effect because the storage was gone by the time the session was cleaned up.

CGI::Session and your HTTP headers

For CGI::Session to work, you must send the Set-Cookie header (via $session->header() or otherwise) when the session's is_new method returns true.  I discovered this by tripping over an awesome new failure mode today:

1. Restart memcached (or otherwise create new session storage).
2. Nothing stays saved in the session.  Can't log in.

When CGI::Session receives a session ID that doesn't exist in session storage, it changes the session ID to prevent session fixation attacks.  Which means that if you only send the header in the absence of a browser cookie, data is written to the new ID, but the browser will re-submit the old ID next request.

(It turns out my real problem was the stupidly simple error of 'trying to write to the wrong memcached server,' but the above did happen to my test page while I was trying to figure out why memcached wasn't saving anything.)