Everything Fails, FCGI::ProcManager::Dynamic Edition

I have been reading a lot of rachelbythebay, which has led me to thinking about the reliability of my own company’s architecture.

It’s top of my mind right now, because an inscrutable race condition caused a half-hour outage of our primary site.  It was a new, never-before-seen condition that slipped right past all our existing defenses.  Using Site as a placeholder for our local namespace, it looked like this:

use Try::Tiny qw(try catch);
try {
  require Site::Response;
  require Site::Utils;
  ...;
} catch {
  # EX_PRELOAD => exit status 1
  exit_manager(EX_PRELOAD, $_);
};
...;
$res = Site::Response->new();

Well, this time—this one time—on both running web servers… it started throwing an error that Method "new" wasn't found in package Site::Response (did you forget to load Site::Response?).  Huh?  Of course I loaded it; I would’ve exited if that had failed.

In response, I added a lot more try/catch, exit_manager() has been improved, and there is a separate site-monitoring service that will issue systemctl restart on the site, if it starts rapidly cycling through workers.

FastCGI in Perl, but PHP Style [2012]

Editor's note: I found this in my drafts from 2012. By now, everything that can be reasonably converted to FastCGI has been, and a Perl-PHP bridge has been built to allow new code to be written for the site in PHP instead. However, the conclusion still seems relevant to designers working on frameworks, so without further ado, the original post follows...

The first conversions of CGI scripts to FastCGI have been launched into production. I have both the main login flow and six of the most popular pages converted, and nothing has run away with the CPU or memory in the first 50 hours. It’s been completely worry-free on the memory front, and I owe it to the PHP philosophy.

In PHP, users generally don’t have the option of persistence. Unless something has been carefully allocated in persistent storage in the PHP kernel (the C level code), everything gets cleaned up at the end of the request. Database connections are the famous example.

Perl is obviously different, since data can be trivially kept by using package level variables to stash data, but my handler-modules (e.g. Site::Entry::login) don’t use them. Such handler-modules define one well-known function, which returns an object instance that carries all the necessary state for the dispatch and optional post-dispatch phases. When this object is destroyed in the FastCGI request loop, so too are all its dependencies.

Furthermore, dispatching returns its response, WSGI style, so that if dispatch dies, the FastCGI loop can return a generic error for the browser. Dispatch isn’t allowed to write anything to the output stream directly, including headers, which guarantees a blank slate for the main loop’s error page. (I once wrote a one-pass renderer, then had to grapple with questions like “How do I know whether HTML has been sent?”, “How do I close half-sent HTML?”, and “What if it’s not HTML?” in the error handler.)

On "Perl 7"

Previous history:

1. SawyerX: Announcing Perl 7.
2. Grinnz: Perl 7: A Risk-Benefit Analysis.
3. RJBS: On Perl 7 and the Perl Steering Comittee.

My comments at this point:

Hey #perl core. Is there anywhere I can upload a cpanfile, so you can know what's important in production? Like Debian's popularity-contest, but specialized. (We install most everything via Ubuntu repositories, so you don't get downloads/test feedback.)

The perl7 thing is probably accelerating our plans to phase Perl out entirely. I'm not sure what we'll do if Ubuntu 22.04 ships a "Perl 7" that can't run Perl 5 code.

To be honest, ~20% of hits to our client-facing site are served by CGI.pm because of some failures on my part (should have used CGI::Emulate::PSGI?) and on management (50% time savings is great, but let's work on features... forever.)

Anyway, it's going to be real hard to sell "work on Perl7 support" to a management team who have been ramping up the priority on "rewrite the application."

(Lightly edited from my twitter thread.)

Now, I realize I'm nobody, and nobody has to listen to me, but I think the popularity-ranking thing is kind of important, and maybe even a "State of Perl" survey to find out what's even broken. All I know from my vantage point is that Perl looks fairly dead already, despite how much I loved it. The Perl 7 crisis has so far been more anxiety-inducing than inspiring.

Updated 2022-03-19: Perl 7 is basically dead? Business as usual? I still can't help illuminate the DarkPAN for them, and management is inching closer to sunsetting our own Perl code. RIP.

Version Locking

We have a few lovely workarounds, arrived at through a small measure of blood, sweat, tears, and research, embedded in our cpanfile these days.  Here are my two favorites:

requires 'Email::MIME', '<= 1.928';
    # 1.929+ screws up Content-Type sometimes
requires 'DateTime::Format::Strptime', '<= 1.57';
    # DynamoDB v0.1.16 (current) buggy with rewritten Strptime

Nobody wants to come back and check these, to see if later releases of the packages work again.  (It’s possible I’m the only one who remembers these hacks are here.)

Email::MIME was especially nasty because it only failed sometimes, and now, how do I prove that the fixes (which may have occurred in 1.931 or 1.933) actually solve the problem?  I can’t prove the negative in test, and I can’t trust the package in production.

As for the other fun bug, “DynamoDB v0.1.16” refers to the package whose full name is Net::Amazon::DynamoDB and released on November 6th, 2012.  I think this one was detectable at install/test time, but it was still No Fun.  A lot of work was expended in finding out its dependency changed, and I’m not excited to redo it all to find out if 1.67 (or 1.63) fixed the issues.

Especially since use of Perl was deprecated company-wide, and we want to get it all ported to a new language.

Editor’s Notes: this was another old draft.

Since it was written, we accidentally introduced an updated version of Email::MIME into production, and it still failed.  We fixed the bug that allowed the update to occur; clearly, upstream’s process is broken.  I don’t think we could be the only people hit, across multiple years and multiple versions of the entire software stack.

I’m not entirely sure what happened with Net::Amazon::DynamoDB—but we may have been able to use a newer version with it.

Why we have a memcached-dynamo proxy

Previously, I mentioned that we have a proxy that speaks the memcached protocol to clients and stores data in dynamodb, “for reasons.”  Today, it’s time to talk about the architecture, and those reasons.

Effectively Using Future in Perl

I’ve been working a lot with the Future library and IO::Async in Perl recently.  There was a bug in our memcache/dynamo proxy, so I ended up doing a lot of investigation about Futures in order to simulate the bug and verify the fix.  So, I want to talk about how Futures work and why it was so difficult for me.

Replacing `mod_fcgid` with `mod_proxy_fcgi`

This is using somewhat outdated technology, and specific to my environment, but I’ve found nothing of value on the internet other than the actual documentation included with FCGI::ProcManager, so I wanted to share what I did to Perl and FastCGI this week.

Recap: mod_fcgid loses its connection table during a “graceful reload” with Apache 2.4 (we have the version included with Ubuntu 17.04, so it’s Apache 2.4.25.) The current connections get broken, and we’ve reached a size where that can interrupt around 20 connections during one reload event.  Which we invoke when deploying new code.

Therefore, I wanted something more reliable, so I built something to use mod_proxy_fcgi to talk to a listening daemon.

Nonlocality

Three years ago, I was porting a Perl CGI-based system to FastCGI, one URL at a time, using mod_rewrite to do the dispatching.  (If the handler.pm exists where the FCGI dispatcher will look for it, invoke this request via FastCGI.)  A consequence of this is that the core library code needs to run on both paradigms, since both front-ends use it.  That runs straight into problems when the CGI access checking functions blithely print and call exit when the access is denied.

Instead of updating 60+ scripts to read e.g. $session = get_session() or Site::CGI->go_login; I decided to get hackier.

Perl Retrospective

Perl was the first language I really did serious work in.  To this day, even though it’s dying, I still like it.  It’s different, which is at once both its major virtue, and its Achilles heel in the modern programming landscape.

Let’s talk about what makes Perl Perlish, startng with the good parts.

(Note that I use a lot of 5.10 features like say and maybe the // operator.  For the most part, assume a use v5.10; line before any code. I may not adhere to use strict; use warnings; in this post, but my production code does.)

The Common Denominator

On my back-burner at work, following a company-wide deprecation of Perl, is porting our Perl-based API services to PHP.

This may be painful, because by the time I wrote the API code, I was fluently using Perl-isms in Perl code.  It’s possible this won’t be a straightforward port because most of that is missing in PHP.

In comparison, back when I first started working here, I ported a small sub-site from Perl to PHP.  It was written in a style where I mostly needed small, mechanical changes like taking out my.  The only “big” change was creating an equivalent for HTML::Template, but that was pretty small itself.

Maybe you can write Java in any language… but then it’s 10 times easier to port to another language.

Perl is Dying

I like Perl. It's been a part of my life for over 14 years, since I had 5.6.0 on Red Hat 7.0. (Funny how Red Hat Enterprise is now 7.0, and the desktop project hasn't borne the name for over twenty releases.)

But, I'm getting the distinct impression that the language is losing its community. It seems to be getting too small to keep up with the nice things.

Case in point: AWS. Trawl metacpan.org, and there's no coherent set of packages for using Amazon services. There are some Net::Amazon modules, some Amazon modules, some AWS modules, and SimpleDB::Client doesn't even have Amazon in the name. UPDATE (2020-08-08): This update has been a long time coming, but I discovered and switched to Paws sometime in 2018. On the other hand, Paws::S3 (S3!!!) issues a warning on load that it's not finished. Yea, unto this very day.

Then there are the duplicate packages like Net::Amazon::DynamoDB and Amazon::DynamoDB, which are worlds apart. The former supports practically the original DynamoDB API, using signature version 2, and accepting numbers, strings, and sets of those types as data types. Not even binary types. Amazon::DynamoDB uses the version 4 signature and an updated data model, along with an async client built on IO::Async. Yes, seriously, in 2014 someone didn't use AnyEvent.

This wouldn't be so much of a problem if I didn't sail straight into RT#93107 while using it. The most insidious thing is that fds only get closed sometimes, so it mostly seems to work. Until it doesn't. But there's a patch... That's been open without comment for many months.

This is not unlike another patch, for AnyEvent::Connection, open even longer. I can understand if the maintainer thinks it's unimportant because it's "only a deprecation" or because an updated common::sense avoids the issue on affected Perl versions. But to not even comment? The package is apparently dead.

I recently ran into a little trouble finding any OAuth 2.0 server modules. I didn’t see anything general, nor anything for Dancer to be an OAuth server. We have plenty of clients and client-integrations to providers, though, and it didn’t take my boss long to find a PHP library that he wanted to use.

But enough about CPAN. Let’s talk Perl core.

Perl has been adding numerous ‘experimental’ features over the years. Since 5.16, the current status of all of these has been gathered in the perlexperiment documentation. Though 5.20.1 has now been released, there are items on that list introduced in 5.10.0.

An experimental feature that is neither accepted nor rejected is dead weight. Authors are discouraged from using them, so they’re not benefitting the majority of users, yet they require ongoing maintenance and care by the perl5-porters. An experimental feature that stays experimental for far, far longer than the deprecation cycle is the worst possible outcome.

Just imagine being excited about Perl and having conversations about, “Can it do «cool thing»?” or “Have they fixed «wart» yet?” and having to respond “Well, it’s experimental, so technically yes, but you shouldn’t use it.”

Eventually, that conversation is going to come down to “Why don’t you use «Ruby|JavaScript|PHP» instead? They have more cool stuff, and only one object system.”

Even if Perl made a major push to accept experimental features in “the next release,” there’s anywhere from months to years before that release is widely deployed as a baseline. Unless one wants to build their own Perl every week for their fresh cloud images, which is a whole other can of worms.

Meanwhile, the more unique stuff about Perl—regex syntax, contexts, sigils and their associated namespaces—aren’t exactly banner features anymore. Other languages have been absorbing regex syntax, and the rest of it tends to create bugs, even by authors who understand it all. That’s especially problematic when I change languages frequently, because the Perl-unique stuff takes longer to reload in my head than the rest of the language.

Overall, I’m thankful for what Perl has been for me. However, it seems like it has lost all its momentum, and it doesn’t have anything particularly unique to offer anymore. Certain parts of it like @_ and the choose-your-own-adventure object system feel particularly archaic, and I have to interact with them all the time.

Overall, it seems like basic stuff (IO::Poll working with SSL, please?) and easy fixes aren’t being taken care of, while new code isn’t showing up or isn’t high quality. If CPAN can’t keep up with the web, and the web is eating everything, then Perl will be eaten.

You are in a twisty little maze of event backends, all different.
Most of which can use most of the other backends as its own provider.

It pains me to write this. I don’t want to sound like a “BSD is dying” troll. I know I’m throwing a lot of criticism in the face of people who have put much more into Perl (and making Perl awesome) than I have. Miyagawa is perhaps singlehandedly responsible for why our API server started its life in Perl. But he can’t do everything, so the present signs are that no new Perl will be added to our codebase… because doing so would burden our future.

Perl Futures

I’ve been doing a project with IO::Async and Future, because Amazon::DynamoDB depends on them.

One of the weirdest bits of the API that I’ve run into is that the underlying framework only weakly references Future objects, so if the client doesn’t keep its own reference, then the callbacks will never be invoked. I may not care about “the future object” itself, but I do care about the callbacks I set on it.

I ended up writing a function:

sub hold_future {
    my $f = shift;
    $f->on_ready(sub { undef $f });
    return ();
}

The sub being set does nothing but close over $f in order to create a circular reference. That keeps the Future alive until the callbacks are run. To keep the future from persisting forever, though, it destroys that reference once the future resolves.

One may safely release the future while the callbacks are running, since the resolution machinery is holding onto a strong reference ($self) at that point. Therefore, the future won’t be destroyed until all the callbacks finish.

Now, the rest of my code has clearer purpose:

sub md_delete {
    my ($cb, $id) = @_;
    hold_future(
        dynamo_delete($id)
        ->on_done(sub { $cb->(1) })
        ->on_fail(sub { $cb->(0); log_event("delete $id: $_[0]") })
    );
}

Where dynamo_delete invokes a lot of retry/exponential backoff machinery, and md_delete is a command handler for Memcached::Server.

Having worked with jQuery Deferreds, I find it odd that Future doesn’t provide a separate Future::Promise object. It just hopes that the receiver of the future doesn’t try to call creator-related functions like done.

I also find it odd that the future doesn’t just stick around itself, automatically persisting until the callback phase unless actively canceled by the receiver. Having to manually manage them feels surprisingly alien, even though I’m used to managing file handles and lexical scopes.

Common Sense vs. Fatal Deprecations

In Perl, use common::sense 3.73 or newer.  Older versions, including Ubuntu Trusty’s libcommon-sense-perl package at 3.72, enable fatal deprecations and will unexpectedly kill your code if you transitively depend on a module using common::sense that throws a deprecation warning.

Say, one whose maintainer received a patch and is apparently letting it rot in their issue tracker forever.

IO::Scalar

Roaming the dusty halls of some legacy system tools, I found some code that takes IO-Stringy as a dependency.  Only to use it for IO::Scalar... only to support sending to non-filehandles... which is an unused feature of the system.  There are no callers of the carefully-built "change output stream" method.

As the icing on the cake, Perl has had native support for doing this since 5.8, by passing a scalar reference instead of a file name: open($fh, '>>', \$string_out);

This legacy code has only ever run on Perl 5.10.0 and newer.  The dependency has always been unnecessary, unused, and overbuilt.

Moose Alternatives

I once wrote about Mouse when I had discovered it.  Amongst all this recent research, I have also discovered Moo.

Moo strives to be even lighter and faster to load than Mouse, which was already Moose without antlers.

I swear, next year I'm going to find an even lighter module called "Mc".

More FastCGI: Apache 2.4, PHP-FPM, PSGI, and hot deployment

Driven by mod_fcgid's failings at graceful restart and preforking, I've been looking hard for alternatives.

tl;dr

External FCGI is best done via mod_fastcgi on Apache prior to 2.4, or mod_proxy_fcgi on 2.4.  mod_proxy in general is super awesome on 2.4.

php-fpm ships with an init script and reloads gracefully via SIGUSR2, so that's all you really need there.

For Perl, gracefully restarting an FCGI app is difficult, so the better approach for PSGI apps is to run them in an HTTP server with Server::Starter support (e.g. Starman or others).

Minimal, Working Perl FastCGI Example, version 2

This is an update to a previous post.  File layout remains the same: "site" is a placeholder for the actual site name, and /home/site/web is the actual repository of the project.  Static files then appear under public, and Perl modules specific to the site in lib/Site (i.e. visible in Perl as Site::Modname when lib is put in @INC).  I am still using mod_fcgid as the FastCGI process manager.

The major improvement: This version handles FCGI-only scripts which have no corresponding CGI URL.  I discovered that limitation of the previous version when I tried to write some new code, where Apache or mod_fcgid realized that the CGI version didn't exist, and returned a 404 instead of passing it through the wrapper.  As a consequence of solving that problem, FcgidWrapper is no longer necessary, which gives the dispatch.fcgi code a much cleaner environment to work in.

Everything I liked about the previous version is preserved here: I can create Site/Entry/login.pm to transparently handle /login.pl as FastCGI, without requiring every other URL to be available in FastCGI form.  It also stacks properly with earlier RewriteRules that turn pretty URLs into ones ending in ".pl".

Apache configuration:

# Values set via SetEnv will be passed in the request;
# to affect Perl startup, it must be FcgidInitialEnv
FcgidInitialEnv PERL5LIB /home/site/web/lib
RewriteCond /home/site/web/lib/Site/Entry/$1.pm -f
RewriteRule ^/+(.+)\.pl$ /home/site/web/dispatch.fcgi [L,QSA,H=fcgid-script,E=SITE_HANDLER:$1]
<directory /home/site/web/fcgi>
    Options ExecCGI FollowSymLinks
    # ...
</directory>

Again, the regular expression of the RewriteRule is matched before RewriteCond is evaluated, so the backreference $1 is available to test whether the file exists.  This time, I also use the environment flag of the RewriteRule to pass the handler to the dispatch.fcgi script.  Since I paid to capture it and strip the leading slashes and extension already, I may as well use it.

That means the new dispatch.fcgi script doesn't have to do as much cleanup to produce the module name:

#!/home/site/bin/perl
use warnings;
use strict;
use FindBin qw($Bin);
use Site::Response;
use Site::Preloader ();
while (my $q = CGI::Fast->new) {
    my ($base, $mod) = ($ENV{SITE_HANDLER});
    $base =~ s#/+#::#g;
    $base =~ s#[^\w:]##g;
    $base ||= 'index';
    $mod = "Site::Entry::$base";
    my $r = Site::Response->new($base, "$Bin/templates");
    eval {
        eval "require $mod;"
            and $mod->invoke($q, $r);
    } or warn "$mod => $@";
    $r->send($q);
}

I remembered to include the $r->send call this time.  I pass the CGI query object so the response can call $q->header.  That's not strictly necessary—FCGI children process one request at a time and copy $q to the default CGI object, meaning header should work fine alone, but I didn't know that yet.

I also remove non-{word characters or colons} from the inbound request for security, since my site uses URLs like /path/somereport.pl.  You may need to carefully adjust that for your site.

Site::Response is initialized as a generic error so that if the module dies, the response written to the client is a complete generic error.  Otherwise, the template is selected and data set, so the send call ships the completed page instead.

The only thing left that I'd like to do is make this configuration more portable between web servers instead of dependent on Apache's mod_rewrite and mod_fcgid, but since Apache isn't killing us at work, it probably won't happen very soon.

A Minimal, Working Perl FastCGI Example

Updated 2013-01-30

Please see the next version, which allows for newly-written FastCGI scripts that do not have a CGI equivalent.

Original post follows.

I couldn't really find one of these on the Internet, so I'm going to document what I have working so far. In the following examples, assume an application named "site" rooted at /home/site/web, with static files served out of public/, app-specific modules contained under lib/, and other perl module paths (vendor, local::lib, etc.) compiled into Perl itself.  Site::* references an app-specific module, naturally.

Apache configuration snippet, using mod_fcgid as the FastCGI process manager:

DocumentRoot /home/site/web/public
FcgidInitialEnv PERL5LIB /home/site/web/lib
FcgidWrapper /home/site/web/handler.fcgi
RewriteEngine on
RewriteCond /home/site/web/lib/Site/Entry/$1.pm -f
RewriteRule ^/(.*)\.pl$ - [QSA,L,H=fcgid-script]

Note that the left of RewriteRule is matched, and the $1 reference assigned, prior to evaluating RewriteCond's.  The rule means, "if the requested filename exists where the FastCGI wrapper will look for it, force it to be handled by FastCGI."  This interacts perfectly with DirectoryIndex: requesting / with a DirectoryIndex index.pl in effect invokes the handler.fcgi for index.pl as long as Site::Entry::index exists.

Now, my handler.fcgi named in the Apache configuration for the FcgidWrapper looks like this:

#!/home/site/bin/perl
use warnings;
use strict;
use CGI::Fast;
use FindBin;
use Site::Preloader ();
while (my $q = CGI::Fast->new) {
    my ($base, $mod) = ($ENV{SCRIPT_FILENAME});
    $base = substr($base, length $ENV{DOCUMENT_ROOT});
    $base =~ s/\.pl$//;
    $base =~ s#^/+##;
    $base =~ s#/+#::#g;
    $base ||= 'index';
    $mod = "Site::Entry::$base";
    my $r = eval {
        eval "require $mod;"
            and $mod->invoke($q);
    };
    warn "$mod => $@" unless defined $r;
}

This means that I can have models named like Site::Login and view-controllers for them under Site::Entry::login (handling /login.pl, naturally).  I still have to rewrite the site from vanilla CGI scripts into module form, but the RewriteRule work above means that the FastCGI versions can be picked up URL-by-URL.  It doesn't require a full-site conversion to a framework to gain any benefits.

There's one additional feature this wrapper has: by using SCRIPT_FILENAME and removing DOCUMENT_ROOT off the front, I can rewrite a pretty URL (prior to the last RewriteRule shown above) to one ending .pl and still have the wrapper work.  SCRIPT_NAME keeps the name of the script as it was in the original request, and does not receive the rewritten value.  (Only SCRIPT_FILENAME does, on my exact setup.)  So I did it this way, rather than re-applying all my rewrites in Perl.

Perl non-CGI: The Missing Overview

After living in mod_php and Perl CGI for far too long, it was time to look at reworking our application to fit something else.  Although we had mod_perl installed and doing little more than wasting resources, I didn’t want to bind the app tightly to mod_perl in the way it was already tightly bound to CGI.  That meant surveying the landscape and trying to understand what modern Perl web development actually looks like.

But first, some history!

Because I Can: SMTPS + MIME::Lite monkeypatch

I had an app that calls to MIME::Lite->send() to send an email, which until recently was using an SMTP server slated for decommissioning Real Soon Now.  It was my job to convert it to Amazon SES, and I figured it would be easier to tell MIME::Lite to use SES's SMTP interface instead of importing the web side's full Perl library tree just for one module out of it.

Ha ha!  SES requires SSL, and neither MIME::Lite nor Net::SMTP have any idea about that.  They were both written before the days of dependency injection, so I had to go to some length to achieve it.  And now, I golfed it a bit for you:

package MyApp::Monkey::SMTPS;
use warnings;
use strict;
use parent 'IO::Socket::SSL';

# Substitute us for the vanilla INET socket
require Net::SMTP;
@Net::SMTP::ISA = map {
  s/IO::Socket::INET/MyApp::Monkey::SMTPS/; $_
}  @Net::SMTP::ISA;

our %INET_OPTS = qw(
  PeerPort smtps(465)
  SSL_version TLSv1
); # and more options, probably

# Override new() to provide SSL etc. parameters
sub new {
  my ($cls, %opts) = @_;
  $opts{$_} = $INET_OPTS{$_} foreach keys %INET_OPTS;
  $cls->SUPER::new(%opts);
}

PeerPort overrides the default of smtp(25) built in to Net::SMTP; I needed a port where the whole connection is wrapped in SSL instead of using STARTTLS, and 465 is the one suitable choice of the three that SES-SMTP supports.

The main caveat about this is that it breaks Net::SMTP for anyone else in-process who wants to send mail to a server that lacks a functional port 465.  But as you may have guessed, that's not a problem for my script, today.