Don’t Let HTTP/2 Nest

For some time, I had problems accessing a dev server with HTTP/2.  Asking cURL to use HTTP/1.1 worked fine, so that’s what I did for a long time.

Today, I found the root cause.  I had nginx set up as reverse-proxy/TLS termination (to emulate ALB), proxying requests to apache2.  Both of them had HTTP/2 enabled!  I needed to deactivate support in Apache, and since the system is Debian/Ubuntu based, that meant:

sudo a2dismod http2
sudo systemctl reload apache2

After that, everything worked.

The problem was that the client would connect to nginx with HTTP/2, and then the request would be sent to Apache. Apache's HTTP/2 module would include an Upgrade: h2, h2c header in the response.  Then nginx would dutifully copy this back to the client.  When cURL or PHP streams received this header, they would detect it as invalid: we can’t upgrade to HTTP/2 from inside HTTP/2.

That error-handling resulted in discarding the response body… but not the HTTP 200 status code, which was extremely puzzling.  How could this successful request have failed?  It failed during header processing, after processing the status and before accepting the body.  (I think browsers must ignore it?  Or maybe they don’t use HTTP/2 through a proxy, even with CONNECT requests?  I would have had to figure out the problem much sooner, if they had seen this Upgrade header and treated it as an error.)

The other weird thing about this is that Apache doesn't have TLS configured, but it still provided h2 as an option in its Upgrade header.  I don’t think that’s a reasonable configuration.  It’s especially not a reasonable default, but I’m not sure whether that’s Apache’s problem, Debian’s, or Ubuntu’s.

A hat beside the “PHP7 vs HHVM” ring

I spent all my free time, all week, exploring the difference in performance between PHP 5.6.22, PHP 7.0.8, and HHVM 3.14.1.  (HHVM 3.14.2 came out later in the week, but I didn’t want to redo everything.)

In the end, it turned out kind of useless as a “real world” test.  The server doesn’t have L3 cache, and the client is a virtual machine.  I also didn’t run a large number of trials, nor tune the server stacks for optimum performance. In fact, the tunings I tried had no effect, at best.

tl;dr conclusions:

• HHVM proxygen is almost completely amazing.  If you have the option, it is probably what you want. It just crashes hard when concurrency exceeds its open file limit.
• nginx+FastCGI hits a resource limit somewhere, and starts failing requests at higher concurrency, like 160.
• apache+FastCGI does the same… but the limit is higher, between 256 and 384. The price for this is that it’s only 86% as many requests per second.
• Providing more FastCGI workers makes the errors hit sooner, but ramp up more slowly.
• I’m really disappointed in nginx.  I expected more.  A lot more.

Apache Config Merging

(I found this post I drafted a year ago. I don't know why it isn't posted, so here it is...)

The Apache documentation tells you how to order directives in the same scopes, but neglects to remind you about the way different scopes merge.  So, as of 2.4, here's an overview of how it all works.

When the server first receives a URL, it searches for matching containers to determine the active configuration.  The following containers are checked at this stage, highest priority first ("last to first" or "first to last" refers to the order that the sections or directives are listed in the configuration file):

1. <If>, from last to first.
2. <Location> and <LocationMatch>, last to first.

Once the server has a physical path, directives are applied, highest priority first:

1. <Files> and <FilesMatch>, last to first.
2. <DirectoryMatch> sections matching the directory, last to first.
3. From most- to least-specific directory path (e.g. /foo/bar before /), regardless of order:
   1. Directives in .htaccess files, if allowed and present.
   2. <Directory> section.  I suspect, but haven't verified, that multiple Directory sections on the same path (e.g. <Directory /foo></Directory> ... <Directory /foo></Directory>) will apply last to first.

Directives are also merged with VirtualHost and server-wide (outside of any other section) contexts, with priority again being given to VirtualHost over the server-wide directives.  That is, a ProxyPass in the server-wide section will apply to all virtual hosts by default, but a ProxyPass within a <VirtualHost> section will be able to replace the server-wide rule.

The ordering of directives within sections, and what happens with duplicates (at multiple priority levels) in general, is defined by each individual module.

1. RewriteRules execute first-to-last when the section containing them is processed.  Stopping is subject to flags and things (mod_rewrite is a powerful beast): the [L] flag and any that imply it end processing of rewrite rules in that section; for example, rules in <VirtualHost> cannot stop rules in <Directory> from applying.
2. ProxyPass and ProxyPassMatch rules execute first-to-last, stopping when any match is found.  Thus the "longest match first" rule given in mod_proxy's documentation.
3. Alias, AliasMatch, Redirect, and RedirectMatch rules execute first-to-last, stopping when any match is found.  Likewise, this produces a "longest match first" rule that is given in the mod_alias documentation.
4. Whether a URL is tried with other modules (like mod_proxy or mod_alias) after RewriteRules have taken effect depends on how the RewriteRule is written and where it is placed.  I am not sure I understand the finer points of this, but the pass-through flag [PT] exists to force mod_rewrite to treat it as a URL and let other modules have a chance to handle it.

The main takeaway here is to remember what level you're working at, all the time.  If you place two ProxyPassMatch directives each in a separate <LocationMatch> block, then their order of application is defined by the rules for <LocationMatch>.  It is only when ProxyPass/Match directives are sharing the same section (say, both within <VirtualHost>) that they use the longest-match-first rule from mod_proxy.

One other quirk of laying out mod_proxy directives is that they're not actually valid within <Directory> or <Files> sections.  mod_proxy itself deals only with the URL space, and if another module has decided on a file-system path, then Apache is implicitly serving the content directly, as origin server.  It is then too late for pure URL-to-URL manipulations, which is the level that mod_proxy works at.

SNS Deployment update

As an update to this old post from 2013…

We have moved to using php-fpm, so naturally suEXEC has been replaced by php-fpm’s configuration. That allows for running multiple pools, each running PHP scripts under their own user.

We have the “main” pool still using the same unprivileged user as Apache, and then there’s a “privileged” pool that uses the privileged user. Only the traffic on the port receiving SNS notifications is directed to this privileged pool. The main pool still has the same permission it would if it were running under mod_php, to ease the transition.

The transition was relatively painless, but only because I’d already converted per-dir .htaccess files into rules in the server config. It was part micro-optimization, part “I will probably want nginx someday.” Although Apache is still serving our needs admirably.

Apache, FastCGI, and the Authorization header

I couldn’t find much about why Apache (up to and including 2.4.x) doesn’t pass the HTTP Authorization header to FastCGI by searching the Internet, so I fished through their source code.

There’s a function that sets up the default variables to pass to CGI programs, called ap_add_common_vars. This function expects that other users on the server can see these variables with ps -e, so to hide usernames and passwords from appearing in there, it specifically avoids passing Authorization through to CGI.

FastCGI uses the same ap_add_common_vars function, and mod_proxy_fcgi doesn’t take any special care to put the Authorization back in for requests that are sent over a socket. Thus, by default, no FastCGI requests will receive usernames and passwords.

The standard workaround is to use mod_rewrite to set an environment flag if the request has authorization. Because nothing unsets the Authorization—ap_add_common_vars simply skips setting a default—that explicitly-set environment variable will be included in the FastCGI request.

RewriteCond %{HTTP:Authorization} .
RewriteRule / - [E:HTTP_AUTHORIZATION=%{HTTP:Authorization}]

That’s fine if you’re already using mod_rewrite. Can it be done without that module? I haven’t benchmarked the difference, but mod_setenvif can do it via the rule:

SetEnvIf Authorization (.+) HTTP_AUTHORIZATION=$1

tl;dr: Authorization isn’t passed to FastCGI because of the default protection from exposing passwords with slow CGI. It can safely be passed to a FastCGI server (requests are sent over a socket, not through world-readable process attributes) but it must be explicitly passed.

Updated 28 Jul 2015: I finally tested the SetEnvIf rule. I also mentioned what Apache line I'm working with.

More FastCGI: Apache 2.4, PHP-FPM, PSGI, and hot deployment

Driven by mod_fcgid's failings at graceful restart and preforking, I've been looking hard for alternatives.

tl;dr

External FCGI is best done via mod_fastcgi on Apache prior to 2.4, or mod_proxy_fcgi on 2.4.  mod_proxy in general is super awesome on 2.4.

php-fpm ships with an init script and reloads gracefully via SIGUSR2, so that's all you really need there.

For Perl, gracefully restarting an FCGI app is difficult, so the better approach for PSGI apps is to run them in an HTTP server with Server::Starter support (e.g. Starman or others).

Notes on FastCGI and webservers

This post is a distillation of what I've learned over the past couple of months.  There's both new information here, and links to everything else on the FastCGI topic that I've written so far.

Contents

1. mod_fcgid
2. php-fpm (nginx)
3. other things I tried to use
4. concluding remarks

Minimal, Working Perl FastCGI Example, version 2

This is an update to a previous post.  File layout remains the same: "site" is a placeholder for the actual site name, and /home/site/web is the actual repository of the project.  Static files then appear under public, and Perl modules specific to the site in lib/Site (i.e. visible in Perl as Site::Modname when lib is put in @INC).  I am still using mod_fcgid as the FastCGI process manager.

The major improvement: This version handles FCGI-only scripts which have no corresponding CGI URL.  I discovered that limitation of the previous version when I tried to write some new code, where Apache or mod_fcgid realized that the CGI version didn't exist, and returned a 404 instead of passing it through the wrapper.  As a consequence of solving that problem, FcgidWrapper is no longer necessary, which gives the dispatch.fcgi code a much cleaner environment to work in.

Everything I liked about the previous version is preserved here: I can create Site/Entry/login.pm to transparently handle /login.pl as FastCGI, without requiring every other URL to be available in FastCGI form.  It also stacks properly with earlier RewriteRules that turn pretty URLs into ones ending in ".pl".

Apache configuration:

# Values set via SetEnv will be passed in the request;
# to affect Perl startup, it must be FcgidInitialEnv
FcgidInitialEnv PERL5LIB /home/site/web/lib
RewriteCond /home/site/web/lib/Site/Entry/$1.pm -f
RewriteRule ^/+(.+)\.pl$ /home/site/web/dispatch.fcgi [L,QSA,H=fcgid-script,E=SITE_HANDLER:$1]
<directory /home/site/web/fcgi>
    Options ExecCGI FollowSymLinks
    # ...
</directory>

Again, the regular expression of the RewriteRule is matched before RewriteCond is evaluated, so the backreference $1 is available to test whether the file exists.  This time, I also use the environment flag of the RewriteRule to pass the handler to the dispatch.fcgi script.  Since I paid to capture it and strip the leading slashes and extension already, I may as well use it.

That means the new dispatch.fcgi script doesn't have to do as much cleanup to produce the module name:

#!/home/site/bin/perl
use warnings;
use strict;
use FindBin qw($Bin);
use Site::Response;
use Site::Preloader ();
while (my $q = CGI::Fast->new) {
    my ($base, $mod) = ($ENV{SITE_HANDLER});
    $base =~ s#/+#::#g;
    $base =~ s#[^\w:]##g;
    $base ||= 'index';
    $mod = "Site::Entry::$base";
    my $r = Site::Response->new($base, "$Bin/templates");
    eval {
        eval "require $mod;"
            and $mod->invoke($q, $r);
    } or warn "$mod => $@";
    $r->send($q);
}

I remembered to include the $r->send call this time.  I pass the CGI query object so the response can call $q->header.  That's not strictly necessary—FCGI children process one request at a time and copy $q to the default CGI object, meaning header should work fine alone, but I didn't know that yet.

I also remove non-{word characters or colons} from the inbound request for security, since my site uses URLs like /path/somereport.pl.  You may need to carefully adjust that for your site.

Site::Response is initialized as a generic error so that if the module dies, the response written to the client is a complete generic error.  Otherwise, the template is selected and data set, so the send call ships the completed page instead.

The only thing left that I'd like to do is make this configuration more portable between web servers instead of dependent on Apache's mod_rewrite and mod_fcgid, but since Apache isn't killing us at work, it probably won't happen very soon.

mod_fcgid and graceful restarts

I see plenty of this in my logs when the server needs reloaded to pick up fresh Perl:

(43)Identifier removed: mod_fcgid: can't lock process table in pid 3218

tl;dr: this appears to be harmless in practice.

The leading portion corresponds to EIDRM (see errno(3)) which comes back out of pthread_mutex_lock and cheerfully gets logged as the failure code of proctable_lock_internal.  The proctable is in turn locked during request handling.

My best guess for the order of events is that the Apache parent receives a graceful restart, unloads and reloads mod_fcgid, which destroys the mutex as a side effect.  After old-generation children tie up their requests, they try to notify their parent that they're available again, only to discover that the mutex is gone.  The child then exits, but it doesn't hurt any clients because they've already been served at this point.

This problem is not fixable in Apache 2.2 because there aren't any hooks for graceful-restart.  It just unloads DSOs without warning, and their first clue anything happened is that they start receiving config events.  By then, the mutex and process table are gone, so the newly-loaded master can't communicate with old-generation children.  Someone did make an attempt to fix this for 2.4 (along with modifying mod_cgid to test their infrastructure) but AFAICT nobody has made this available in mod_fcgid for 2.4 yet.

A Minimal, Working Perl FastCGI Example

Updated 2013-01-30

Please see the next version, which allows for newly-written FastCGI scripts that do not have a CGI equivalent.

Original post follows.

I couldn't really find one of these on the Internet, so I'm going to document what I have working so far. In the following examples, assume an application named "site" rooted at /home/site/web, with static files served out of public/, app-specific modules contained under lib/, and other perl module paths (vendor, local::lib, etc.) compiled into Perl itself.  Site::* references an app-specific module, naturally.

Apache configuration snippet, using mod_fcgid as the FastCGI process manager:

DocumentRoot /home/site/web/public
FcgidInitialEnv PERL5LIB /home/site/web/lib
FcgidWrapper /home/site/web/handler.fcgi
RewriteEngine on
RewriteCond /home/site/web/lib/Site/Entry/$1.pm -f
RewriteRule ^/(.*)\.pl$ - [QSA,L,H=fcgid-script]

Note that the left of RewriteRule is matched, and the $1 reference assigned, prior to evaluating RewriteCond's.  The rule means, "if the requested filename exists where the FastCGI wrapper will look for it, force it to be handled by FastCGI."  This interacts perfectly with DirectoryIndex: requesting / with a DirectoryIndex index.pl in effect invokes the handler.fcgi for index.pl as long as Site::Entry::index exists.

Now, my handler.fcgi named in the Apache configuration for the FcgidWrapper looks like this:

#!/home/site/bin/perl
use warnings;
use strict;
use CGI::Fast;
use FindBin;
use Site::Preloader ();
while (my $q = CGI::Fast->new) {
    my ($base, $mod) = ($ENV{SCRIPT_FILENAME});
    $base = substr($base, length $ENV{DOCUMENT_ROOT});
    $base =~ s/\.pl$//;
    $base =~ s#^/+##;
    $base =~ s#/+#::#g;
    $base ||= 'index';
    $mod = "Site::Entry::$base";
    my $r = eval {
        eval "require $mod;"
            and $mod->invoke($q);
    };
    warn "$mod => $@" unless defined $r;
}

This means that I can have models named like Site::Login and view-controllers for them under Site::Entry::login (handling /login.pl, naturally).  I still have to rewrite the site from vanilla CGI scripts into module form, but the RewriteRule work above means that the FastCGI versions can be picked up URL-by-URL.  It doesn't require a full-site conversion to a framework to gain any benefits.

There's one additional feature this wrapper has: by using SCRIPT_FILENAME and removing DOCUMENT_ROOT off the front, I can rewrite a pretty URL (prior to the last RewriteRule shown above) to one ending .pl and still have the wrapper work.  SCRIPT_NAME keeps the name of the script as it was in the original request, and does not receive the rewritten value.  (Only SCRIPT_FILENAME does, on my exact setup.)  So I did it this way, rather than re-applying all my rewrites in Perl.

mod_fcgid and PHP_FCGI_CHILDREN

While attempting to validate a configuration for some server benchmarking, I happened upon some curious pstree output which read in part:

httpd─┬─httpd───10*[php-cgi───8*[php-cgi]]
      └─23*[httpd]

I had expected at most 8 php-cgi children, having configured PHP_FCGI_CHILDREN=8 in the wrapper script, but mod_fcgid was clearly starting multiple instances of the wrapper, so I was getting an absurd number of children.

I discovered FcgidMaxProcesses and further benchmarks ensued:

MaxProc	CHILDREN	req/sec	MiB/sec
4	5	1279	4.0
4	2	1221	3.8
10	2	2488	7.8

The first two rows, in spite of the second only providing 40% of the available PHP children, are within error of each other in actual performance.  The third line produces a total of 20 children again, but now 10 of them instead of 4 are visible in mod_fcgid for handling requests.

Clearly, there are too many cooks in the kitchen: both mod_fcgid and php-cgi (which, I should mention, was built with --disable-fpm) are trying to act as master, so PHP ends up wasting $PHP_FCGI_CHILDREN - 1 processes because mod_fcgid won't pass concurrent requests for them to handle.

What happens if you don't configure PHP_FCGI_CHILDREN at all?  php-cgi handles the request directly, as if it were a child to begin with, and pstree looks more like this:

httpd─┬─httpd───12*[php-cgi]
      └─25*[httpd]

That looks rational.  How does it perform?

MaxProc	CHILDREN	req/sec	MiB/sec
10	unset	2580	8.2
20	unset	2617	8.3

Concurrency was held the same at 16 throughout all tests today, so it looks like we've run into the limits of the 100 mbit network.  (Serious benchmarking will happen once my gigabit gear arrives.)  Still, the fact that performance keeps up with 10×2 children portends well; it means there's no penalty for unsetting PHP_FCGI_CHILDREN and mod_fcgid isn't broken without it.

War Story: Apache, SSL, and name-based vhosts

Note: this post was written about a year ago, before we completed some major upgrades to our infrastructure.  I meant to post it as soon as we were done, but it got buried under too many other posts and drafts.  The original post follows, without edits for temporal accuracy.

You can do it The Right Way and use SNI if:

1. You don't care about Internet Explorer (7 and 8) on Windows XP.
2. You have Apache 2.2.12 or newer.
3. You have openssl 0.9.8f or newer with TLS extensions; extensions are included by default in 1.0.

If all of these are okay for you, then setup is a matter of enabling NameVirtualHost *:443 and setting up TLS vhosts in much the same way as regular vhosts.

Otherwise, you have to try a bit harder.