Don’t Let HTTP/2 Nest

For some time, I had problems accessing a dev server with HTTP/2.  Asking cURL to use HTTP/1.1 worked fine, so that’s what I did for a long time.

Today, I found the root cause.  I had nginx set up as reverse-proxy/TLS termination (to emulate ALB), proxying requests to apache2.  Both of them had HTTP/2 enabled!  I needed to deactivate support in Apache, and since the system is Debian/Ubuntu based, that meant:

sudo a2dismod http2
sudo systemctl reload apache2

After that, everything worked.

The problem was that the client would connect to nginx with HTTP/2, and then the request would be sent to Apache. Apache's HTTP/2 module would include an Upgrade: h2, h2c header in the response.  Then nginx would dutifully copy this back to the client.  When cURL or PHP streams received this header, they would detect it as invalid: we can’t upgrade to HTTP/2 from inside HTTP/2.

That error-handling resulted in discarding the response body… but not the HTTP 200 status code, which was extremely puzzling.  How could this successful request have failed?  It failed during header processing, after processing the status and before accepting the body.  (I think browsers must ignore it?  Or maybe they don’t use HTTP/2 through a proxy, even with CONNECT requests?  I would have had to figure out the problem much sooner, if they had seen this Upgrade header and treated it as an error.)

The other weird thing about this is that Apache doesn't have TLS configured, but it still provided h2 as an option in its Upgrade header.  I don’t think that’s a reasonable configuration.  It’s especially not a reasonable default, but I’m not sure whether that’s Apache’s problem, Debian’s, or Ubuntu’s.