Don’t Let HTTP/2 Nest

For some time, I had problems accessing a dev server with HTTP/2.  Asking cURL to use HTTP/1.1 worked fine, so that’s what I did for a long time.

Today, I found the root cause.  I had nginx set up as reverse-proxy/TLS termination (to emulate ALB), proxying requests to apache2.  Both of them had HTTP/2 enabled!  I needed to deactivate support in Apache, and since the system is Debian/Ubuntu based, that meant:

sudo a2dismod http2
sudo systemctl reload apache2

After that, everything worked.

The problem was that the client would connect to nginx with HTTP/2, and then the request would be sent to Apache. Apache's HTTP/2 module would include an Upgrade: h2, h2c header in the response.  Then nginx would dutifully copy this back to the client.  When cURL or PHP streams received this header, they would detect it as invalid: we can’t upgrade to HTTP/2 from inside HTTP/2.

That error-handling resulted in discarding the response body… but not the HTTP 200 status code, which was extremely puzzling.  How could this successful request have failed?  It failed during header processing, after processing the status and before accepting the body.  (I think browsers must ignore it?  Or maybe they don’t use HTTP/2 through a proxy, even with CONNECT requests?  I would have had to figure out the problem much sooner, if they had seen this Upgrade header and treated it as an error.)

The other weird thing about this is that Apache doesn't have TLS configured, but it still provided h2 as an option in its Upgrade header.  I don’t think that’s a reasonable configuration.  It’s especially not a reasonable default, but I’m not sure whether that’s Apache’s problem, Debian’s, or Ubuntu’s.

Using Cloudflare DNS over TLS system-wide on Ubuntu

My current Linux distributions (Ubuntu 23.04 and the Ubuntu-derived Pop!_OS 22.04) use NetworkManager for managing connections, and systemd-resolved for resolving DNS queries.  I’ve set up Cloudflare’s public DNS service with DoT (DNS over TLS) support twice… and I don’t really have a solid conclusion.  Is one “better?”  🤷🏻

Contents

• Per-connection mode with NetworkManager only
• Globally with systemd-resolved / NetworkManager
• Useful background info

Broadband Asymmetry

Someone, somewhere on the intertubes, recently asked: “Why are we seeing 10:1 speed asymmetry?”

I’m pretty sure the speed asymmetry started for good technical reasons: through some signal magic, companies could deliver 48+ Kbps down and 33 Kbps up. ADSL took advantage of the technological landscape of the time (browsing and email were asymmetric) to deliver faster speeds where the customers cared, and indeed, my first DSL services were (if memory serves) in the 1.5/0.38 Mbps area, only a 4:1 asymmetry. My current service is ADSL at 9.3:1 (7.0/0.75), which is both notably closer to 10:1, and hasn’t qualified as broadband since the 4/1 Mbps definition went into effect.

Even though YouTube made video hit the web in a big way—they were there at the crossover point between better codecs and better bandwidth, plus some cleverness* on their part—most traffic was still downstream. The video being delivered was much larger than the return traffic that acknowledged receipt of the video.

The restricted upload rates are thus firmly grounded in historical reality, and they persist today because, I suspect, of two reasons.

One, there’s obviously a chicken-and-egg problem where uploads are less frequent because upload rates are low, and the rates are lower because uploads are less frequent. There’s a natural tendency for uploads to be less frequent anyway (how many funny cat pictures do you look at per picture you upload?) but low upload rates discourage actual upload usage in and of themselves.

Two, I think ISPs are rewarded if they keep upload rates low. Settlement-free peering has traditionally required each side to send “about equal” traffic as the other. If an ISP strongly encourages downloads through 10:1 or more asymmetry, then they will never come close to sending “about equal” traffic out of their network... and they can demand payment from anyone who wants access to their customers, such as Netflix.

I still believe that ISPs should be charging their own customers enough to support their own customers’ data requests including adequate network investment, but that doesn’t invalidate the reality.

As for 10:1 specifically, I can only speculate. It may be, that’s simply the size where sending email and uploading to Facebook “doesn’t seem to take too long” for users. And if more people sent more video to Facebook, then ISPs may reshuffle their plans to provide more upstream “so you can Facebook.” Regardless, in the absence of an obvious technical reason, I must assume it serves a specific marketing purpose.

* At one time, they showed 320-pixel video in a 425-pixel player. Although scaling technically hurts quality, it crossed the gap between “small” and “nicely sized,” looking much better on 1000-pixel browsers.

An AWS VPN: Part III, Routing

The last two posts in this series (part I, IPSec and part II, L2TP) covered enough to get an L2TP/IPSec VPN connection up, to the point where arbitrary traffic can be exchanged between the client and the server.  But, there's a missing feature yet.  Remember this picture from part II?

If we assigned a third network for the VPN, how does the client know that the protected network is even back there to send traffic to?  It's not the remote end of the VPN link (172.16.0.1) so it'll get routed via the gateway and fail at some point.

The answer is that someone conveniently threw in an extra hack for us: the VPN client sends a DHCPINFORM message over the L2TP connection, and the server just has to respond to it with a few vital options.

An AWS VPN: Part II, L2TP/PPP

In part I of this series, we covered the IPSec layer.  With IPSec up and running, we can move on to configuring L2TP; routing will be covered in part III.

In the "usual" VPN setup, we'd cut a hole in the existing DHCP pool in the protected network to reserve as addresses to be given to VPN clients.  In a /24 subnet, network gear might get addresses up to 30, the DHCP pool could run from 31-150, and other static IPs (hi, printers!) might live at 200-254, with 151-199 open for VPN access.

Unfortunately, my protected network in this scenario is AWS, which I can't actually carve random IPs out of to assign to connecting clients.  I have to assign one in a different netblock and hope none of the three (the client, AWS, and my new L2TP-only address pool) conflict.

Luckily, there are three private network spaces defined, and everyone always forgets about the weird one.  Let's set up the connection something like this:

Of course I didn't actually use 172.16.0.x myself, that's asking for almost as much trouble as 192.168.0.x.  (Also, I guess I could have colored the NAT-to-server link green as well, because that's the same network space in my AWS setup.  Too late, we're going!)

An AWS VPN: Part I, IPSec

I recently set up an IPSec/L2TP VPN.  This post is about how I did it, and how I debugged each individual part.  In honor of how long it took me, this will be a three-part series, one for each day of work I spent on it.  Part II will cover L2TP/PPP, and part III will get into routing and DNS.

First things first: IP addresses have been changed.  Some details will apply to Linux, OpenSWAN, xl2tpd, pppd, iproute2, dnsmasq, or road-warrior configurations in particular, but the theory is applicable cross-platform.  I want to cover both, because the theory is invaluable for debugging the practice.

This also covers a few details about putting the VPN server on an AWS instance in EC2 Classic with an elastic IP.  But first, let's take a look at the general structure we have to work with:

A client connecting from a network we don't control, over networks that are possibly evil (hence the need for a VPN), to a server that provides access to vague "other resources" in non-globally-routable space behind that server.  We don't know where the client is going to be, and/or the corporate net is sufficiently large, that we can't use dirty NAT tricks to move each network "into" the other.  (But, spoiler, we will take some inspiration from there.)

Without further ado, here's part I: IPSec.  Part II and III are forthcoming.

Working from Home

Since I've radically changed my work VM configuration, I needed to change my home PC in order to connect properly to the new setup.

Before we dive in, a brief review of the old setup: the host at work, which we will call `audia`, ran a VirtualBox guest named `dev`.  dev had its own set of configs and configuration knobs so that I could write 127.0.0.1 dev.audia.pc dev into /etc/hosts, point Firefox to dev:10080, and code running on dev would be able to recognize "this is development," not require TLS, and know that http://dev:10080/ is the URI root of the app.  I configured VirtualBox's port forwarding to listen on 0.0.0.0 from audia, and when I connected to the VPN, my home computer (call it `starlet`) had its own /etc/hosts listing that used the internal IP of audia for dev.  The same URI from home would then get connected via port-forwarding to the dev VM.

This all changed when I wanted to test a SAN certificate and some redirection rules.  I didn't want certificate mismatch errors in Firefox, and I needed dev to think it was the production server so that it would use the production ruleset.  That became the underlying justification for devproxy.  After that point, dev had the production certificates, hostnames, and rules.  It had a host-only network added, which audia now used, and I took out the port forward.  (This was the problem behind getting Lost in the Complexity.)

But without the port forward, starlet can't reach dev anymore, only audia.  Of course, I came up with a solution:

(Despite what may be implied, the VPN isn't connected to starlet.  I just didn't feel like redrawing the whole tube.)

Tor for VirtualBox Guests (idea only; no code)

To give a guest only one network card with host-only networking, yet still let it access the Internet, we can let it connect to an HTTP proxy running on the host.  If this proxy is polipo, we can configure it to connect to tor’s SOCKS server as its upstream:

Then what happens?  Any and all Internet traffic from the guest VM is delivered via Tor.  Since the guest doesn't have Internet access of its own, any software which doesn't cooperate with the proxy cannot communicate.  Although malware on a compromised guest could still exfiltrate data, it hides the host's true external IP address from the malware.  (Assuming, dangerously, no security bugs in polipo nor VirtualBox that would allow a compromise of the host.)

I said that first, but maybe not very clearly, on twitter.

But… given an appropriate proxy, traffic can be forwarded over any transport.  A proxy could accept data from the guest and transmit it via VPN.  On the other hand, building a VPN client into VirtualBox to offer a VPN network type would let a client connect to a VPN without necessarily allowing other host processes access to it, nor requiring the VPN to be mediated by an additional (dual-homed) guest.

Polipo doesn't have to use tor as a backend, either; it's also perfectly capable of forwarding using ssh's SOCKS proxy.  (This is known as "dynamic tunnel mode" in some clients.)  Compared to the amount of software and configuration needed to set up the average VPN, ssh is just as secure and much easier to get running.

Sending traffic via proxy is an effective way to apply further modifications to the destination stream, without needing the cooperation of software connecting to the proxy.

The Router Trinity

There are technically two separate companies housed in the suite, with separate subnets set up for themselves, but they wanted to be able to exchange data at some point in the past.  The link between the two networks is accomplished by no less than three consumer-grade "cable/dsl routers."

Let's start with a picture so you have something to boggle at: