An AWS VPN: Part III, Routing

The last two posts in this series (part I, IPSec and part II, L2TP) covered enough to get an L2TP/IPSec VPN connection up, to the point where arbitrary traffic can be exchanged between the client and the server.  But, there's a missing feature yet.  Remember this picture from part II?

If we assigned a third network for the VPN, how does the client know that the protected network is even back there to send traffic to?  It's not the remote end of the VPN link (172.16.0.1) so it'll get routed via the gateway and fail at some point.

The answer is that someone conveniently threw in an extra hack for us: the VPN client sends a DHCPINFORM message over the L2TP connection, and the server just has to respond to it with a few vital options.

An AWS VPN: Part II, L2TP/PPP

In part I of this series, we covered the IPSec layer.  With IPSec up and running, we can move on to configuring L2TP; routing will be covered in part III.

In the "usual" VPN setup, we'd cut a hole in the existing DHCP pool in the protected network to reserve as addresses to be given to VPN clients.  In a /24 subnet, network gear might get addresses up to 30, the DHCP pool could run from 31-150, and other static IPs (hi, printers!) might live at 200-254, with 151-199 open for VPN access.

Unfortunately, my protected network in this scenario is AWS, which I can't actually carve random IPs out of to assign to connecting clients.  I have to assign one in a different netblock and hope none of the three (the client, AWS, and my new L2TP-only address pool) conflict.

Luckily, there are three private network spaces defined, and everyone always forgets about the weird one.  Let's set up the connection something like this:

Of course I didn't actually use 172.16.0.x myself, that's asking for almost as much trouble as 192.168.0.x.  (Also, I guess I could have colored the NAT-to-server link green as well, because that's the same network space in my AWS setup.  Too late, we're going!)

An AWS VPN: Part I, IPSec

I recently set up an IPSec/L2TP VPN.  This post is about how I did it, and how I debugged each individual part.  In honor of how long it took me, this will be a three-part series, one for each day of work I spent on it.  Part II will cover L2TP/PPP, and part III will get into routing and DNS.

First things first: IP addresses have been changed.  Some details will apply to Linux, OpenSWAN, xl2tpd, pppd, iproute2, dnsmasq, or road-warrior configurations in particular, but the theory is applicable cross-platform.  I want to cover both, because the theory is invaluable for debugging the practice.

This also covers a few details about putting the VPN server on an AWS instance in EC2 Classic with an elastic IP.  But first, let's take a look at the general structure we have to work with:

A client connecting from a network we don't control, over networks that are possibly evil (hence the need for a VPN), to a server that provides access to vague "other resources" in non-globally-routable space behind that server.  We don't know where the client is going to be, and/or the corporate net is sufficiently large, that we can't use dirty NAT tricks to move each network "into" the other.  (But, spoiler, we will take some inspiration from there.)

Without further ado, here's part I: IPSec.  Part II and III are forthcoming.

VPN Terminology: PPTP and L2TP/IPSEC

A little reminder before I get started:

Do not use PPTP

Seriously, do not use it. It’s insecure and it occasionally has terrible performance due to its underlying design. Don’t use it.

Just don’t.

With that aside…

Remembering PPP

Back in the days of dialup, people used physical circuits like the telephone network to connect to the Internet. On that last hop between the home and the ISP, the protocol that ferried IP directly was PPP, the Point-to-Point Protocol.

PPP was designed for an ISP to serve dial-in lines, so it had a bit more to worry about than just ferrying IP packets. It had to assign addresses between the endpoints (IPCP) as well as ensure that Internet service was being given to a paying user by authenticating them (PAP or CHAP).

In those simpler times, one physical machine at the ISP’s end would connect to both the Internet and the telephone network.

Some people, though, needed so much internet that they wanted to dial in with multiple lines, and have packets routed down both of them. This was known as “multilink PPP” and it had a problem. What if the ISP doesn’t have just one machine handling dial-up connections?

It could be the case that a client would attempt to set up a multilink PPP session, but their calls were on different machines, and The Internet couldn’t really route to both of them. So, faced with different PPP servers, the client had little choice but to hang up, dial in again, and hope that this time, they hit the same servers.

Tunneling

PPTP was largely motivated by the desire to solve this problem, by splitting Big Machines into two parts: one (or more) to handle dial-in lines, maintaining connections to one (or more?) machines which were actually on the Internet.

Handling calls from the clients were the PPTP Access Concentrators, or PACs, which would then deliver traffic to the PPTP Network Server, or PNS. The “tunnel” between these two parts was a single channel/circuit (I presume) that carried the PPTP control traffic and tunneled packets between PAC and PNS for de/multiplexing.

In this scenario, the client-to-PAC connection is still regular PPP; a portion of the PPP exchange (setting up multilink) is forwarded to the PNS, who can then see all client links, even across PACs. Thus, it can route effectively. From the point of view of the Internet, it really is routing all the traffic to one machine, the PNS.

The PNS must also handle the IPCP portion of PPP setup, since it’s the server with IP connectivity.

Much of the terminology in the PPTP RFC is defined with reference to this situation, and says nothing about VPNs themselves, because the protocol itself was not originally defined as a VPN protocol.

VPN puts a network in your networks

However, if the tunnel is encrypted then we can convert it into a VPN technology by moving the tunnel off ISP equipment and onto the public Internet.

In this case, the VPN client is playing the role of PAC and the server plays PNS again; the connection between them is a TCP channel over which all network packets are forwarded. Like regular PPP, there are control messages, and also data packets riding on the single established channel.

This is where performance trouble turns out to be lurking: if a TCP connection is forwarded, it gets forwarded reliably. It’s never going to be lost on the journey over the PPTP VPN. That inner TCP connection can’t detect the path MTU by packet loss, and this does not play well with systems that self-control their network rates based on packet loss.

(Ironically, recent bufferbloat mitigations that try to measure RTT and slow down when RTT increases are much better suited to this scenario.)

Layer 2 Tunneling (L2TP)

Someone decided to fix that, so an L2TP connection is carried over UDP packets instead. They implement their own reliability protocol for control messages, and rely on the tunnelled connections to handle any packet loss. That fixes the performance issues facing PPTP.

L2TP is also simplified a bit from PPTP, in that it doesn’t offer any encryption of its own. That’s why it has to be run over IPSEC to form a VPN; but otherwise, it’s conceptually near identical to PPTP.

(Of course, all those “P” abbreviations also become “L,” but actually crossing them all out cluttered the image too much.)

With encryption provided by the IPSEC layer, any weakness of L2TP becomes irrelevant. But it seems like a horrible hack, because IPSEC isn’t concerned with hosts; it’s concerned with networks and routing.

Like PPTP, the L2TP RFC is not that worried about defining all this stuff in terms of how an L2TP VPN would be set up in practice.  Rather, it reads like a minimal number of changes were made to the PPTP RFC in order to make the required implementation changes obvious to people already writing PPTP VPN software.

IP Security (IPSEC)

IPSEC itself establishes an encrypted channel with integrity checking; it’s just anonymous.  L2TP provides a point to get a specific username/password pair after the IPSEC key exchange (IKE) is done.

If you’re using a pre-shared key (PSK), this oddly means that you need two passwords to connect up the VPN: one site-wide password to establish the IPSEC channel, and a separate per-user password to set up the L2TP tunnel inside.

Sediment

There you have it: VPNs today are generally complicated and awful because people have been adapting other tools to the job, and the layers have just been accumulating.

It should be noted that, as long as this post is, it does not cover other approaches to VPNs: it remains laser-focused on L2TP. OpenVPN and sshuttle are out of scope.

Working from Home

Since I've radically changed my work VM configuration, I needed to change my home PC in order to connect properly to the new setup.

Before we dive in, a brief review of the old setup: the host at work, which we will call `audia`, ran a VirtualBox guest named `dev`.  dev had its own set of configs and configuration knobs so that I could write 127.0.0.1 dev.audia.pc dev into /etc/hosts, point Firefox to dev:10080, and code running on dev would be able to recognize "this is development," not require TLS, and know that http://dev:10080/ is the URI root of the app.  I configured VirtualBox's port forwarding to listen on 0.0.0.0 from audia, and when I connected to the VPN, my home computer (call it `starlet`) had its own /etc/hosts listing that used the internal IP of audia for dev.  The same URI from home would then get connected via port-forwarding to the dev VM.

This all changed when I wanted to test a SAN certificate and some redirection rules.  I didn't want certificate mismatch errors in Firefox, and I needed dev to think it was the production server so that it would use the production ruleset.  That became the underlying justification for devproxy.  After that point, dev had the production certificates, hostnames, and rules.  It had a host-only network added, which audia now used, and I took out the port forward.  (This was the problem behind getting Lost in the Complexity.)

But without the port forward, starlet can't reach dev anymore, only audia.  Of course, I came up with a solution:

(Despite what may be implied, the VPN isn't connected to starlet.  I just didn't feel like redrawing the whole tube.)

Tor for VirtualBox Guests (idea only; no code)

To give a guest only one network card with host-only networking, yet still let it access the Internet, we can let it connect to an HTTP proxy running on the host.  If this proxy is polipo, we can configure it to connect to tor’s SOCKS server as its upstream:

Then what happens?  Any and all Internet traffic from the guest VM is delivered via Tor.  Since the guest doesn't have Internet access of its own, any software which doesn't cooperate with the proxy cannot communicate.  Although malware on a compromised guest could still exfiltrate data, it hides the host's true external IP address from the malware.  (Assuming, dangerously, no security bugs in polipo nor VirtualBox that would allow a compromise of the host.)

I said that first, but maybe not very clearly, on twitter.

But… given an appropriate proxy, traffic can be forwarded over any transport.  A proxy could accept data from the guest and transmit it via VPN.  On the other hand, building a VPN client into VirtualBox to offer a VPN network type would let a client connect to a VPN without necessarily allowing other host processes access to it, nor requiring the VPN to be mediated by an additional (dual-homed) guest.

Polipo doesn't have to use tor as a backend, either; it's also perfectly capable of forwarding using ssh's SOCKS proxy.  (This is known as "dynamic tunnel mode" in some clients.)  Compared to the amount of software and configuration needed to set up the average VPN, ssh is just as secure and much easier to get running.

Sending traffic via proxy is an effective way to apply further modifications to the destination stream, without needing the cooperation of software connecting to the proxy.

PPTP is legacy and insecure

Unlike IPSec and L2TP+IPSec, a PPTP VPN tunnel is carried over TCP, which means all packets traveling inside the tunnel are delivered reliably—including any tunneled TCP traffic.  Therefore, the inner TCP layer never sees packet loss, wreaking havoc with its congestion control mechanisms.

Although the RFC refers to the tunnel as "separate", that's only conceptual.  The traffic is carried inside of the single TCP connection.  For example, SSH application data delivered over PPTP is wrapped in its usual TCP/IP, then PPP+GRE (the tunnel/"user data" from PPTP's perspective), then forwarded over the TCP/IP control connection to the PPTP server.

L2TP produces a similar wrapping structure, but the outermost connection is UDP instead.  Its IP frames are wrapped in PPP+GRE and then delivered to the server over UDP/IP.

Consider L2TP+IPSec first.  (At least, my experience with vanilla IPSec has been less than recommendable.)

Update: In the news this year (2012), PPTP's key exchange is broken, and that's actually been known about for years.  There's even a cloud service to crack it for you if you're lazy.  Current advice is actually:
Never use PPTP.

Reference: Moxie Marlinspike's post detailing the 2⁵⁶ complexity of the attack: the md4(password) is used as DES keys which encrypt the same known plaintext, meaning that each DES encryption can potentially reveal any of the segments of the hashed password string. The plaintext password is not necessary for the protocol. As a bonus, the final DES key is padded with five zero octets since the hash isn't long enough for three DES keys.