TIL: HTTP Upgrade

I’m thinking about creating devproxy in a different language. Tracking down some relevant specs, I found the CONNECT RFC.

This RFC includes not only the definition of CONNECT, but alternatively, the use of an Upgrade header to convert a regular HTTP connection to HTTPS, either with optional or mandatory encryption. It was like STARTTLS for HTTP, in a way.

CONNECT won out in the real world, of course, but I find this lost feature kind of fascinating.

Quick comparison:

• Upgrade is a hop-by-hop header. The browser/proxy and proxy/upstream connection MAY be using different levels of encryption.
• When using Upgrade, the proxy needs a valid TLS certificate to handle encrypting traffic with its clients.
• Also, this means the proxy can still view/cache/log the data that was encrypted on the wire.

CONNECT is basically the opposite: once the request is made and the proxy allows it, the proxy reverts to being just as dumb as any router on the Internet. All it can do is shuttle the bytes, so the same bytes that leave the origin end up at the client without any caching or interpretation. Since CONNECT is mainly used for HTTPS, those bytes are most often encrypted, as well.

Google may have tried the Upgrade header when first developing SPDY, but they didn’t like the extra round-trip nor the ability for intermediate devices on the network to interfere (intentionally or otherwise.) So it didn’t end up getting resurrected from the dustbin of history for that, either.

So maybe I didn’t learn about it today, but only rediscovered it.

CGI::Session and your HTTP headers

For CGI::Session to work, you must send the Set-Cookie header (via $session->header() or otherwise) when the session's is_new method returns true.  I discovered this by tripping over an awesome new failure mode today:

1. Restart memcached (or otherwise create new session storage).
2. Nothing stays saved in the session.  Can't log in.

When CGI::Session receives a session ID that doesn't exist in session storage, it changes the session ID to prevent session fixation attacks.  Which means that if you only send the header in the absence of a browser cookie, data is written to the new ID, but the browser will re-submit the old ID next request.

(It turns out my real problem was the stupidly simple error of 'trying to write to the wrong memcached server,' but the above did happen to my test page while I was trying to figure out why memcached wasn't saving anything.)

Auth doesn't belong in the session

PHP locks session access by default, from the time you call session_start() until session_commit(), or until the response is fully written if you didn't commit it earlier.  If you store your authenticated state ("user bob; expires at 12:30") inside the session, then you have to open the session any time you need to know who the user is.  If that makes you set up your app to open the session automatically and leave it open the whole request, then you're hurting parallelism if you have read-only operations.

If you store the auth info in a separate, MAC'd* cookie instead, then you can read the auth state without affecting the session.  Of course, the auth cookie is the most powerful one, so all possible protections should apply: HttpOnly and Secure, served over HTTPS.

* Don't let your users impersonate each other by editing their own cookies.

AmazonS3 'headers' and 'meta' options in the PHP SDK

When you're using create_object, or several other methods of the AmazonS3 class, the $opts parameter often allows for both headers and meta keys.  Although headers is documented as "HTTP headers to send along with the request", it turns out that they are returned with the object (i.e. in the response) when that object is requested from S3.  In contrast, keys in meta are canonicalized, prepended with x-amz-meta-, and returned that way.

That is, if you want to upload filenames like "$id-$md5.pdf" but deliver them to the user as "ContosoUserAgreement.pdf" in the Save-As dialog, then headers should contain a Content-Disposition key with a value of attachment; filename="ContosoUserAgreement.pdf".

If you put it in meta instead, then the HTTP headers on retrieval will contain a x-amz-meta-content-disposition header instead, which will not be honored by the browser.

I found all this out by uploading something like 12,000 files with the wrong metadata.  I then wrote a script to fix it, which ran straight into the problem that update_object doesn't work, so you have to use copy_object instead.  Note that when using copy_object with 'metadataDirective' => 'REPLACE', you need to specify all the metadata you want, because it does what it says: it deletes all old metadata before adding the new metadata from the copy_object request, so that only the new metadata exists afterwards.

Plain Old Data

I’m coming to the conclusion that there’s actually no such thing as “plain data;” it always has some metadata attached.  If it doesn’t, it might be displayed incorrectly, and then a human needs to interfere to determine the correct metadata to apply to fix the problem.  (Example: View → Character Encoding in Firefox.)  Pushed to the extreme, even “just numbers” have metadata: they can be encoded as text, a binary integer/float (IEEE 754 or otherwise) of some size/endianness, or an ASN.1 encoding.

Another conclusion I’m reaching is that HTTP conflates all kinds of metadata.  Coupled with the lack of self-contained metadata in file formats and filesystems, things start to accumulate hacks.

Layer 7 Routing: HTTP Ate the Internet

In the beginning was TCP/IP, and the predominant model was that servers would listen for clients using a pre-established port number.  Then came Sun RPC, in which RPC servers were established dynamically, and listened on semi-random ports (still, one port per service provided); the problem was solved by baking the port mapper into the protocol.  The mapper listens on a pre-established port, and the client first connects there to inquire, "On what port shall I find service X?"

Then came HTTP, the layer 6 protocol masquerading as layer 7.