Daisywriter

Growing up, my dad had an old daisy wheel printer hooked up to our computers.  (We had various Commodore hardware; we did not get a PC until 1996, and I'm not sure the Amiga 500 was really decommissioned until the early 2000s.)

The daisy wheel was like a typewriter on a circle.  There were cast metal heads with the letters, numbers, and punctuation on them.  The wheel was spun to position, and then what was essentially an electric hammer punched the letter against the ribbon and the paper.  Then the whole wheel/ribbon/hammer carriage moved to the next position, and the cycle repeated.

This was loud; like a typewriter, amplified by the casing and low-frequency vibrations carried through the furniture.  No printing could be done if anyone in the house had gone to bed, or was napping.

It was also huge. It could have printed on legal paper in landscape mode.

Because of the mechanical similarity to typewriters, the actual print output looked like it was really typewritten.  Teachers would compliment me for my effort on that, and I'd say, "Oh, my dad has a printer that does that."

Nowadays, people send a command language like PostScript, PCL, or even actual PDF to the printer, and it draws on the page.  Everything is graphics; text becomes curves.  But the Daisywriter simply took ASCII in from the parallel port, and printed it out.

My Path to Programming

Inspired by Dan Luu’s post about the same, in turn based on Tavish’s.

For me, it really started with a sibling rivalry.  I wanted to try everything my older brother did, and secretly, I wanted to be better at it than he was. We had wars about who could write better BASIC programs.  When my dad bought AssemPro for our Amiga, my brother decided to learn it: therefore, so did I.

TIL: HTTP Upgrade

I’m thinking about creating devproxy in a different language. Tracking down some relevant specs, I found the CONNECT RFC.

This RFC includes not only the definition of CONNECT, but alternatively, the use of an Upgrade header to convert a regular HTTP connection to HTTPS, either with optional or mandatory encryption. It was like STARTTLS for HTTP, in a way.

CONNECT won out in the real world, of course, but I find this lost feature kind of fascinating.

Quick comparison:

• Upgrade is a hop-by-hop header. The browser/proxy and proxy/upstream connection MAY be using different levels of encryption.
• When using Upgrade, the proxy needs a valid TLS certificate to handle encrypting traffic with its clients.
• Also, this means the proxy can still view/cache/log the data that was encrypted on the wire.

CONNECT is basically the opposite: once the request is made and the proxy allows it, the proxy reverts to being just as dumb as any router on the Internet. All it can do is shuttle the bytes, so the same bytes that leave the origin end up at the client without any caching or interpretation. Since CONNECT is mainly used for HTTPS, those bytes are most often encrypted, as well.

Google may have tried the Upgrade header when first developing SPDY, but they didn’t like the extra round-trip nor the ability for intermediate devices on the network to interfere (intentionally or otherwise.) So it didn’t end up getting resurrected from the dustbin of history for that, either.

So maybe I didn’t learn about it today, but only rediscovered it.

When Did PHP Get That?

5.6.3

• 64-bit pack codes

5.6.0

• Variadic functions with "..." (and related packing/unpacking with the ... operator)
• ** operator for exponentiation
• Constant scalar expressions (const FOO = 2*60*60)
• use const and use function make constants/functions importable from other namespaces
• htmlspecialchars() and friends use the INI default_charset setting as default (update to 5.4 behavior)
• php://input can be reused and $HTTP_RAW_POST_DATA can be disabled (and will be, in the future)
• File uploads can be >2GB
• Constant-time string comparison (aka timing-safe comparison) added as hash_equals
  • I am still curious: why hash_equals? What's the relation to hashing?

5.5.0

• Generators
• finally keyword for exception handling with try/catch
• Password hashing API: password_hash and related functions
• list() in foreach: foreach($nest as list($x, $y))
• PBKDF2 support in hash and openssl (hash_pbkdf2, openssl_pbkdf2)
• empty() accepts arbitrary expressions, not just lvalues
• array and string literal dereferencing: "php"[2] or [1,2,10][2]
• But is that useful?
• ::class syntax: namespace A\B { class C { }; echo C::class; /* A\B\C */ }
• OpCache extension added
• MySQL-nd supports sha256-based authentication, which has been available since MySQL 5.6.6
• curl share handles via curl_share_init and friends [added to this post on 2015-02-17]
• DEPRECATED: preg_replace's /e modifier, in favor of preg_replace_callback

5.4.0
Removed:

• safe_mode
• register_globals
• magic_quotes
• call-time pass-by-reference
• using TZ when guessing the default timezone
• Salsa10 and Salsa20 hash algorithms

Added:

• Traits
• Class::{expr} syntax
• Short array syntax: [2,3] for array(2,3)
• Binary literal syntax: 0b1101
• Closures can use $this in the body: function some_method () { return function () { return $this->protected_thing(); } }
• Using new objects without assigning them: (new Foo)->bar()
• CLI server
• session_status function
• header_register_callback function
• mysqli_result now implements Traversable: $r=$dbh->query(...); foreach ($r as $res) { ... }
• htmlspecialchars() and friends default to UTF-8 instead of ISO-8859-1 for character set.
• This would be changed to use the default_charset INI setting in 5.6.
• 5.4's documentation pointed out that default_charset could be used by sending an empty string, "", as the character set to htmlspecialchars—and said you didn't really want it.

5.3.0
Added:

• Namespaces
• Late static binding and associated 'static::method()' call syntax
• "Limited" goto support
• Closures aka anonymous functions, although they cannot make use of $this until 5.4
• __callStatic() and __invoke() magic methods, to support $obj($arg) call syntax
• 'nowdoc' and double-quoted 'heredoc' syntax
• $x=<<<'FOO' consumes text until FOO and does not do any interpolation / variable substitution on it
• $x=<<<"FOO" acts just like traditional $x=<<<FOO, both consuming text until FOO and doing interpolation / variable substitution as if it were a double-quoted string.
• Constants definable outside of classes with the const keyword, not just the define function
• Short ternary operator: a ?: b equivalent to a ? a : b
• $cls::$foo syntax for accessing static members of a class whose name is stored in a variable ($cls in this example)
• Chained exceptions: the constructor sprouted an extra parameter for a previous exception (generally should mean "the cause of this exception being constructed")
• DEPRECATED: many features that would be removed in 5.4.0, including register_globals, magic_quotes, and safe_mode.

VPN Terminology: PPTP and L2TP/IPSEC

A little reminder before I get started:

Do not use PPTP

Seriously, do not use it. It’s insecure and it occasionally has terrible performance due to its underlying design. Don’t use it.

Just don’t.

With that aside…

Remembering PPP

Back in the days of dialup, people used physical circuits like the telephone network to connect to the Internet. On that last hop between the home and the ISP, the protocol that ferried IP directly was PPP, the Point-to-Point Protocol.

PPP was designed for an ISP to serve dial-in lines, so it had a bit more to worry about than just ferrying IP packets. It had to assign addresses between the endpoints (IPCP) as well as ensure that Internet service was being given to a paying user by authenticating them (PAP or CHAP).

In those simpler times, one physical machine at the ISP’s end would connect to both the Internet and the telephone network.

Some people, though, needed so much internet that they wanted to dial in with multiple lines, and have packets routed down both of them. This was known as “multilink PPP” and it had a problem. What if the ISP doesn’t have just one machine handling dial-up connections?

It could be the case that a client would attempt to set up a multilink PPP session, but their calls were on different machines, and The Internet couldn’t really route to both of them. So, faced with different PPP servers, the client had little choice but to hang up, dial in again, and hope that this time, they hit the same servers.

Tunneling

PPTP was largely motivated by the desire to solve this problem, by splitting Big Machines into two parts: one (or more) to handle dial-in lines, maintaining connections to one (or more?) machines which were actually on the Internet.

Handling calls from the clients were the PPTP Access Concentrators, or PACs, which would then deliver traffic to the PPTP Network Server, or PNS. The “tunnel” between these two parts was a single channel/circuit (I presume) that carried the PPTP control traffic and tunneled packets between PAC and PNS for de/multiplexing.

In this scenario, the client-to-PAC connection is still regular PPP; a portion of the PPP exchange (setting up multilink) is forwarded to the PNS, who can then see all client links, even across PACs. Thus, it can route effectively. From the point of view of the Internet, it really is routing all the traffic to one machine, the PNS.

The PNS must also handle the IPCP portion of PPP setup, since it’s the server with IP connectivity.

Much of the terminology in the PPTP RFC is defined with reference to this situation, and says nothing about VPNs themselves, because the protocol itself was not originally defined as a VPN protocol.

VPN puts a network in your networks

However, if the tunnel is encrypted then we can convert it into a VPN technology by moving the tunnel off ISP equipment and onto the public Internet.

In this case, the VPN client is playing the role of PAC and the server plays PNS again; the connection between them is a TCP channel over which all network packets are forwarded. Like regular PPP, there are control messages, and also data packets riding on the single established channel.

This is where performance trouble turns out to be lurking: if a TCP connection is forwarded, it gets forwarded reliably. It’s never going to be lost on the journey over the PPTP VPN. That inner TCP connection can’t detect the path MTU by packet loss, and this does not play well with systems that self-control their network rates based on packet loss.

(Ironically, recent bufferbloat mitigations that try to measure RTT and slow down when RTT increases are much better suited to this scenario.)

Layer 2 Tunneling (L2TP)

Someone decided to fix that, so an L2TP connection is carried over UDP packets instead. They implement their own reliability protocol for control messages, and rely on the tunnelled connections to handle any packet loss. That fixes the performance issues facing PPTP.

L2TP is also simplified a bit from PPTP, in that it doesn’t offer any encryption of its own. That’s why it has to be run over IPSEC to form a VPN; but otherwise, it’s conceptually near identical to PPTP.

(Of course, all those “P” abbreviations also become “L,” but actually crossing them all out cluttered the image too much.)

With encryption provided by the IPSEC layer, any weakness of L2TP becomes irrelevant. But it seems like a horrible hack, because IPSEC isn’t concerned with hosts; it’s concerned with networks and routing.

Like PPTP, the L2TP RFC is not that worried about defining all this stuff in terms of how an L2TP VPN would be set up in practice.  Rather, it reads like a minimal number of changes were made to the PPTP RFC in order to make the required implementation changes obvious to people already writing PPTP VPN software.

IP Security (IPSEC)

IPSEC itself establishes an encrypted channel with integrity checking; it’s just anonymous.  L2TP provides a point to get a specific username/password pair after the IPSEC key exchange (IKE) is done.

If you’re using a pre-shared key (PSK), this oddly means that you need two passwords to connect up the VPN: one site-wide password to establish the IPSEC channel, and a separate per-user password to set up the L2TP tunnel inside.

Sediment

There you have it: VPNs today are generally complicated and awful because people have been adapting other tools to the job, and the layers have just been accumulating.

It should be noted that, as long as this post is, it does not cover other approaches to VPNs: it remains laser-focused on L2TP. OpenVPN and sshuttle are out of scope.

Perl non-CGI: The Missing Overview

After living in mod_php and Perl CGI for far too long, it was time to look at reworking our application to fit something else.  Although we had mod_perl installed and doing little more than wasting resources, I didn’t want to bind the app tightly to mod_perl in the way it was already tightly bound to CGI.  That meant surveying the landscape and trying to understand what modern Perl web development actually looks like.

But first, some history!

War Story: The day I nearly invented server management

Long before I knew about puppet (or heard about chef via vagrant), I was working on a system with around 100 virtual hosts, nearly all of them configured from a small pool of standard features.  This made for a long, complicated vhosts.conf file, which we managed by keeping it in alphabetical order and using search a lot.  I realized that a lot of the duplication could be removed by generating it from a template file.

Layer 7 Routing: HTTP Ate the Internet

In the beginning was TCP/IP, and the predominant model was that servers would listen for clients using a pre-established port number.  Then came Sun RPC, in which RPC servers were established dynamically, and listened on semi-random ports (still, one port per service provided); the problem was solved by baking the port mapper into the protocol.  The mapper listens on a pre-established port, and the client first connects there to inquire, "On what port shall I find service X?"

Then came HTTP, the layer 6 protocol masquerading as layer 7.

War Story: power.sh

When I attended community college, the computers in the labs were running Windows 95, pretty much in a state of constant hilarity.  I'll get to that some other time, though; today's wacky hijinks are about The Server: the most secure machine on all of the campus, since it was the master authentication source.

Character Sets, Encodings, MySQL, and your data

This post is a companion to Perl and Unicode in Brief, an attempt to cover similar ground more concisely.  And this post is a revised version of the one you're currently reading.

I'm currently moving data from a (relatively old now) MySQL 5.0 server into Amazon RDS.  I've been here before, when I was moving data from MySQL 4.x into 5.0 and mangling character sets.  This time, I want to make 100% sure everything comes across with maximum fidelity, and also get the character encoding as stored to be labeled correctly in MySQL.

First, a quick definition or two:

• Character Set: a specific table to translate between characters and numbers.  Example: ASCII defines characters for numbers 0-127; "A" is 65.  This can also be described as "a set of characters, and their corresponding representation inside the computer."
• Character Encoding: a means of "packing" numbers from the character set into a container.  Example: UTF-8.  The Unicode character 0x2013 becomes 0xE2,80,99. The "E" signifies "Part 1 of 3", and part of the remaining bytes simply indicate "Continued"; the 0x2013 is then divided up to fit in the parts of the bytes that aren't indicating their "Part 1" or "Continued" status.  In the specific case of UTF-8, the encoding is designed so that the ASCII range 0-127 (0x00-7F) is encoded without change: a leading 0-7 means "Part 1 of 1".
• 8-bit character encoding: In older, simpler days, character sets defined only as many characters as could fit in 8 bits, and defined the encoding as simply the numbers.  Character number 181 would encode as a byte (8 bits) with value 181.
• A character encoding implies the associated character set, because the encoding defines how numbers in its character set become individual bytes.  How characters in other sets would be encoded is left undefined and basically impossible.

This last point is why MySQL lets you set "character sets" to UTF-8, though the latter is an encoding.

War Story: The Training Jump

One of the first things I did at my current job was to rewrite a Perl/CGI (the module, and the actual cgi-script execution model) site into PHP.  Part of this site implemented a single-signon (SSO) system for a partner site that hosts our training videos.  Clicking the link led to the innocuously-named "training_jump.pl" CGI script.

The goal in life of the training_jump is to redirect a user to the partner site, passing along the username and email address.  The partner site creates the user if necessary, starts a session for them on its server, and ultimately displays the actual training content.

Inside training_jump is an innocuous-looking "use OAuth::Lite;" line.  I didn't know what OAuth was at the time, so of course I went and looked it up: OAuth is designed to let a site like ExampleMashup authenticate someone as "twitter user chris" without needing to ask chris directly for their twitter password.  Of course, this makes no sense, because in our case, we possess the account, not our partner.  Likewise, once the login is complete, the user should end at our partner's site rather than our own.  We have nothing to use the oauth token for, because we don't perform any operation at the partner site aside from the login.

Yet here inside training_jump was OAuth.  The user hit training_jump; we redirected to the partner by IP address (!) with the OAuth request token, all the necessary user data, a callback URL (training_jump again); they duly redirected; we collected the response token and redirected the user back to the partner with that token as the parameter.  The end result is still kind of fragile, in that AFAIK, it only works in the first browser you sign up with.  If you log in with Firefox, then try it in Chrome, the latter gives you an error somewhere along the line instead of videos.

IIRC, research at the time indicated that there was no good PHP OAuth library, and/or the suitable libraries didn't implement the exact flavor of OAuth API that was being used by the Perl code.  I'm absolutely certain I considered replacing the Perl entirely, but I don't remember why I rejected PHP OAuth as a solution.

I couldn't simply continue using training_jump as-is, because the CGI module and PHP store their session data in different locations, in different formats.  The username in the PHP session wouldn't be accessible to pass through the authentication dance, and it was clearly inadvisable to modify training_jump to accept a username as a URL parameter.

Nowadays, training_jump has been succeeded by the cleverly named training_jump2, which actually reads request variables on stdin and produces an answer on stdout.  (The format of this text is much like LiveJournal's ancient API, from back when I had a LiveJournal client.  There was no convenient interchange format, as the Perl code didn't have JSON installed at the time and PHP didn't have XML.  "Lightweight" eats you again.)  The PHP training_jump manages the connection between server environment and training_jump2, and training_jump2 simply had its server environment replaced with communicating over pipes.

We're in the negotiation phase of moving to the provider's newer platform, which has a proper, encrypted SSO system.  training_jump2 is slated to become irrelevant, eventually.  In the meantime, it's the only bit of Perl CGI that never made the jump to mod_php.

sapphirepaw's Introduction to Pointers, version 2

I programmed in assembly for some time, using pointers without understanding what they were, or that they were called pointers.  When I finally got to learning C, the pointer syntax was downright inscrutable, but when I got it, suddenly all of C and all of assembler laid clear before me, all at once.  It was a beautiful thing.

I was reminded about this while reading this post from HN.  It inspired me to try explaining pointers from the opposite direction.  Instead of trying to teach pointers via C syntax, let me try to start with pointers outside of programming, then discuss them in relation to C and PHP.

Perception

They say you don't get a second chance to make a first impression, but that depends on who you are.  Apple seems to have managed a couple of major architecture transitions and their own Vista without too much ill will, yet Microsoft was practically crucified for Vista with no architecture transitions.

Fair warning: many links in this post lead to tvtropes.

Variable scope, require, and use

I ran into some interesting problems in Perl, which invoked more learning around the require/use mechanisms and how constants are interpreted.  In this post, I'll lay out some general terms about variable scoping, such as lexical scope, dynamic scope, the differences between them, and how they all interact in Perl.  And then I'll cover require and use with that foundation in place.

If you've been wondering about lexicals or closures, this is your post.  I've tried to lay things out more or less from basic principles, despite the verbosity of the approach, because this has taken me years to understand.  I started programming with Perl in 2000 and still learned a bit more about it today.  Yes, it's 2011 now.  Hopefully, you can read this and get it in less time.

Pagelib Retrospective

I have always been in love with magic: one of the first languages I taught myself once I was old enough to understand what I was doing was Perl, and it fit me like a skin-tight sci-fi spacesuit that we thought we were all going to wear in the future, back in the 1980's.  Steve Yegge's rant about Perl was entertaining, but I can't say as I identified with it much, since references weren't hard for me, nor do I have his problem of forgetting everything when I don't use the language for a while.  Maybe I don't drink enough.

Still, being in love with magic and DWIM doesn't mean that there aren't any hazards.  It's rather like fairy-tale magic, which can result in nasty, surprising, unexpected consequences if used without the utmost care.  This is a story about me exercising a complete and utter lack of care, utmost or otherwise.