Tuesday, October 18, 2011

Character Sets, Encodings, MySQL, and your data

This post is a companion to Perl and Unicode in Brief, an attempt to cover similar ground more concisely.  And this post is a revised version of the one you're currently reading.

I'm currently moving data from a (relatively old now) MySQL 5.0 server into Amazon RDS.  I've been here before, when I was moving data from MySQL 4.x into 5.0 and mangling character sets.  This time, I want to make 100% sure everything comes across with maximum fidelity, and also get the character encoding as stored to be labeled correctly in MySQL.

First, a quick definition or two:
  • Character Set: a specific table to translate between characters and numbers.  Example: ASCII defines characters for numbers 0-127; "A" is 65.  This can also be described as "a set of characters, and their corresponding representation inside the computer."
  • Character Encoding: a means of "packing" numbers from the character set into a container.  Example: UTF-8.  The Unicode character 0x2013 becomes 0xE2,80,99. The "E" signifies "Part 1 of 3", and part of the remaining bytes simply indicate "Continued"; the 0x2013 is then divided up to fit in the parts of the bytes that aren't indicating their "Part 1" or "Continued" status.  In the specific case of UTF-8, the encoding is designed so that the ASCII range 0-127 (0x00-7F) is encoded without change: a leading 0-7 means "Part 1 of 1".
  • 8-bit character encoding: In older, simpler days, character sets defined only as many characters as could fit in 8 bits, and defined the encoding as simply the numbers.  Character number 181 would encode as a byte (8 bits) with value 181.
  • A character encoding implies the associated character set, because the encoding defines how numbers in its character set become individual bytes.  How characters in other sets would be encoded is left undefined and basically impossible.
This last point is why MySQL lets you set "character sets" to UTF-8, though the latter is an encoding.


Tuesday, October 11, 2011

iPad vs. Tablet PC

One of them succumbed to death by risk-aversion.

One of them couldn't let go of the tether and fly.

I think Linus said the same of svn: paraphrased, "If you're trying to make 'a better CVS' then you have already lost, because CVS is too broken to fix."

Hey, sapphirepaw: make sure what you do is good on its own, not "an X only different".

Saturday, October 8, 2011

Steve Jobs

I'm getting old: if I were to pass on at the same age Jobs did, my life would be more than half over already.

What separates me from Jobs?  There's the matter of leverage, where he could take his vision and coordinate the prototyping and development of it, into the iPod, the iPhone, the Macbook Air, the iPad.  There's also the matter of having vision.

In 2006 or so, I beheld my first iPod in real life, an old (FireWire based) model with a physical click-wheel.  In 2008 I picked up a different, small MP3 player and for the first time, immediately noticed the limitations of digital control.  Without having handled the iPod and getting a feel for the analog response of the wheel, I probably wouldn't have given the buttons a second thought.  Do you want to scroll on the generic?  Click-click-click-click.  Or click-and-hold, guess at how long you need to go (since the screen is slow enough to be unreadable at this scrolling speed, and they don't slow updates to compensate), and release.

The point here is, Jobs saw humans as inherently analog, and adapted all of his machines to analog control.  It's a simple thing, but Jobs was apparently devoted to HCI.  The "vision" simply falls out of that.

It's not like the limitations of digital control weren't apparent in the 1980s.  Compare Rad Racer to a real car's steering wheel.  Anyone focused on "how it feels" could have been Jobs back then, inventing 2010 in the 16-bit era instead of carrying 8-bit paradigms through the 1990s.

In contrast, I seem to lack vision because I'm busy implementing arbitrarily complex business rules at work, and staying away from the bleeding edge of gadgetry.  I'm not in the consumer space; I'm not taking any research toward the consumer space; and I'm not thinking about what's next for it, either (at least, not beyond what turns out to actually be the next thing*.)  But, I'm also having little impact on the wider world, writing code that never leaves the house.  It's important, but after I am gone, will these be the best years of my life?  Will I think college was the best time of my life, forever?

I think it's time to put my free time to better use and do something instead of watching the world slowly develop towards Jobs' vision on its own.


* I have a dead draft which discusses the crazy idea of "having a set-top box inside the remote" in 2006 or so.  It then points out that h.264-over-wifi ought to handle the bandwidth to do exactly that from your iPhone now.  It starts fleshing out what would be necessary to make it happen, then abruptly ends with a note: "Two days after I started writing this, Apple announced AirPlay."

Thursday, October 6, 2011

Setting Everything on Fire

I created a new user, gave them wheel group, and in case I needed another admin user, added %wheel to sudoers through visudo.  Then, I was trying to do more stuff, and...

[sudo] password for ec2-user: _

Wait.  What?  Not only does ec2-user have no password, but I didn't change its NOPASSWD line in sudoers.

It turns out that ec2-user is also in group wheel, and when confronted with the two permission sets, sudo did what I didn't mean: applied the %wheel rule and started requiring passwords for ec2-user.  Of course su was no help either: root likewise has no password set, because you have sudo as ec2-user....

Friday, September 9, 2011

War Story: The Training Jump

One of the first things I did at my current job was to rewrite a Perl/CGI (the module, and the actual cgi-script execution model) site into PHP.  Part of this site implemented a single-signon (SSO) system for a partner site that hosts our training videos.  Clicking the link led to the innocuously-named "training_jump.pl" CGI script.

The goal in life of the training_jump is to redirect a user to the partner site, passing along the username and email address.  The partner site creates the user if necessary, starts a session for them on its server, and ultimately displays the actual training content.

Inside training_jump is an innocuous-looking "use OAuth::Lite;" line.  I didn't know what OAuth was at the time, so of course I went and looked it up: OAuth is designed to let a site like ExampleMashup authenticate someone as "twitter user chris" without needing to ask chris directly for their twitter password.  Of course, this makes no sense, because in our case, we possess the account, not our partner.  Likewise, once the login is complete, the user should end at our partner's site rather than our own.  We have nothing to use the oauth token for, because we don't perform any operation at the partner site aside from the login.

Yet here inside training_jump was OAuth.  The user hit training_jump; we redirected to the partner by IP address (!) with the OAuth request token, all the necessary user data, a callback URL (training_jump again); they duly redirected; we collected the response token and redirected the user back to the partner with that token as the parameter.  The end result is still kind of fragile, in that AFAIK, it only works in the first browser you sign up with.  If you log in with Firefox, then try it in Chrome, the latter gives you an error somewhere along the line instead of videos.

IIRC, research at the time indicated that there was no good PHP OAuth library, and/or the suitable libraries didn't implement the exact flavor of OAuth API that was being used by the Perl code.  I'm absolutely certain I considered replacing the Perl entirely, but I don't remember why I rejected PHP OAuth as a solution.

I couldn't simply continue using training_jump as-is, because the CGI module and PHP store their session data in different locations, in different formats.  The username in the PHP session wouldn't be accessible to pass through the authentication dance, and it was clearly inadvisable to modify training_jump to accept a username as a URL parameter.

Nowadays, training_jump has been succeeded by the cleverly named training_jump2, which actually reads request variables on stdin and produces an answer on stdout.  (The format of this text is much like LiveJournal's ancient API, from back when I had a LiveJournal client.  There was no convenient interchange format, as the Perl code didn't have JSON installed at the time and PHP didn't have XML.  "Lightweight" eats you again.)  The PHP training_jump manages the connection between server environment and training_jump2, and training_jump2 simply had its server environment replaced with communicating over pipes.

We're in the negotiation phase of moving to the provider's newer platform, which has a proper, encrypted SSO system.  training_jump2 is slated to become irrelevant, eventually.  In the meantime, it's the only bit of Perl CGI that never made the jump to mod_php.

Sunday, September 4, 2011

sapphirepaw's Introduction to Pointers, version 2

I programmed in assembly for some time, using pointers without understanding what they were, or that they were called pointers.  When I finally got to learning C, the pointer syntax was downright inscrutable, but when I got it, suddenly all of C and all of assembler laid clear before me, all at once.  It was a beautiful thing.

I was reminded about this while reading this post from HN.  It inspired me to try explaining pointers from the opposite direction.  Instead of trying to teach pointers via C syntax, let me try to start with pointers outside of programming, then discuss them in relation to C and PHP.

Monday, August 8, 2011

What Would "Better" Be? PDF Reader Edition

I opened up a PDF in the default Gnome PDF reader last night, and it was once again a terrible experience.  It opened with the zoom set to "fit page width", and the scrolling set to continuous.  There's no concept of a persistent user preference, or user preferences that override the document preferences.

Then I got to considering the underlying reasons why I didn't like the default display.