An Exercise in Optimizing PHP

Last winter, I was optimizing a PHP reporting script for no real reason besides practicing optimization.

Radical Clojure

Apparently, last month I missed Yegge's post and followup regarding software liberals and conservatives.

One of the things that caught my eye, that's clearly a Nerd Trap but so powerful that I need to answer anyway, is this little quote:

But the reality is that Clojure is quite conservative.  As far as I'm concerned, if you can't set the value of a memory variable without writing code to ask for a transaction, it's... it's... all I can think of is Morgan Freeman in the Shawshank Redemption, working at the grocery store, complaining "Forty years I been asking permission to piss. I can't squeeze a drop without say-so."

Emphasis and vivid FUD in original.

There's just one missing word, though, that makes all the difference:

You can't set the value of a shared memory variable outside of a transaction.

Shared.  Global.  Possibly still in use by someone else.

Clojure is utterly pointless without understanding that time is explicit and everything is an immutable value (unless you have a Java native thing.)  Values last, unmodified, until a reader asks for an update, and so a writer must be forbidden from modifying (destroying) that memory.  There's a whole paradigm hiding there, which you can see again if you look at Datomic.  Clojure without immutable values might as well be JRuby.

The other trick up Clojure's sleeve is that 'transaction' implies something rather more heavyweight outside of Clojure than in it.  Again, because immutability is pervasive, Clojure's transactions don't have to do read tracking.  When anyone reads an old value, it's going to stay what they read.  It's like RCU, conceptually, except that every read is protected and nobody needs to copy because the messy details are taken care of on write.

And if it's not actually shared?  Then you should use a var, which only one thread can see, and therefore doesn't need to be updated in a transaction!

Clojure is an interesting language.  I'd still recommend it.


Added 26 Sept: I think that liberal/conservative as applied by Yegge divide the languages according to how much a language insists on its own philosophy.  Things like Erlang and Clojure get ranked "conservative" right in with Pascal because they have Ways to Do Things, even if they are progressive ideas (functional and parallel/async are strongly encouraged.)  Then Perl is more "liberal" because it's equally well-suited to OO, procedural, functional, and concatenative programming, i.e. just barely, and if it didn't have regexes as syntax, it would be long dead already.  Python is more conservative than Ruby despite sharing characteristics because "there should be one—and preferably only one—obvious way to do it."  That's a clear Conservative value, right in the middle of the Zen of Python.

In any case, the insight that everyone thinks they're liberal is accurate.  Upon some thinking, which is hard and time-consuming and therefore not generally applied on the Internet, "conservative" is a fair enough label for my tendencies at this point in time—because I got burned throwing around too much fire, then came off that job a few months later to take positions in more conservative cultures.  But I've never thought of myself as a stodgy dinosaur programmer.

AWS: as-* command parameters

It turns out that the auto scaling commands like as-create-auto-scaling-group are thin wrappers around the AWS SDK for Java; you can read all the commands placed in /opt/aws/apitools/as/bin but they eventually just invoke Java with a classpath set and invoke com.amazon.webservices.Cli or so.

Thus, the CLI commands are essentially documented by the Java API Reference.  In particular, the as-* family reflects operations on com.amazonaws.services.autoscaling which consume data types in com.amazonaws.services.autoscaling.model.

What I was specifically looking for was documentation on the --health-check and --grace-period operations, and ....model.CreateAutoScalingGroupRequest finally has me covered.  Health check can be either 'EC2' or 'ELB', to select either the standard (and presumably default) EC2 "is the host hardware alive? can we ping the vm?" instance checks; or the "can we access this URL on the instance's http server?" health check configured in setting up an elastic load balancer.  Respectively.

Likewise, --grace-period is the time delay between starting up a new instance, and when auto scaling is allowed to start asking for its health.

HTTP(S) Endpoints and SNS

If you subscribe an HTTP endpoint to SNS, the first thing AWS does is send a subscription confirmation to that endpoint, which should be ready to handle it.  (If not, you can manually re-subscribe it through the console when the service is up, and AWS re-sends the confirmation.)  It turns out that the actual messages as exchanged with endpoints are defined in the Getting Started Guide, Appendix D; not the API reference!  The API reference only contains the administrative calls that can be made to the API, such as ConfirmSubscription.

It also turns out that the signature of the message sent to the endpoint is message-specific; Notification messages use a different set of fields to be signed than SubscriptionConfirmation and UnsubscriptionConfirmation.  Again, the details (and examples) of this are not in the API documentation, but Appendix D of the Getting Started Guide.

Security is Hard

"We constantly find '0days' as part of pentests and use them against our customers. Just the other day, we used an 0day SQL injection bug in [popular manufacturer's name deleted] firewall to break into a customer."

—Rob Graham via Ars Technica

A firewall.  Had an SQL injection bug.

A firewall.  A security product.

With the most basic of web security bugs embedded.

Obviously, being a black hat these days is like shooting fish in a barrel.  With a cannon.

Think of the Olden Days!

My first Linux machine had 128 MB of RAM.  The bzip2 warnings that you needed at least 4 MB of RAM to decompress any archive seemed obsolete at the time (even our then-4.5-year-old budget computer had shipped with twice that for Windows 95 RTM) and downright comical now that I have 4,096 MB at my disposal.

I was compressing something the other day with xz, which was taking forever, so I opened up top and only one core was under heavy use.  Naturally.  In the man page is a -T<threads> option... that isn't implemented because won't someone think of the memory!

OK, sure.  It appears to be xz -6 based on the resident 93 MB; with four cores, it's still under 10% of RAM.  The only ways it could come close to hurting are to run at xz -9 which consumes 8 times the memory and would seriously undermine the "reasonable speed" goal even with four threads; to run with 44 cores but not more RAM; or to run it on a dual-thread system on 256 MB.  The concern seems to be nearly obsolete already... will we be reading the man page in 2024 and finding that there are no threads because they use memory?

The point of this little rant is this: someone has a bigger, better system than you.  Either one they paid a lot of money for and would like to see a return on investment, or one they got further into the future than yours.  If you tuned everything to work on your system today, left or right shift by 1, then you have a small window of adaptability that will soon be obsolete.  Especially pertinent here is that parallelizing compression does not add requirements to the decompressor.  A single-thread system will unpack just as well, it just takes longer; unlike the choice of per-thread memory which forces the decompressor to allocate enough to handle the compression settings.

(Like gzip and bzip2, there exist some parallel xz utilities.  But only pbzip2 has made it into the repository.)

Troubleshooting cloud-init on Amazon Linux

cloud-init drops files as it works under /var/lib/cloud/data – you'll find your user-data.txt there, and if it was processed as an include, you'll also have user-data.txt.i.

If you're using #include to run a file from s3 and it wasn't public (cloud-init has no support yet for IAM Roles, nor special handling for S3), then user-data.txt.i will contain some XML indicating "Access Denied".  Otherwise, you should see your included script wrapped in an email-ish structure, and an unwrapped (and executable) version under /var/lib/cloud/data/scripts.

Update 23 Aug: Per this thread, user data is run once per instance by default, so you can't test it by simple reboots unless you have edited /etc/init.d/cloud-init-user-scripts to change once-per-instance to always.  Or use your first boot to set up an init script for subsequent boots.  But this doesn't apply if you build an AMI—see the 1 Oct/8 Oct update below for notes on that.

Update 2 Sep: I ended up dropping an upstart job into /etc/init/sapphirepaw-boot from my script; where the user data is #include\nhttp://example.s3....com/stage1.pl and the upstart job is a task script that runs curl http://example.s3....com/stage1.pl | perl.  stage1.pl is public, and knows how to get the IAM Role credentials from the instance data, then use them to pull the private stage2.pl.  That, in turn, actually knows how to read the EC2 tags for the instance and customize it accordingly.  Finally, stage2.pl ends up acting as the interpreter for scripts packed into role-foo.zip (some of them install further configuration files and such, so a zip is a nice, atomic unit to carry them all in).

Note that I have just duct-taped together my own ad-hoc, poorly specified clone of chef or puppet.  A smarter approach would have been to pack an AMI with one of those out-of-the-box, then have stage2.pl fetch the relevant recipe and use chef/puppet to apply it.  Another possibility would be creating an AMI per role, with no changes necessary on boot (aside, perhaps, from `git pull`) to minimize launch time.  That would prevent individual instances from serving multiple roles, but that could be a good thing at scale.

But now I'm just rambling; go forth and become awesome.

Update 1 Oct, 8 Oct: To cloud-init, "once per instance" means once per instance-id.  Building an AMI caches the initial boot script, and instances started from that AMI run the cached script, oblivious to whether the original has been updated in S3.  My scripts now actively destroy cloud-init's cached data.  Also, "the upstart job" I mentioned was replaced by a SysV style script because the SysV script I wanted to depend on is invisible to upstart: rc doesn't emit individual service events, only runlevel changes.