Troubleshooting cloud-init on Amazon Linux

cloud-init drops files as it works under /var/lib/cloud/data – you'll find your user-data.txt there, and if it was processed as an include, you'll also have user-data.txt.i.

If you're using #include to run a file from s3 and it wasn't public (cloud-init has no support yet for IAM Roles, nor special handling for S3), then user-data.txt.i will contain some XML indicating "Access Denied".  Otherwise, you should see your included script wrapped in an email-ish structure, and an unwrapped (and executable) version under /var/lib/cloud/data/scripts.

Update 23 Aug: Per this thread, user data is run once per instance by default, so you can't test it by simple reboots unless you have edited /etc/init.d/cloud-init-user-scripts to change once-per-instance to always.  Or use your first boot to set up an init script for subsequent boots.  But this doesn't apply if you build an AMI—see the 1 Oct/8 Oct update below for notes on that.

Update 2 Sep: I ended up dropping an upstart job into /etc/init/sapphirepaw-boot from my script; where the user data is #include\nhttp://example.s3....com/stage1.pl and the upstart job is a task script that runs curl http://example.s3....com/stage1.pl | perl.  stage1.pl is public, and knows how to get the IAM Role credentials from the instance data, then use them to pull the private stage2.pl.  That, in turn, actually knows how to read the EC2 tags for the instance and customize it accordingly.  Finally, stage2.pl ends up acting as the interpreter for scripts packed into role-foo.zip (some of them install further configuration files and such, so a zip is a nice, atomic unit to carry them all in).

Note that I have just duct-taped together my own ad-hoc, poorly specified clone of chef or puppet.  A smarter approach would have been to pack an AMI with one of those out-of-the-box, then have stage2.pl fetch the relevant recipe and use chef/puppet to apply it.  Another possibility would be creating an AMI per role, with no changes necessary on boot (aside, perhaps, from `git pull`) to minimize launch time.  That would prevent individual instances from serving multiple roles, but that could be a good thing at scale.

But now I'm just rambling; go forth and become awesome.

Update 1 Oct, 8 Oct: To cloud-init, "once per instance" means once per instance-id.  Building an AMI caches the initial boot script, and instances started from that AMI run the cached script, oblivious to whether the original has been updated in S3.  My scripts now actively destroy cloud-init's cached data.  Also, "the upstart job" I mentioned was replaced by a SysV style script because the SysV script I wanted to depend on is invisible to upstart: rc doesn't emit individual service events, only runlevel changes.

AmazonS3 'headers' and 'meta' options in the PHP SDK

When you're using create_object, or several other methods of the AmazonS3 class, the $opts parameter often allows for both headers and meta keys.  Although headers is documented as "HTTP headers to send along with the request", it turns out that they are returned with the object (i.e. in the response) when that object is requested from S3.  In contrast, keys in meta are canonicalized, prepended with x-amz-meta-, and returned that way.

That is, if you want to upload filenames like "$id-$md5.pdf" but deliver them to the user as "ContosoUserAgreement.pdf" in the Save-As dialog, then headers should contain a Content-Disposition key with a value of attachment; filename="ContosoUserAgreement.pdf".

If you put it in meta instead, then the HTTP headers on retrieval will contain a x-amz-meta-content-disposition header instead, which will not be honored by the browser.

I found all this out by uploading something like 12,000 files with the wrong metadata.  I then wrote a script to fix it, which ran straight into the problem that update_object doesn't work, so you have to use copy_object instead.  Note that when using copy_object with 'metadataDirective' => 'REPLACE', you need to specify all the metadata you want, because it does what it says: it deletes all old metadata before adding the new metadata from the copy_object request, so that only the new metadata exists afterwards.

Private Streaming with CloudFront: A Guide

Update, 1 Oct 2012: This post is largely obsolete, as Amazon recently added private streaming support to the CloudFront section of the AWS Console.  The original post follows.

I'll just assume you're aware of the IaaS offering known as Amazon Web Services, AWS.  CloudFront is a CDN in the AWS micropayments-as-you-go style, which offers the ability to serve non-public content stored in S3.  This is a compendium of the things I learned setting up a private streaming distribution for use with PHP.

This is going to be fairly low-level, since I like to drink deeply of the systems I'm working with.  I don't think AWS works smoothly enough yet that you can put the API on the "it's magic" side of the line.

Renaming buckets on S3

A technical note, since the search engines of the internet don't seem to have noticed: Amazon's S3 management console lets you cut and paste files now (including whole folders).  So the process to "rename" an S3 bucket is simply:

1. Create the new bucket with the desired name.
2. Go to the old bucket and select all files: click the first and then shift+click the last.
3. Above the file listing, in the button row, is one marked "Actions", which opens a menu that includes "Cut" and "Copy".  Pick one.
4. Go to the new bucket, click Actions, and Paste your files.

Done.  No 3rd-party software required.

Why would anyone want to rename a bucket?  In our case, we created a StudlyCapsStyle bucket, which can't be used with CloudFront's dns-compatible-style.


In double-checking this post for accuracy, I noticed that Cut/Copy are available on the right-click menu for a single selection, but not the multi-select.  Weird.