AWS Auto Scaling, Load Balancers, and Availability Zones

We ran into an odd situation: one of our EC2 instances was attached to a target group, but “unused” on the load balancer, which ended up tripping the “not enough healthy hosts” alarm.

This ended up being my fault: I had removed a subnet (thus, availability zone) from the load balancer, but did not realize it was still associated with the Auto Scaling group.

When increasing capacity, AWS picked one of the four subnets available on the group, as instructed. The instance was cheerfully launched into that subnet, but it happened to be the one that wasn’t associated with the load balancer. Conversely, the fifth zone the load balancer used to have was pointless, because Auto Scaling would never launch an instance there.

(This all came about as part of an effort to reduce our availability zone footprint and improve colocation of resources. There are probably diminishing returns, and three zones to a region should be enough.)

tl;dr: Auto Scaling has a subnet configuration that is independent of any load balancer’s configuration. For successful operation, the load balancer must have every subnet that is assigned to any Auto Scaling group using that load balancer.

Fixing a Random ALB Alarm Failure

tl;dr: if an Auto Scaling Group’s capacity is updated on a schedule, the max instance lifetime is an exact number of days, and instances take a while to reach healthy state after launching… Auto Scaling can terminate running-but-healthy instances before new instances are ready to replace them.

I pushed our max instance lifetime 2 hours further out, so that the max-lifetime terminations happen well after scheduled launches.

Rebuilding Dependencies Frequently

Reddit asked:

how often do you rebuild your OS level dependencies?

In practice, around twice a month, due to accumulation of security updates. Sometimes bit-rot plays a role as well, but mostly, it’s the endless stream of updates. We want images to quickly launch into a secure state.

Waiting until the instance comes up to apply updates has two drawbacks: one, each new instance duplicates the work. Two, if there’s an update that requires a reboot, we’ve found through repeated, painful experience that there’s no way to reliably reboot our instances programatically.

IIRC though, that question was in the context of Docker containers, while our process is to precompile our app and its dependencies into a monolithic AMI that we deploy to instances in AWS. I like to think it’s basically the same, but I haven’t really used Docker much.

GCC options on EC2

AWS second-generation instances (such as my favorite, m3.medium) claim to launch with Xeon E5-2670 v2 processors, but sometimes the v1 variant.  These chips are Ivy Bridge and Sandy Bridge, respectively.  AFAICT, this means that AMIs with code compiled for Sandy Bridge should run on any second-generation instance, but may not necessarily run on previous-generation instances.  What little I can find online about previous-generation instances shows a mix of 90nm Opteron parts (having only up to SSE3) and Intel parts from Penryn-based Harpertown up through Westmere.

GCC changed their -march options in the recently-released version 4.9.  GCC 4.8, which notably shipped in Ubuntu 14.04 (Trusty Tahr), used some inscrutable aliases; the equivalent to 4.9's sandybridge is corei7-avx for instance.  Here's a table of the subset that's most interesting to me:

GCC 4.8	GCC 4.9
corei7	nehalem
–	westmere
corei7-avx	sandybridge
corei7-avx-i	ivybridge
corei7-avx2	haswell

Broadwell and Westmere are not explicitly supported in the older release.  Based on its definition in the gcc-4.9 sources, I believe the equivalent set of flags for Westmere in gcc 4.8 would be -march=corei7 -maes -mpclmul.  And naturally, the corei7-avx2 option for Haswell would be the best for targeting Broadwell.

SNS driven deployment (architectural overview)

3 Sep 2015: a note about how we're using php-fpm instead of suEXEC now has been published here.  Now back to your original post...

One of the problems I faced with getting Auto Scaling up and running was, "If I can't just ssh into the one live server and issue a git pull by hand, how do I push code out?"

Normal people would use capistrano, or Paste Deploy, or something, extended to access their auto scaling group to locate the hosts to use.  Or build on Heroku or Elastic Beanstalk to begin with.  I, however, am clearly not normal.

Bootstrapping an EC2 Instance, 2013 Edition

In the interests of "not implying old posts on my blog are still state-of-the-art", here's a quick overview of how I manage EC2 instances at work, modernized.

EC2 utilities vs. $AWS_CREDENTIAL_FILE

Most of the AWS command line tools accept their login credentials from a file named in the AWS_CREDENTIAL_FILE environment variable and formatted like so:

AWSAccessKeyId=AKIAEXAMPLE
AWSSecretKey=Base64FlavoredText/Example

The EC2 tools predate this scheme and still refuse to use it, preferring the credentials to be set directly in the environment.  I decided to over-engineer it and pull the EC2 environment variables from the file:

export AWS_ACCESS_KEY=$(grep '^AWSAccessKeyId' "$AWS_CREDENTIAL_FILE" | cut -d= -f2)
export AWS_SECRET_KEY=$(grep '^AWSSecretKey'   "$AWS_CREDENTIAL_FILE" | cut -d= -f2)

(Those will probably wrap on blogger; in code, they're two lines, each beginning with "export".)  Now I can put the credentials in one place, and they're available to all of the tools.

The Pointlessness of sudo's Default Run-As User

Amazon Linux ships with the default configuration*:

ec2-user ALL = NOPASSWD: ALL

Which means, ec2-user is allowed to run any command, without providing a password, while logged in from any machine.  But only as root—since the Runas_Spec is missing, the default of (root) is assumed.

This is entirely pointless because it also ships with the common PAM configuration, in which /etc/pam.d/su contains:

auth sufficient pam_rootok.so

So the game of Simon Says, in order to bypass the root-only sudo restriction so you can run as any user, password-free, without touching files in /etc in advance, becomes:

sudo su -s /bin/bash $TARGET_USER <pwn.sh

Normally, su uses the shell for the user as listed in /etc/passwd, but if we're interested in a /sbin/nologin account, then we can set any other shell listed in /etc/shells with the -s flag.

When you give any account root access, they probably have the whole machine.  I'm not sure what sudo was hoping to accomplish by "limiting" the default Runas_Spec to root.

* It also ships with Defaults requiretty which means you actually need someone to allocate you a controlling terminal for sudo to work, even though ec2-user doesn't need a password, and visiblepw is disabled by default.

AWS: as-* command parameters

It turns out that the auto scaling commands like as-create-auto-scaling-group are thin wrappers around the AWS SDK for Java; you can read all the commands placed in /opt/aws/apitools/as/bin but they eventually just invoke Java with a classpath set and invoke com.amazon.webservices.Cli or so.

Thus, the CLI commands are essentially documented by the Java API Reference.  In particular, the as-* family reflects operations on com.amazonaws.services.autoscaling which consume data types in com.amazonaws.services.autoscaling.model.

What I was specifically looking for was documentation on the --health-check and --grace-period operations, and ....model.CreateAutoScalingGroupRequest finally has me covered.  Health check can be either 'EC2' or 'ELB', to select either the standard (and presumably default) EC2 "is the host hardware alive? can we ping the vm?" instance checks; or the "can we access this URL on the instance's http server?" health check configured in setting up an elastic load balancer.  Respectively.

Likewise, --grace-period is the time delay between starting up a new instance, and when auto scaling is allowed to start asking for its health.

Troubleshooting cloud-init on Amazon Linux

cloud-init drops files as it works under /var/lib/cloud/data – you'll find your user-data.txt there, and if it was processed as an include, you'll also have user-data.txt.i.

If you're using #include to run a file from s3 and it wasn't public (cloud-init has no support yet for IAM Roles, nor special handling for S3), then user-data.txt.i will contain some XML indicating "Access Denied".  Otherwise, you should see your included script wrapped in an email-ish structure, and an unwrapped (and executable) version under /var/lib/cloud/data/scripts.

Update 23 Aug: Per this thread, user data is run once per instance by default, so you can't test it by simple reboots unless you have edited /etc/init.d/cloud-init-user-scripts to change once-per-instance to always.  Or use your first boot to set up an init script for subsequent boots.  But this doesn't apply if you build an AMI—see the 1 Oct/8 Oct update below for notes on that.

Update 2 Sep: I ended up dropping an upstart job into /etc/init/sapphirepaw-boot from my script; where the user data is #include\nhttp://example.s3....com/stage1.pl and the upstart job is a task script that runs curl http://example.s3....com/stage1.pl | perl.  stage1.pl is public, and knows how to get the IAM Role credentials from the instance data, then use them to pull the private stage2.pl.  That, in turn, actually knows how to read the EC2 tags for the instance and customize it accordingly.  Finally, stage2.pl ends up acting as the interpreter for scripts packed into role-foo.zip (some of them install further configuration files and such, so a zip is a nice, atomic unit to carry them all in).

Note that I have just duct-taped together my own ad-hoc, poorly specified clone of chef or puppet.  A smarter approach would have been to pack an AMI with one of those out-of-the-box, then have stage2.pl fetch the relevant recipe and use chef/puppet to apply it.  Another possibility would be creating an AMI per role, with no changes necessary on boot (aside, perhaps, from `git pull`) to minimize launch time.  That would prevent individual instances from serving multiple roles, but that could be a good thing at scale.

But now I'm just rambling; go forth and become awesome.

Update 1 Oct, 8 Oct: To cloud-init, "once per instance" means once per instance-id.  Building an AMI caches the initial boot script, and instances started from that AMI run the cached script, oblivious to whether the original has been updated in S3.  My scripts now actively destroy cloud-init's cached data.  Also, "the upstart job" I mentioned was replaced by a SysV style script because the SysV script I wanted to depend on is invisible to upstart: rc doesn't emit individual service events, only runlevel changes.

UI Consistency: vim and emacs

I was adding a little more vim customization this morning, to work around the fact that the vim setup on the ec2 instance is completely silly about editing php code (php.vim includes html.vim, which acts like it owns the entire file, and thus calls set matchpairs+=<:> and set comments=…; this is okay for actual html, but php.vim doesn’t restore these settings, so you get error bells when writing $foo->bar() from the unbalanced ‘>’, and you can’t format your PHP comments anymore because they’re completely unrecognized.)

Anyway, this led to the realization that my vim setup, as large as it is growing, is built on a large batch of external code, which is in turn dependent on both vim version and the distribution.  Due to subtle changes to hundreds of defaults—especially complex values like the aforementioned matchpairs and comments settings—it’s impractical to predict how a given vimrc will behave in a random environment.

Thanks to site-lisp, I am guessing that emacs has essentially the same issue: your initialization may interact in with site-lisp code to produce per-host variation in behavior.

For something I’m going to spend my day in, that’s a painful situation.  The variation interrupts the flow of expected results, pushing the editor back up into my consciousness.  But would I want an editor I couldn’t customize?

Notes on Sharing a Unix Account

After accidentally breaking an ec2 instance while trying to set up a separate user account with admin rights, I decided to keep using ec2-user and set it up to coexist with any other people who logged in on the account.  I don't expect my boss wants all my shell/vim/etc. customizations.

To achieve this, I took advantage of openssh's environment support (which required enabling PermitUserEnvironment yes in /etc/ssh/sshd_config) to set a variable when I log into the server with my key pair:

environment="VUSER=sapphirepaw" ssh-rsa ...

Next, a one-line change to ~/.bashrc:

[ -n "$VUSER" -a -r "$HOME/.$VUSER/bashrc" ] && . "$HOME/.$VUSER/bashrc"

That newly-sourced bashrc then takes care of setting up the rest of the world, with code like:

mydir="$HOME/.$VUSER"
export SCREENRC="$mydir/screenrc"
export VIMINIT="source $mydir/ec2init.vim"
alias st="svn status -q"

Notice that vim doesn't support any sort of "find the vimrc here" environment variable, but it does allow for arbitrary Ex commands to run, so I used that instead.  (Hat tip to this helpful message.)  ec2init.vim then reads:

let s:rdir="/home/ec2-user/.sapphirepaw"
let &rtp=s:rdir . "/vimfiles," . &rtp . "," . s:rdir . "/vimfiles/after"
exec "source " . s:rdir . "/vimrc"

This expands all the variables soon enough to be useful, and also means that if I ever move/reconfigure the root directory name, I will have only one place to change it in vim.  And from there, all my settings are loaded.  Life is good again.

Setting Everything on Fire

I created a new user, gave them wheel group, and in case I needed another admin user, added %wheel to sudoers through visudo.  Then, I was trying to do more stuff, and...

[sudo] password for ec2-user: _

Wait.  What?  Not only does ec2-user have no password, but I didn't change its NOPASSWD line in sudoers.

It turns out that ec2-user is also in group wheel, and when confronted with the two permission sets, sudo did what I didn't mean: applied the %wheel rule and started requiring passwords for ec2-user.  Of course su was no help either: root likewise has no password set, because you have sudo as ec2-user....