Server Management: We Should Be Able to Recreate Pets, Too

For anyone not familiar with the analogy, there are two types of server or private instance: “cattle” exist en masse and have names like atl-prod3-24.  If one has problems, the first solution is replacement.  “Pets,” on the other hand, are unique instances that matter a lot, often with whimsical names.  Problems with pets are approached by working to cure what ails them.

Once upon a time, we had a single pet server which was our bastion host, cron job runner, and SFTP file-exchange point with third parties.  We split those roles apart; the cron jobs get access to the SFTP files (for input or output) by sharing an EFS mount.  The SFTP machine everyone uses can be firewalled off from the rest of our codebase and AWS resources.  (Not that I expected 0days, but there was a notable close call a couple of years after this split was made.  Further detailed in Half a Second.)

After a couple of Debian upgrades, the SFTP host had some network issues.  Investigation found a post putting the blame on ‘the image builder’ and telling the person with the question (my same question, unfortunately) that because of that, they were on their own.

It seems likely, then, that we’ll want to rebuild this pet from scratch at some point.  I fixed everything this time, but who knows what might happen on the next upgrade?  It seems like it would be far more stable if we could re-customize a clean base image, instead of doing a series of in-place upgrades.  This is especially true if each Debian upgrade isn’t prepared for anything that is specific to the Debian AWS cloud images.

A long aside about the networking problems I faced follows.

Lazy Init Only Scatters Latency

People report on the Internet that their “Hello World” NodeJS functions in AWS Lambda initialize in 100–150 ms on a cold start, while our real functions spend 1000–1700 ms.  Naturally, I went looking for ways to optimize, but what I found wasn’t exactly clear-cut.

A popular option is for the function to handle multiple events, choosing internal processing based on the shape of the event.  Maybe a large fraction of their events don’t need to handle a PDF, so they can skip loading the PDF library up front.

Unfortunately, my needs are for a function which handles just two events (git push and “EC2 instance state changed” events) and in both cases, the code needs to contact AWS services:

1. git push will always fetch the chat token from Secrets Manager
2. Instance-state changes track run-time using DynamoDB (and may need the chat token)

If I push enough of the AWS SDK initialization off to run time, all I’m actually doing is pushing the delay over to run time.  I would need to be able to have a relatively high-frequency request that didn’t use the AWS SDK in order to separate the SDK latency from average processing.  Even then, it still wouldn’t work if the first request needed AWS!

Nonetheless, I did the experiments, and as far as I can tell, lazy init does exactly what I predicted: causes more run time and less init time, for a similar total, on a cold start.  Feeding it 33% more RAM+CPU lets it run 22% faster and consume more memory total, which suggests that it’s doing less garbage collection.  It would be nice to have GC stats, then!

Warm-start execution, for what it’s worth, is 10% of the overall runtime of a cold start, or what was about 25% of the cold-start run time before any changes were made.  Either GC or CPU throttling is hampering cold-start performance.

(I’d also love to know how AWS is doing the bin-packing in the background.  Do they allocate “bucket sizes” and put 257–512 MB functions into 512 MB “reservations,” or do they actually try to fill hosts precisely?  Actually, it’s probably oversubscribed, but by how much?  “Run code without thinking about servers,” they said, and I replied, “Don’t tell me what to do!”)

The experiment I didn’t do was whether using esbuild to publish a 1.60 MB single-file CommonJS bundle, instead of a 0.01 MB zip with ESM modules, will do anything.  Most sources say that keeping the file size down is the number one concern for init speed.  At this point, I think if I wanted more speed, I would port to Go.

Notes on an ECS Deployment

In order to try FrankenPHP and increase service isolation, we decided to split our API service off of our monolithic EC2 instances.  (The instances carry several applications side-by-side with PHP-FPM, and use Apache to route to the applications based on the Host header.  Each app is not supposed to meddle in the neighbor’s affairs, but there’s no technical barrier there.)

I finally got a working deployment, and I learned a lot along the way.  The documentation was a bit scattered, and searching for the error messages nearly useless, so I wanted to pull all of the things that tripped me up together into a single post.  It’s the Swiss Cheese Model, except that everything has to line up for the process to succeed, rather than fail.

1. Networking problems
2. ‘Force Redeployment’ is the normal course of operation
3. The health check is not optional
4. Logs are obscured by default
5. The ports have to be correct (Podman vs. build args)
6. The VPC Endpoint for an API Gateway “Private API” is not optional
7. There are many moving parts

Let’s take a deeper look.

HTTPS in a Caddyfile, and also Not Doing That

I ended up putting together a couple of Caddyfiles for the Caddy server.  The project really wants users to choose automatic HTTPS, but it wasn’t a good fit for using with a load balancer and auto-scaling.  Surprisingly, it is willing to contact the production Let’s Encrypt API even without an email address.

Anyway, I have a few pieces here to cover:

• The easiest way to ignore HTTPS, and returning a fixed error message on HTTP
• Mysterious error messages, relating to the way Caddy chooses certificates and configuration blocks
• Using the tls directive to get self-signed certificates for a public name, for using HTTPS between Caddy and a load balancer which does not validate certificates
• How to use trusted_proxies—and a bit more trickery—so that, using HTTP behind a load balancer, PHP knows the client’s connection is HTTPS
• How to compose multiple URL mappings, and using a front controller for PHP apps
• One small note on FrankenPHP (added: 2025-11-26)
• Closing Notes

Aside from the above links, Caddy provides more conceptual documentation at the Automatic HTTPS article.

My end goal is to get FrankenPHP running, but as it is built on Caddy, I wanted to understand that part before moving ahead.

AWS Auto Scaling, Load Balancers, and Availability Zones

We ran into an odd situation: one of our EC2 instances was attached to a target group, but “unused” on the load balancer, which ended up tripping the “not enough healthy hosts” alarm.

This ended up being my fault: I had removed a subnet (thus, availability zone) from the load balancer, but did not realize it was still associated with the Auto Scaling group.

When increasing capacity, AWS picked one of the four subnets available on the group, as instructed. The instance was cheerfully launched into that subnet, but it happened to be the one that wasn’t associated with the load balancer. Conversely, the fifth zone the load balancer used to have was pointless, because Auto Scaling would never launch an instance there.

(This all came about as part of an effort to reduce our availability zone footprint and improve colocation of resources. There are probably diminishing returns, and three zones to a region should be enough.)

tl;dr: Auto Scaling has a subnet configuration that is independent of any load balancer’s configuration. For successful operation, the load balancer must have every subnet that is assigned to any Auto Scaling group using that load balancer.

We Finally Put Up a WAF

Someone sent an awful lot of requests at a system for long enough that management noticed the issue. Working with the responsible admin, I ended up proposing AWS WAF “to see what would happen.”

What happened: WAF blocked 10,000 requests per minute, and someone got the message.  This released the pressure on the DynamoDB table behind the system, allowing it to jump straight from max to min capacity (1/16th) after fifteen minutes.

It seems some automated vulnerability scanner had gotten into an infinite loop.  There were a lot of repeated URLs in the access logs, like it wasn’t clearing pages from its queue if they got an OK response but unexpected data.  The reason “everything” returns OK is because an unknown URL (outside of a specific static-content prefix) returns a page with the React app root, and lets JavaScript worry about rendering whatever should be there.

I went ahead and put the same WAF on my systems, promptly breaking them.  Meanwhile, our automated testing provider started reporting failures, with every request from the original system returning Forbidden.

The testing platform… is a bot.  I had to write them an exception.

Turning my attention back to my systems, I put together a second WAF so I could have different policies.  My system includes an API or two, so I needed to allow HTTP Libraries and non-browsers. I linked in the exception for the testing platform as well.  Things went much more smoothly after that.

I know that the WAF is fundamentally “enumerating badness,” but it is clearly better than zero filtering.  It is also much less effort and risk, which is why this sort of thing persists.

Separate Components Allow “Least Privilege”

At work, our AWS image-building process exists as a set of scripts that are run manually, in-order, if everything looks okay to the human at the terminal.

The problem is, the image auto-installs security updates once it launches; if enough accumulate, instances launched from the image start failing to become ready for traffic before the timeout is up.  The system is guaranteed to decay if I don’t periodically build a fresh image, manually.

One reason it isn’t automated (say, rebuilding itself weekly using AWS infrastructure) is that the process takes more permissions than normal business operations.  It must be able to run and terminate instances, tag things, and reconfigure what image is launched by Auto Scaling.  Those would be scary permissions to give to any instances, which generally only have write access to specific data storage locations.

However, if the various steps (build, test, update configuration) each existed as separate objects visible to AWS, we could give each component unique permissions.  The configuration-update step would be the only one to have access to the image ID in SSM Parameter Store.  Likewise, that step would be forbidden from creating or terminating EC2 instances.

Due to implementation details, for us and our system, we could split it into the following components:

1. Configuration and test script deployment (run by developers off-AWS; stores to S3)
2. Ubuntu base image lookup (read-only)
3. Image build sequence (create/wait/destroy EC2 instances, tag instance and image, read configuration scripts from S3, invoke the AWS-maintained SSM Automation to create the image from the instance)
4. Image test sequence (create/destroy EC2 instances, read the test script from S3)
5. Configuration update (describe the image, update a specific SSM Parameter Store item)
6. Garbage collection (read running configuration to determine “unused” status, deregister images, delete their snapshots)

We only need to update the configuration scripts if we make changes to them; otherwise, doing a rebuild simply pulls in the available security updates.  Nothing inside AWS gets special permission to write these files.

After that, we can run step 2 to find an appropriate base AMI on our cron host.  That requires no write access, so all the cron host really needs is permission to start and monitor the latter tasks.  Tasks 3-5, in particular, are simple enough (once inputs are determined in step 2) to be run via AWS SSM Automation. I imagine there will be a “coordinating” automation that runs those three tasks in order, and the granular tasks exist mainly for ease of debugging each one.  Finally, garbage collection is somewhat compute-heavy but requires no waiting, so AWS Lambda might be the best option for it.

The end result will be a major improvement: each step will actually have minimal privileges, and the union of all privileges is much less than the admin privileges that I am technically granting to the process.  It “works” in the sense that we trust my entire laptop, but what if we didn’t need to?

Fixing a Random ALB Alarm Failure

tl;dr: if an Auto Scaling Group’s capacity is updated on a schedule, the max instance lifetime is an exact number of days, and instances take a while to reach healthy state after launching… Auto Scaling can terminate running-but-healthy instances before new instances are ready to replace them.

I pushed our max instance lifetime 2 hours further out, so that the max-lifetime terminations happen well after scheduled launches.

CloudSearch's Tricky prefix Operator

We ran into an interesting problem with CloudSearch.  Maybe I did it wrong, but I stored customer names in CloudSearch as “text” type with “English” analysis.  We do generic-search-bar scans with prefix searches, like (or (prefix field=name 'moon') (prefix field=address 'moon')).

Then, a developer found that a search term of “john” would find customers with a name of “johns”, but a search for “johns” would not!  The root cause turned out to be that the English analyzer stems everything that is a plausible plural, storing “Johns” as “john”.

Normally, this isn’t a problem.  When—and only when—using a prefix search, stemming is not applied to the terms for those matches.  Thus, doing a prefix search of “johns” will match “johnson” but not “johns”.  Doing a regular search through the CloudSearch Console will turn up the expected customers, and so might checking the database directly, adding to the confusion.  It even works as expected with most names, because “Karl” or “Packard” don’t look like plurals.

We added a custom analyzer with no stemming, set our text fields to use it, and reindexed.

Our Long-Term AWS CloudSearch Experience

AWS has announced the deprecation of CloudSearch, among other services, just as I wanted to share why we chose it, and how it worked out.

Competitors

The field we considered when choosing CloudSearch included Sphinx, ElasticSearch (the real one and AWS’ ripoff), MySQL FULLTEXT indexes, rolling our own in-SQL trigram search, and of course, CloudSearch.

We had operational experience with Sphinx. It performed well enough, but it is oriented toward documents, not the smaller constellation of attributes I was interested in here.  It took quite a chunk of memory to index our tickets (description/comments), required a pet machine, and didn’t vibe correctly with the team.  I didn’t want to commit to putting 100 times more entries in it, then defending it politically for all eternity.

ElasticSearch appeared to be hyper-focused on log searching specifically, more like what we’re already doing with Papertrail.  It was not clear that it could be used for other purposes, let alone how to go about such things.

We actually had an in-SQL trigram search already, but only for customer names.  I built it because MySQL’s full-text index features were not in great health at the time. (I thought full-text indexes were deprecated ever since, but in checking today, this appears not to be the case.  Even the MySQL 9.0 docs don’t mention it.) I started populating an additional trigram index for all the data I was interested in searching, and it blew up our storage size so fast I had to stop it and definitely find something else. That’s also how I found out that RDS can’t reclaim storage; once it expands, it has expanded for good.

The problem with using MySQL’s full-text indexing was the related integer fields that needed to be indexed.  We wanted to have a general search field, where the user could put in “Sunesh” or “240031” and get the related customer or transaction number, without a complex multi-part form.  Doing that with nothing but MySQL features seemed difficult and/or slow.

“Do nothing” wasn’t really an alternative, either; to search all the relevant fields, MySQL wanted to do two full table scans.  Searches would be running against the largest tables in the database, which makes even a single full scan prohibitively expensive.

CloudSearch

CloudSearch got a great review in my collection of blurbs about AWS services, but further experience has been somewhat less rosy.

For background, CloudSearch is arranged into one-dimensional domains, with a limited second dimension in the form of array attributes.  To contain costs, I chose to index our customers, attaching their VINs as array attributes, rather than have separate customer and vehicle domains or redundantly index the customer attributes on every vehicle.  This results in a domain with 2.5M records.  (Doing some serious guesswork, that means around 12M contracts in total.  Give or take a couple million.)

Things worked fine with a ‘small’ search instance for a while, but it didn’t handle bursty traffic.  Last month, I resized the instance to ‘medium’, and rebuilt the index… which took an unknown number of hours between 2 and 18, inclusive.

Why don’t I know exactly how long it took?  Well, that’s the next problem: metrics. CloudSearch only keeps metrics for three hours, and doesn’t have an event log.  (They appear to go into CloudWatch, but with a custom 3-hour expiration time.) When did the rebuild finish?  Dunno!  Did the system get overwhelmed overnight?  Too bad; that’s gone! With the basic metrics being so anemic, there’s definitely nothing as useful as RDS’ Performance Insights, which is what I would really want here.

Our instance has managed to survive adequately at medium for a while, but I don’t know when I’ll have to scale it up as we roll out this search to more parts of the system.  We just don’t have the metrics here to plan capacity.

Considering that, and the deprecation of it by AWS, I would love to have an alternative… except it would just be CloudSearch, improved.

AWS CodeDeploy’s Blue/Green Deployment Gotcha

Once, well after I no longer remembered how the whole thing was bootstrapped, I accidentally deleted the target group associated with a CodeDeploy application that was configured for blue/green deployment.  That’s how I found out (rediscovered?) that CodeDeploy doesn’t create a target group for blue/green deployments, it copies an existing one.  Since I had just deleted that existing one, I couldn’t do a (re)deployment and bring the system back online!

(Also, it cemented my opinion that prompts should be like, “Type ‘delete foo-production-dAdw4a1Ta’ to delete the target group” rather than “Type ‘delete’ to delete.” Guess which way the AWS Console is set up.)

I started up an instance to add to a new target group, and it promptly fell over.  The AMI had health monitoring baked in, and one of the health checks was “CodeDeploy has installed the application on the instance.”  Since it was not CodeDeploy starting the instance for the purpose of installing the application, the health check failed, and Auto Scaling dutifully pulled it down to replace it.

Meanwhile, the lack of healthy instances was helpfully sending alerts and bringing my boss’ attention to the problem.

[Now I wonder if it could have worked to issue a redploy at this point.  The group was there to copy, even if the instances weren’t functional.  I guess we’ll never know; I’m not about to delete the target group again, just to find out!]

I ended up flipping the configuration to using EC2 health checks instead of HTTP, and then everything was stable enough to issue a proper redeployment through CodeDeploy.  With service restored, I finally put the health checks back to HTTP.

And then, with production in service again, I finally got to work on moving staging from in-place to blue/green.  Ironically, I would have learned the lesson either way; but by breaking production, it really stuck with me.

Stateful Deployment was Orthogonal

I used to talk about “stateful, binary” deployment, thinking that both things would happen together:

1. We would deploy from a built tarball, without any git pull or composer install steps
2. We would record the actual version (or whole tarball path) that was deployed

This year, we finally accumulated enough failures caused by auto-deploy picking up pushed code that wasn’t ready that we decided we had to solve that issue. It turned out to be unimportant that we weren’t deploying from tarballs.

We introduced a new flag for “auto mode” for the instance-launch scripts to use. Without the flag, deployment happens in manual mode: it performs the requested operation (almost) as it always has, then writes the resulting branch, commit, and (if applicable) tarball overlay as the deployed state.

In contrast, auto mode simply reads the deployed state, and applies that exact branch, commit, and overlay as requested.

I say “simply,” but watch out for what happens to a repository which doesn’t have any state stored.  This isn’t a one-time thing: when adding new repositories later, their first deployment won’t have state yet, either.  This can disrupt both auto and manual deployments.

AWS: Requesting gp3 Volumes in SSM Automation Documents

I updated our EC2 instance-building pipeline to use the gp3 volume type, which offers more IOPS at lower costs.

Our initial build runs as an SSM [Systems Manager] Automation Document.  The first-stage build instance is launched from an Ubuntu AMI (with gp2 storage), and produces a “core” image with our standard platform installed.  This includes things like monitoring tools, our language runtime, and so forth.  The core image is then used to build final AMIs that are customized to specific applications.  That is, the IVR system, Drupal, internal accounting platform, and antique monolith all have separate instances and AMIs underlying them.

Our specific SSM document uses the aws:runInstances action, and one of the optional inputs to it is BlockDeviceMappings.  Through some trial and error, I found that the value it requires is the same structure as the AWS CLI uses:

- DeviceName: "/dev/sda1"
  Ebs:
    VolumeType: gp3
    Encrypted: true
    DeleteOnTermination: true

Note 1: this is in YAML format, which requires spaces for indentation.  Be sure “Ebs” is indented two spaces, and the subsequent lines four spaces.  The structure above is a 1-element array, containing a dictionary with two keys, and the “Ebs” key is another dictionary (with 3 items.)

Note 2: the DeviceName I am using comes from the Ubuntu AMI that I am using to start the instance.  DeviceName may vary with different distributions.  Check the AMI you are using for its root device setting.

The last two lines (Encrypted and DeleteOnTermination) may be unnecessary, but I don’t like leaving things to chance.

Doing this in a launch template remains a mystery.  The best I have been able to do, when trying to use the launch template, Amazon warns me that it’s planning to ignore the entire volume as described in the template.  It appears as if it will replace the volume with the one from the AMI, rather than merging the configurations.

I know I have complained about Amazon in the past for not providing a “launch from template” operation in SSM, but in this case, it appears to have worked out in my favor.

Passing data from AWS EventBridge Scheduler to Lambda

The documentation was lacking images, or even descriptions of some screens ("Choose Next. Choose Next.")  So, I ran a little experiment to test things out.

When creating a new scheduled event in AWS EventBridge Scheduler, then choosing AWS Lambda: Invoke, a field called "Input" will be available.  It's pre-filled with the value of {}, that is, an empty JSON object. This is the value that is passed to the event argument of the Lambda handler:

export async function handler(event, context) {
  // handle the event
}

With an event JSON of {"example":{"one":1,"two":2}}, the handler could read event.example.two to get its value, 2.

It appears that EventBridge Scheduler allows one complete control over this data, and the context argument is only filled with Lambda-related information.  Therefore, AWS provides the ability to include the <aws.scheduler.*> values in this JSON data, to be passed to Lambda (or ignored) as one sees fit, rather than imposing any constraints of its own on the data format.  (Sorry, no examples; I was only testing the basic features.)

Note that the handler example above is written with ES Modules.  This requires the Node 18.x runtime in Lambda, along with a filename of "index.mjs".

Experiences with AWS

Our core infrastructure is still EC2, RDS, and S3, but we interact with a much larger number of AWS services than we used to.  Following are quick reviews and ratings of them.

CodeDeploy has been mainly a source of irritation.  It works wonderfully to do all the steps involved in a blue/green deployment, but it is never ready for the next Ubuntu LTS after it launches.  As I write, AWS said they planned to get the necessary update out in May, June, September, and September 2022; it is now January 2023 and Ubuntu 22.04 support has not officially been released. Ahem. 0/10 am thinking about writing a Go daemon to manage these deployments instead.  I am more bitter than a Switch game card.

CodeBuild has ‘environments’ thrown over the wall periodically.  We added our scripts to install PHP from Ondřej Surý’s PPA instead of having the environment do it, allowing us to test PHP 8.1 separately from the Ubuntu 22.04 update.  (Both went fine, but it would be easier to find the root cause with the updates separated, if anything had failed.) “Build our own container to route around the damage” is on the list of stuff to do eventually.  Once, the CodeBuild environment had included a buggy version of git that segfaulted unless a config option was set, but AWS did fix that after a while.  9/10 solid service that runs well, complaints are minor.

CodeCommit definitely had some growing pains.  It’s not as bad now, but it remains obviously slower than GitHub.  After a long pause with 0 objects counted, all objects finish counting at once, and then things proceed pretty well.  The other thing of note is that it only accepts RSA keys for SSH access.  6/10 not bad but has clearly needed improvement for a long time.  We are still using it for all of our code, so it’s not terrible.

CodePipeline is great for what it does, but it has limited built-in integrations.  It can use AWS Code services… or Lambda or SNS.  8/10 conceptually sound and easy to use as intended, although I would rather implement my own webhook on an EC2 instance for custom steps.

Lambda has been quarantined to “only used for stuff that has no alternative,” like running code in response to CodeCommit pushes.  It appears that we are charged for the wall time to execute, which is okay, but means that we are literally paying for the latency of every AWS or webhook request that Lambda code needs to make.  3/10 all “serverless” stuff like Lambda and Fargate are significantly more expensive than being server’d.  Would rather implement my own webhook on an EC2 instance.

SNS [Simple Notification Service] once had a habit of dropping subscriptions, so our ALB health checks (formerly ELB health checks) embed a subscription-monitor component that automatically resubscribes if the instance is dropped.  One time, I had a topic deliver to email before the actual app was finished, and the partner ran a load test without warning.  I ended up with 10,000 emails the next day, 0 drops and 0 duplicates.  9/10 has not caused any trouble in a long time, with plenty of usage.

SQS [Simple Queue Service] has been 100% perfectly reliable and intuitive. 10/10 exactly how an AWS service should run.

Secrets Manager has a lot of caching in front of it these days, because it seems to be subject to global limits.  We have observed throttling at rates that are 1% or possibly even less of our account’s stated quota.  The caching also helps with latency, because they are either overloaded (see previous) or doing Serious Crypto that takes time to run (in the vein of bcrypt or argon2i).  8/10 we have made it work, but we might actually want AWS KMS instead.

API Gateway has ended up as a fancy proxy service.  Our older APIs still have an ‘API Definition’ loaded in, complete with stub paths to return 404 instead of the default 403 (which had confused partners quite a bit.) Newer ones are all simple proxies.  We don’t gzip-encode responses to API Gateway because it failed badly in the past. 7/10 not entirely sure what value this provides to us at this point.  We didn’t end up integrating IAM Authentication or anything.

ACM [AWS Certificate Manager] provides all of our certificates in production.  The whole point of the service is to hide private keys, so the development systems (not behind the load balancer) use Let’s Encrypt certificates instead.  10/10 works perfectly and adds security (vs. having a certificate on-instance.)

Route53 Domains is somewhat expensive, as far as registering domains goes, but the API accessibility and integration with plain Route53 are nice.  It is one of the top-3 services on our AWS bill because we have a “vanity domain per client” architecture.  9/10 wish there was a bulk discount.

DynamoDB is perfect for workloads that suit non-queryable data, which is to say, we use it for sessions, and not much else.  It has become usable in far more scenarios with the additions of TTL (expiration times) and secondary indexes, but still remains niche in our architecture.  9/10 fills a clear need, just doesn’t match very closely to our needs.

CloudSearch has been quietly powering “search by name” for months now, without complaints from users.  10/10 this is just what the doctor ordered, plain search with no extra complexity like “you will use this to parse logs, so here are extra tools to manage!”

That’s it for today.  Tune in next time!

Amazon CloudSearch Security Policies

I have been looking into CloudSearch to introduce a "search everything" feature to my employer's core app.  The interaction of user and resource policies was a little bit confusing, but I think it works as follows.

A CloudSearch resource policy is needed to allow access outside the owner's account, or to restrict access by IP address.  A blank CloudSearch policy is a perfectly functional option for the owner.  Although the UI says, "this will only allow access through the console," it actually means that only access policies set on the IAM user/role making the request are relevant.  "The console" just happens to be running as the logged-in user, with those IAM permissions.

As I understand it, once CloudSearch is accessed, the permission checks proceed along these lines:

1. Does the CloudSearch resource policy allow the account?  If there's no policy, only the owner is permitted; otherwise, the policy is used to determine whether cross-account or anonymous access is permitted.  (Caution: it's not clear to me whether a policy that specifies cross-account access, but doesn't include the owner, will deny the owner.)  If the account is not permitted, then the request is denied.
   
2. Is there an IP restriction?  If so, and the caller's IP is not permitted, then the request fails.  If there is no IP restriction, then the connection is permitted.
3. Does the caller have permission through IAM (user or role) in their account to make this request?  If there's no explicit grant, then the request is, once again, denied.
   

Putting my own AWS account ID in as the allowed account, with no IP address restrictions, did not seem to be sufficient to grant access.  When I gave my instance access through an IAM role, no CloudSearch resource policy was necessary to allow access the domain.

The documentation notes that IP address restrictions must be given in the CloudSearch resource policy.  I believe this arises because the IP can only be checked once a connection is made to CloudSearch.  Prior to that point, the caller's IP address is not available for checking.

Likewise, if I understand the documentation properly, cross-account access needs both a CloudSearch resource policy set to allow access from the caller's AWS account, and the caller's IAM also needs to allow access to the CloudSearch domain.  However, we only have the one account, so I haven't fully tested this scenario.

A New SSH Key Philosophy

I rolled over my SSH keys this morning; but more than that, I generated two additional keys, so now I have four distinct domains:

1. Local systems
2. EC2 systems
3. AWS CodeCommit (read/write)
4. AWS CodeCommit (read only) – this one already existed

Previously, I would load “the SSH key” into ssh-agent, and do everything with one key.  CodeCommit would always be accessed in read/write mode; even systems that only needed read-only access for testing would access the read/write key through the forwarded agent.

Because there was only one skeleton key, which I needed frequently, it would be available to any executing code, any time I was working. All hosts it would work on were clearly indicated in .ssh/config and .git/config files. Any code on a guest VM would also be able to access it, through the forwarded agent.  The guest’s configuration also included the hosts, because I once developed from the guest environment.  Back then, the host was Windows, making it more comfortable to access EC2 and git from the guest.

The first two keys I generated represent different frequencies of access. I access the local systems much more frequently than EC2, to debug code under development running on them.  Splitting those keys means that the EC2 key will be unlocked and available much less frequently.

As for CodeCommit, I decided to separate “shell” keys from “git” keys. Our threat model generally gives up on security if an attacker has shell access, so if I can keep local malware away from remote shell access, it adds a layer to the defense.  In addition, this key is also accessed more frequently than the EC2 key.

Finally, I quit forwarding the SSH agent by default to test systems.  They already have the (sufficient) read-only key installed when their base image is created, so all I had to do was “stop changing the configuration” in my personal setup scripts for those guest VMs.  This reduces unnecessary trust (and coupling) between host and guest.

Of note, I am already guarding the AWS keys with aws-vault, which I wrote about previously. I was sufficiently pleased with aws-vault that I’m using it on my home Linux installation, as well.

Overall, it took me an hour or two to work out the plan and reconfigure everything, but I’m happy with the improved security posture.

Deployment May Be Stateful

Our deployment process can technically accept a commit hash or an alternate branch to deploy, but by default, it updates to the currently checked-out branch tip. This default also applies to the auto-update code that brings our pre-baked AMI up-to-date when it launches.

For the most part, this is fine.  We keep master in a deployable state, and that’s always the desired version to deploy.  Thus, the whole system is stateless…

But, it also means that we can’t use our fancy “change branch” or “deploy commit” operations very much.  If we do, then the desired version is no longer what the AMI will auto-deploy when new instances launch from it.  We have to either build a new AMI (for the branch) or restore the deployability of master before any new instances launch.

If we reach the “deploy from tarball” goal, then life would be easier.  Builds could happen from any branch or commit naturally, and we could prevent a broken tarball from auto-deploying by simply deleting it.

Adventures with kASLR

Things I discovered recently about kASLR:

• Linux added kaslr a while ago, in 3.14.
• It was enabled via kaslr on the kernel command line.
• kaslr wasn’t compatible with hibernation, originally.  This appears to have changed in 4.8.
• It was enabled by default in 4.12, with nokaslr to disable it.
• kaslr support is compiled in by CONFIG_RANDOMIZE_BASE=y.
• Ubuntu 18.04 LTS (bionic) has that setting enabled in the generic and aws kernels, which are based on 4.15.
• /etc/default/grub may be useless on Ubuntu for setting command line flags.

Under the hood, there are “up to” 512 base addresses for the kernel (depending on the specific machine’s memory map), and kaslr decompresses the image into a random available one during bootup.  This puts the base kernel “somewhere” in a 1 GB-sized area, aligned at 2 MB.

The kernel command line is available in /proc/cmdline.  However, it didn’t have my kaslr customization, which sent me on a quest.  I discovered that Debian/Ubuntu configure a bunch of scripts to produce the final grub configuration file, using /etc/default/grub.d/*.cfg.  These are processed after /etc/default/grub.  There turned out to be a “cfg” file that unconditionally replaced GRUB_CMDLINE_LINUX_DEFAULT, which is where I had put our kaslr flag.  This affected both of our instance types: VirtualBox appeared to have one unintentionally left over from the installer, while AWS had one placed there as part of the cloud image build.

But given that kaslr appears to be default, instead of setting up a local configuration file, I ended up removing the code that was trying in vain to set kaslr.

Why we have a memcached-dynamo proxy

Previously, I mentioned that we have a proxy that speaks the memcached protocol to clients and stores data in dynamodb, “for reasons.”  Today, it’s time to talk about the architecture, and those reasons.