Notes on an ECS Deployment

In order to try FrankenPHP and increase service isolation, we decided to split our API service off of our monolithic EC2 instances.  (The instances carry several applications side-by-side with PHP-FPM, and use Apache to route to the applications based on the Host header.  Each app is not supposed to meddle in the neighbor’s affairs, but there’s no technical barrier there.)

I finally got a working deployment, and I learned a lot along the way.  The documentation was a bit scattered, and searching for the error messages nearly useless, so I wanted to pull all of the things that tripped me up together into a single post.  It’s the Swiss Cheese Model, except that everything has to line up for the process to succeed, rather than fail.

1. Networking problems
2. ‘Force Redeployment’ is the normal course of operation
3. The health check is not optional
4. Logs are obscured by default
5. The ports have to be correct (Podman vs. build args)
6. The VPC Endpoint for an API Gateway “Private API” is not optional
7. There are many moving parts

Let’s take a deeper look.

Container Friction

I find it inconvenient that a number of container settings are immutable.  Forget a volume?  Forget a port mapping?  Want to bring up a container, make a change, and then mark the root filesystem as read-only?  Start with a read-only root, and then realize it should have been read-write after all?

Too bad.  Go set up all of the other settings again, along with the desired change, and don’t miss anything or get any settings wrong this time.  If it’s “change something before going read-only,” it’s time to create a Containerfile and build a new image, too.  No matter what, don’t forget to delete the original container first, to free its name for reuse.

There’s no way to create a container based on an existing container’s settings.  There’s no direct option for it.  There’s not even an indirect option to export the settings of a container, and then use them as a template during container creation later.

It’s worse in Podman Desktop, which has no memory.  It doesn’t even have separate “last used directory” memory for building from a Containerfile versus choosing a volume to mount.  Why not make everyone go back and forth through the filesystem every time?

At least in the CLI, the previous commands could be in the shell history.  “Could be,” because I might have been distracted for two months, and the commands got pushed out of the history in the meantime.  (Speaking of, “the container” has its Containerfile baked in, but doesn’t remember where on the filesystem that Containerfile came from.  Not even locally.  Maybe it’s just me, but it’s always a treasure hunt to resume a project.)

I can see the logic of “not allowing settings drift” in production container environments, but it seems like Podman Desktop should be optimized for the developer experience and experimentation instead.

Online Builds

As a long-time coder and tinkerer, who views computers as deterministic if we understand them properly, modern tooling feels wrong to me.

• Python expects code to be distributed in a form where it has to contact PyPI for dependencies. (You can get around this—like the awscli installer—but I never did figure out how they build that.)
• Python expects code to be distributed in a form where the installation process executes arbitrary code. This transitively happens with all dependencies. 🐝
• composer install (usually) expects to be able to fetch code from GitHub.  Running it (non-interactively and with --no-dev, of course) as part of deployment makes deployment depend on the internet working.
• Containerfile ADD and COPY will happily take URLs as sources, including URLs that are intended to be mutable, like GitHub /latest/ release artifact URLs. Projects may recommend using such URLs.
• curl … | sudo sh also deeply connects the internet to the process, and treats the script itself as ephemera, discarding it as the process completes. If the script makes its own internet connections, the problem with preserving the canonical source is multiplied.

Quite aside from “the internet connection has to be up,” the referenced URLs must keep working over time.  A Containerfile built as recommended for the docker-php-extension-installer inherently requires the up-to-date source of code to remain at the github.com site, and under the mlocati user.

Building reliability and reproducibility into the process is left up to the user.  Those features can only be included if the thought, “what if…?” crosses someone’s mind.

However, saving remote resources into a local build context protects them from loss, but requires the maintainer of that build to update those resources.  Probably manually.  If it can’t be changed out from under me the next time I run podman build, then it also isn’t getting updates to follow changes in the base image.  It takes some discipline to track where these things come from, and sometimes, how to reproduce them.  For instance, when GPG keys for an Ubuntu PPA needed to be converted to binary before use, it wasn’t enough to leave only the URL written down.

Thus, it’s more work, but the result is stable, and that’s important to me.

Podman Desktop Isn’t Great

Since buying a new computer, my primary desktop is no longer Linux, and containers are no longer native.  I decided that the path of least resistance would be to try Podman Desktop, but it leaves a lot to be desired.

Scattered Thoughts on Distrobox

Distrobox’s aim is to integrate a container with the host OS’ data, to the extent possible.  By default, everything (process, network, filesystem) are shared, and in particular, the home directory is mounted into the container as well.  It is not even trying to be “a sandbox,” even when using the --unshare-all option.

I also found out the hard way that Distrobox integrates devices explicitly. If some USB devices are unplugged, the container will not start.  This happened because I pulled my laptop away from its normal dock area (with USB hub, keyboard, and fancy mouse) and tried to use a distrobox.  Thankfully, I wasn’t fully offline, so I was able to rebuild the container.  [Updated 2024-11-28: This danger is persistent.  Creating a container without the USB devices connected, then running it later with the devices, will make it fail to start if the devices are unplugged.  This ended up being impossible to live with.]

Before it stopped performing properly in Ubuntu Studio 23.10, I used distrobox to build ares and install it into my home directory.  This process yielded a ~/.local/share/applications/ares.desktop file, which my host desktop picked up, but which would not actually work from the host.  I always needed to be careful to click the “ares (on ares)“ in the menu after exporting, to start it on the ares container.

I have observed that distroboxes must be stopped to be deleted, but then distrobox will want to start them to un-export the apps.  Very recent distrobox versions ask whether to start the container and do the un-exporting, but there’s still a base assumption that you wanted the distrobox specifically to export GUI apps from it.  It clearly doesn’t track whether any apps are exported, because it always asks, even if none are.

Podman Fully Qualified Image Names

I installed Podman.  Then, of course, I tried to pull an image, but not the example ubuntu image.  This immediately generated an error:

Error: short-name "golang" did not resolve to an alias and no unqualified-search registries are defined in "/etc/containers/registries.conf"

(The “ubuntu” example the tutorial used is actually configured as a short-name in the file /etc/containers/registries.conf.d/shortnames.conf on my system. It’s also possible that later Podman versions will prompt for a repository.)

Luckily, there’s a fairly straightforward way to convert the image names to fully-qualified ones that Podman will accept.

Two basic concepts apply:

1. Docker images are always on docker.io
2. Docker images without a prefix (as in “golang” or “postgres”) are under library

That is, once we rewrite Docker commands for Podman, they might look like:

podman pull docker.io/library/postgres:alpine
podman pull docker.io/gitea/gitea:latest

These are the equivalent to pulling postgres and gitea/gitea with Docker.

If you’re looking at an image on Docker Hub, and its URL is something like hub.docker.com/_/golang, then the _ is the “lack of prefix.”  It indicates that the URL for Podman will be docker.io/library/golang.

If the URL is, instead, hub.docker.com/r/gitea/lgtm, then that shows the prefix (user gitea) and container (lgtm) after the /r/: the final URL for Podman will be docker.io/gitea/lgtm.

The final thing to note is that this information applies to the FROM directive in a Dockerfile as well, and it’s compatible with Docker.  Docker and Podman both happily accept FROM docker.io/library/golang:1.19-alpine.

The sea change to vendoring and containers

Not so long ago, we would install code to the global runtime path, where it would be available machine-wide.  CPAN was perhaps the earliest to do this, but PEAR, rubygems, and probably others followed.  After all, CPAN was seen as a great strength of Perl.

Users of such a library would say something like use Image::EXIF; or include_once "Smarty/Smarty.class.php"; Notice that these are a mechanism to follow some include path to resolve relative filenames.

But as servers got larger faster than applications, co-installing those applications became attractive.  The problem was that they can have different dependency requirements, which complicates the management of a shared include path.

This is about where Bundler happened for Ruby, and its ideas were brought to other ecosystems: Perl got carton, and PHP got composer. These systems build a directory within the project to hold the dependencies, and basically ignore the global include path entirely.

At the cost of disk space, this bypasses the problems of sharing dependencies between projects.  It bypasses all the problems of working with multiple versions of dependencies that may be installed in different hosts’ include paths.  It also makes the concept of the language’s include path obsolete, at least in PHP’s case: the Composer autoloader is a more-capable “include path” written in code.  Finally, it allows piecemeal upgrades—a small service can make a risky upgrade and gain experience, before a larger or more important service on the machine needs to upgrade.

Piecemeal upgrades also enable independent teams to make more decisions on their own.  Since their dependencies are not shared with other users, changes to them do not have to be coordinated with other users.

Containers are the next step in the evolution of this process.  Pulling even more dependencies into a container brings all the advantages of local management of packages into a broader scope.  It allows piecemeal upgrading of entire PHP versions.  And it makes an application cost an order of magnitude more space, once again.

In our non-containerized environment, I can switch the PHP version on a host pretty easily, using co-installable versions and either update-alternatives (for the CLI) or a2disconf/a2enconf (for FPM), but this means the services do not really have a choice about their PHP version.  It has been made before the application's entry point is reached.

Where is 'localhost'? (Docker networking)

Building up to the devproxy2 release, I set things up to test in Docker, using a known version of Go.  Nothing worked for quite some time.

Core problem: localhost inside the container is not the host network.  I had to configure it to listen on all addresses (by asking Go to connect to ":8080" without a host address), per this other Stack Overflow answer. If I had thought to check the exact error message client side (curl: empty reply from server), I could have solved this one first, not last.

Once I fixed the core problem, everything started working, and I didn’t go back and test the other things I had tried.  So everything else is only a potential problem.

Potential problem: localhost inside the container is not the host network.  I configured it to connect to host.docker.internal instead, per this Stack Overflow answer.  This is the same as the core problem, but in the other direction.

Potential problem: according to my notes, the default network bridge in Docker doesn’t have Internet access. Creating a user-defined bridge solves this.  Furthermore, using Docker Compose automatically creates a user-defined bridge for the stack.  By the time I solved the core problem, I was already using Docker Compose for the networking feature.

Containers are Interop

I mean this in the same sense as “XML was created to solve the interoperability problem.”

The container craze is about the interoperability problem across environments. By vendoring the entire distribution and communicating only over the network, they essentially provide isolation for all the dependencies of a service. Maybe that part is the same, in essence, as the nix package manager.

But then containers have one more trick: they run anywhere with a “Linux syscall interface” underneath. Any environment with Docker support can run Docker containers, anywhere. (As long as the binaries run on the host, at least.) It’s not entirely simple—orchestration is an issue, and Docker is working on that, too—but the containers themselves become highly portable, since they’re shipped as black boxes. They don’t depend on code outside themselves, and as such, that outside code cannot break them so easily.

And maybe, by so fully entwining a Linux distro to our app, we’re forgetting how to be cross-distro or cross-platform. And the old coder in me wants to grump about that. Yet, that’s also a kind of freedom. Not everyone has to learn how to write cross-platform code if the container environment defines exactly one platform to run on.

Maybe we’re losing something, but we’re also gaining ease-of-use and accessibility in the deal.

Brief notes on Packer

packer.io says they build identical images for multiple platforms such as EC2, VirtualBox, VMWare, and others (with even more on the way via plugins).  However, it's more about customizing a machine already running than it is about fabricating a new machine out of whole cloth and distributing the result to disparate services.  That is, the AMI builder starts from another AMI; the VirtualBox builder starts from an installer ISO (with commands to make the install unattended) or an OVF/OVA export and customizes that to produce an output OVF/OVA export.  This is, as I understand it, in contrast to Vagrant or Docker images, which build one image that runs on many systems.

Packer is about customizing existing machines that already exist in some form, then building the result as a new image for the same system.  If you're really interested in taking "one base image" to many environments, then Docker is probably the better choice there.  It's built on Linux Containers, where the virtualization happens inside the host kernel instead of a dedicated hypervisor, which makes it possible to run a bit-for-bit identical container both  inside EC2 (even PV) and inside VirtualBox.

In another contrast to vagrant, packer supports a shell provisioner that lets you upload a shell script and run wild.  Last I knew, which is probably woefully out of date, vagrant supported puppet by default and chef-solo as an option.  Since I happen to have an existing build system constructed around shell scripts for provisioning (with a dash of perl or php for the trickier parts), albeit specifically for EC2 instances, packer might suit my codebase and philosophy a little better.

Put another way, and beating the dead horse a bit more, Vagrant and Docker are about creating One True Image and later instantiating that single image on different underlying services (including EC2 and VirtualBox).  Packer is about coordinating the construction of native images for those services; it's about making predictable, repeatable changes to a running instance in the service and exporting the result for later use.

Unrelated to the above, packer is written in go 1.2 and distributes binaries for various platforms.  Docker is also written in go (but I'm not sure of the version target), leaving Vagrant in Ruby as the odd man out.  Having been subjected to rvm to install octopress at one point, I'm much more amenable to smaller dependency footprints... like static binaries produced by Go.  (Then again, I heard later that everyone hates rvm.)

Update 14 Feb 2014: I thought of one more useful contrast among the projects.  Docker and Packer are aimed toward building immutable, self-contained images.  Vagrant is focused on setting up (repeatable) development environments--particularly, in how it includes only the VirtualBox provider by default, and uses shared folders to persist changes as you develop.