LLMs Are an Inside Threat

There was another “my LLM deleted production” incident (via).  In this particular case, the agent trawled the filesystem to locate another credential with the rights to delete the database and all backups.

This demonstrates an LLM in the role of a “motivated attacker."  When faced with obstacles, it doesn’t simply halt; it inspects the error, adapts its approach, and overcomes the obstacles.  Small comfort that the filesystem outside the workspace is “read only,” if the damage is done beyond the filesystem boundary.

For the foreseeable future, I’ll continue to run the (mandated) LLM tool in a VM.  It’s all alone with the code in a mock production image, and aws-vault, documents, emails, D-Bus, and the GUI are all hidden away on the other side of the hypervisor.

The Heart of the Process (2012)

Editor's Note: I found this in my drafts, dated 2012-01-25.  I have opted to retain the content unchanged, merely updating links for ubiquitous HTTPS and replacing broken links as needed.  Please enjoy this work of fiction.

The President saw that things took time to implement in code, and involved programmers, testers, and a deployment.  As someone who liked to make snap decisions and have the results available immediately, this did not sit well in his heart.  Long he meditated, then announced his solution:

Henceforth, the system would be Database-Driven.

ScreenZen as an RSI Aid

It hurts my hands to hold a phone for a long time, but apps, games, and the internet in general can be pretty compelling stuff.  I tried using the built-in Screen Time features, but they are fairly bare-bones.  One of the issues was that I would open an app “for five minutes,” and then get the message out of nowhere that there were only five minutes left on the app for the entire day. It meant I had just lost an hour or two, and maybe wouldn’t be able to use the app later.

(For the sake of argument, assume that I never bypass the limit, knowing it only makes everything worse.)

That’s where ScreenZen came in.  I needed a way—ideally, free—to say, “I only want to do these things for so much time, and then I want to be pushed off of them for a while.” When I found ScreenZen, that’s how I configured it.  It took me a bit to sort out the options and settings, so I wanted to walk through what has worked for me.

• The basic settings were pretty obvious: “open each app up to [six] times a day, for [twenty] minutes each.” That’s the number of times the app(s) can be unlocked, and how long they are usable for (in real time) once unlocked.
• I set “Strict block – after daily open goal,” so there’s a real consequence to opening things too much.
• I chose some nice offscreen activities, and set that as the intervention screen.  (Back when I started using ScreenZen, this transformed it from “tap and then wait” to “wait and then tap,” which was much better for mindfulness.)
• The time between the app being locked and being able to unlock it again is under “Advanced – cooldown time.”

I’ve also set up the schedules so that, in the wind-down before bedtime, more things are locked.  ScreenZen has become my “automatic sleep timer” for podcasts.  A short cooldown time there also prevents me from hearing it pause, deciding ‘yeah… sure… i feel awake,’ resuming the podcast, and then immediately falling asleep.  Sometimes, when only Downtime starting would stop playback, I’d find myself seeking half an hour back in an episode to get something familiar, but now, it’s not usually more than ten minutes.

Overall, this has turned out to be much better than trying to control everything through Apple’s Screen Time. If I’m on the phone for a reason, it generally doesn’t take 20 minutes, but if I get into “rat pulling a lever” mode, ScreenZen will interrupt me much sooner than Apple would.  Furthermore, because the stress of holding the phone is nonlinear over time, switching a one-hour daily limit into five half-hour-or-less sessions becomes a reasonable option.

And yes.  I would rather get off the phone than put it in a stand, or add a grip to it.  When it hurts, it’s mainly because I’m unaware of how much time I’m wasting on it.

The main downside to all this is that, if I need a how-to video on something, I have to plan ahead.  I sit down at a non-pocketable computer, watch the video, and take notes if necessary.  The videos cut out the boring parts, making it definitely impossible to follow along in real time.

That, and for other people, well… ScreenZen works hard to be “mindful” and not actually that “controlling.” It is willing to offer a bypass in several places.  I have managed not to touch it for almost an entire year, but I know it would be rather tempting for someone who doesn’t have physical reasons to avoid it.

So to wrap up… that link again is ScreenZen and it’s available for iOS, Android, and macOS.  This is not an ad nor a paid review, that isn’t an affiliate link (unless Blogger has made it one for their benefit), and this post is 100% human-generated.

Online Builds

As a long-time coder and tinkerer, who views computers as deterministic if we understand them properly, modern tooling feels wrong to me.

• Python expects code to be distributed in a form where it has to contact PyPI for dependencies. (You can get around this—like the awscli installer—but I never did figure out how they build that.)
• Python expects code to be distributed in a form where the installation process executes arbitrary code. This transitively happens with all dependencies. 🐝
• composer install (usually) expects to be able to fetch code from GitHub.  Running it (non-interactively and with --no-dev, of course) as part of deployment makes deployment depend on the internet working.
• Containerfile ADD and COPY will happily take URLs as sources, including URLs that are intended to be mutable, like GitHub /latest/ release artifact URLs. Projects may recommend using such URLs.
• curl … | sudo sh also deeply connects the internet to the process, and treats the script itself as ephemera, discarding it as the process completes. If the script makes its own internet connections, the problem with preserving the canonical source is multiplied.

Quite aside from “the internet connection has to be up,” the referenced URLs must keep working over time.  A Containerfile built as recommended for the docker-php-extension-installer inherently requires the up-to-date source of code to remain at the github.com site, and under the mlocati user.

Building reliability and reproducibility into the process is left up to the user.  Those features can only be included if the thought, “what if…?” crosses someone’s mind.

However, saving remote resources into a local build context protects them from loss, but requires the maintainer of that build to update those resources.  Probably manually.  If it can’t be changed out from under me the next time I run podman build, then it also isn’t getting updates to follow changes in the base image.  It takes some discipline to track where these things come from, and sometimes, how to reproduce them.  For instance, when GPG keys for an Ubuntu PPA needed to be converted to binary before use, it wasn’t enough to leave only the URL written down.

Thus, it’s more work, but the result is stable, and that’s important to me.

Nostalgia for the Amiga

When I was young, my father owned a series of Commodore machines, the last one being an Amiga 500.  BASIC was for chumps, and our only other alternative was an assembler and a single misprint-riddled book for it.  We didn’t have the internet, or any local user group/mentoring.  We made the asm stuff work, by which I mean, mostly I stole the results of my older brother’s reverse engineering.

Years later, just before the Java phenomenon reached our little community college, I started learning C++ (officially) and C (from the internet), and some mysteries started falling into place.

• The large blocks of declare statements in assembly were a C struct.
• The weird parts of that struct like next_window: dc.l 0 were spaces for the OS to put a pointer to the next window, creating a singly-linked list of windows on screen.
• The difference between label and #label snapped into sharp relief when I reached Pointer Enlightenment. They were *p and p, respectively. (Somewhat; C doesn’t distinguish between absolute and PC-relative addressing.)
• Loading the label at the start of a block of declare statements into a register before making a library call was passing a pointer to the OS.
• The 68000’s data/address register split wasn’t arbitrary; only address registers could be used with an offset, so the pointer-to-struct went in an address register, for reading fields within the struct.
• That weird WaitForEvent function in the OS library was the core of cooperative multitasking.  The Amiga also had preemptive multitasking, but the time slice was large enough to make the system visibly laggy if a process was uncooperative. Like all of my asm programs.

That experience of simultaneous, two-way enlightenment (asm to C and back) with pointers was incredible, but has also been a once-in-a-lifetime high.  It also gave me a lot of “if-only” feelings.  Had I understood all this stuff back then, I could have done so much more with the Amiga.

(Incidentally, that next-window pointer being in the window structure makes it invasive.  I guess this means my first encounter with invasive lists could have been before Linux was ever released.)

I also felt a sort of awe or magic about the hardware design, that was missing from the PC space.  The Amiga was half computer, half game console internally, with a bunch of custom chips to do really neat graphics. Although it had a limited number of colors in the palette, the co-processor could switch out the palette between scan lines, allowing quite a few colors per screen.  HAM mode [hold-and-modify] could also produce “thousands of colors” out of a 16-color palette, if one could accept some color-fringing artifacts.

We didn’t have a VCR or camcorder (the money was spent on the computer) but apparently, the Amiga was great at video, too.

But it was soon proven to be a dead end.  When RAM gets fast enough, a boring linear RGB framebuffer is the best option, and bitplanes (or HAM mode) turn from “cool hacks” into mere workarounds for the era’s limitations.  The 68000 family wasn’t able to keep up with Intel’s immense fab budget to fling clockspeed with the best of them, and Commodore died before a PowerPC transition could have happened.  And who even knows if they could have been as successful as Apple with it?

The Amiga’s ability to “just plug it into a TV” became a limitation over time, as the business world put all its money into PCs with dedicated monitors.  They were more expensive, but also better.  Our first PC would run in 800⨉600, although it could also produce a flickery 1024⨉768.  By comparison, Workbench on the Amiga ran in “medium” resolution, 640⨉200.  High resolution only doubled the line count, and flickered intensively on our monitor.  Ultimately, monitors came down in price and offered much better than TV quality; I went off to university (post-community-college) with a monitor capable of 1280⨉1024.

This entry was somewhat prompted by “Classical "Single user computers" were a flawed or at least limited ideas”.

Thoughts from Trying Generators in PHP

I am late to the party, but I have been playing with Generators in PHP more, and running into the limitations of module boundaries.

Some module might produce a Generator so that iteration can be performed in chunks, reducing peak RAM.  For example, producing results one store at a time, instead of loading up all stores into a giant array.  Code that processes an entire database table, but wants to lower lock contention and memory use can also benefit; it can use a Generator to isolate the fetch-in-pages logic from processing the individual records.  The consumer sees one stream of results, while the Generator fetches more as needed.

In short, there are plenty of use cases.

The problem comes when a caller wants to pass “the data” produced by the Generator to another function or method that specifically takes an array.  Once that happens, either the destination needs to be reworked to accept the broader iterable type, or the efforts toward efficiency are erased by an iterator_to_array() call.

(Of course, back when generators were introduced to PHP, I didn’t use type declarations, so I could have gotten away with throwing a generator at something that assumed it would receive an array or PDOStatement. Dealing with larger teams and beginning to use an IDE were both great reasons to add the type information, and the array type forbids passing a Generator in its place.)

A separate issue is that anything consuming a Generator (thus, anything type-hinted iterable) needs to be aware of its once-only nature.  This only sometimes becomes a problem—for instance, if a template wants to output the data set and also some aggregate statistics over it for display before the main output.

Generators can also produce “return” values, which can be fetched by code that knows it is dealing with a Generator after the regular values are produced.  (I might change my mind later, with more experience, but it doesn’t pass the vibe check.  It feels a lot like requiring methods of a class to be called in a specific order, which is usually best to avoid.)  It implies that the entire system should lean into handling Generators in particular, and not allow them to mix with other iterable types.

These are (mostly) things I was vaguely aware of from reading about Python generators, but they weren’t on my mind while writing PHP.

The fiserv Outage

Editor’s Note: this post was penned offline early Friday evening, before the author had knowledge of the issue being resolved, and fiserv processing the backlog as of 16:45.  We have chosen to simply add some links, now that we are online to retrieve them.  The post follows.

As I write, on Friday, 2025-05-02, fiserv has been offline all day, or substantially all day.  This company acts as a third party to a number of banks, providing wire transfers, ACH, and/or direct deposit services, and possibly even online/mobile banking.  A number of large banks, including Ally Bank, Bank of America, Capital One, and Synchrony have been affected in some way by this outage, as was my regional bank.

I don’t know anything about the root causes yet.  It would be irresponsible to speculate about those causes, so of course I am going to.

Every Change Might Be Breaking

We originally had the “automatic minor version upgrade” option active at Amazon RDS.  This option simply does not work very well.  Sometimes, for no clear reason (and without notification), it would stop applying upgrades, and require manual updates to get moving again.  We mostly lived with it, and then we hit the worst case scenario: it did perform the upgrade, and then one of our scripts stopped working.

Not only that, it managed to break while I was on vacation.

(Obligatory xkcd about spacebar heating.)

Since then, we don’t use that option.  When I’m good and ready, I peruse the changelogs, then schedule the update to happen when I will be in the office to handle unexpected issues.

For their part, AWS recommends testing the app against the new version of the database before performing any upgrades.  This is implicitly a recommendation against using automatic minor upgrades, because there is no automated process to test the upgrade first.

One knows an analysis tool is looking at AWS with a security-first paradigm when it recommends switching the automatic upgrade option back on for the database.  It is technically correct that new releases MAY contain security fixes, but upgrading to them MAY cause an automated denial of service.  It is not a simple, inconsequential task.

A Problem of Semantic Versioning

For a while, we’ve been unable to upgrade to PHPUnit 11 due to a conflict in transitive dependencies.  The crux of the problem is:

1. Psalm (5.25.0) directly requires nikic/php-parser: ^4.16, prohibiting 5.x.
2. PHPUnit (11.3.1) transitively requires nikic/php-parser: ^5.1, prohibiting 4.x.

It is possible in the short term to retain PHPUnit 10.x, but it brings to light a certain limitation of Semantic Versioning: it tells you how to create version numbers for your own code base, but it does not carry information about the dependencies of that code.

When the required PHP runtime version goes up, what kind of change is that?  SemVer prescribes incrementing the major number for “incompatible API changes,” or the patch for “backward compatible bug fixes.”

So, is it a bug fix?  Is it incompatible? Or is the question ill-formed?

It feels wrong to increment the patch version with such a change.  Such a release states, “We are now preventing the installation on {whatever Enterprise Linux versions} and below, and in exchange, you get absolutely nothing. There are no new features.  Or perhaps we fixed bugs, but now you can’t access those fixes.”  That sounds… rude.

Meanwhile, it seems gratuitous to bump the major version on a strict time schedule, merely because an old PHP version is no longer supported upstream every year.  It appears to cause a lot of churn in the API, simply because making a major version change is an opportunity to “break” that API.  PHPUnit is particularly annoying about this, constantly moving the deck chairs around.

In between is the feature release.  I have the same misgivings as with the patch version, although weaker.  Hypothetically, a project could release X.3.0 while continuing to maintain X.2.Y, but I’m not sure how many of them do.  When people have a new shiny thing to chase, they don’t enjoy spending any time on the old, tarnishing one.

What if we take the path of never upgrading the minimum versions of our dependencies?  I have also seen a project try this.  They were starving themselves of contributors, because few volunteers want to make their patch work on PHP 5.2–8.1.  (At the PHP 8.1 release in 2021, PHP 5.2 had reached its “end of life” about 11 years prior, four years after its own release in 2006.) Aside from that issue, they were also either unable to pick up new features in other packages they may use, or they were forever adding run-time feature detection.

As in most things engineering, it comes down to trade-offs… but versions end up being a social question, and projects do not determine their answers in isolation.  The ecosystem as a whole has to work together.  When they don’t, users have to deal with the results, like the nikic/php-parser situation.  And maybe, that means users will migrate away from Psalm, if it’s not moving fast enough to permit use with other popular packages.

Our Long-Term AWS CloudSearch Experience

AWS has announced the deprecation of CloudSearch, among other services, just as I wanted to share why we chose it, and how it worked out.

Competitors

The field we considered when choosing CloudSearch included Sphinx, ElasticSearch (the real one and AWS’ ripoff), MySQL FULLTEXT indexes, rolling our own in-SQL trigram search, and of course, CloudSearch.

We had operational experience with Sphinx. It performed well enough, but it is oriented toward documents, not the smaller constellation of attributes I was interested in here.  It took quite a chunk of memory to index our tickets (description/comments), required a pet machine, and didn’t vibe correctly with the team.  I didn’t want to commit to putting 100 times more entries in it, then defending it politically for all eternity.

ElasticSearch appeared to be hyper-focused on log searching specifically, more like what we’re already doing with Papertrail.  It was not clear that it could be used for other purposes, let alone how to go about such things.

We actually had an in-SQL trigram search already, but only for customer names.  I built it because MySQL’s full-text index features were not in great health at the time. (I thought full-text indexes were deprecated ever since, but in checking today, this appears not to be the case.  Even the MySQL 9.0 docs don’t mention it.) I started populating an additional trigram index for all the data I was interested in searching, and it blew up our storage size so fast I had to stop it and definitely find something else. That’s also how I found out that RDS can’t reclaim storage; once it expands, it has expanded for good.

The problem with using MySQL’s full-text indexing was the related integer fields that needed to be indexed.  We wanted to have a general search field, where the user could put in “Sunesh” or “240031” and get the related customer or transaction number, without a complex multi-part form.  Doing that with nothing but MySQL features seemed difficult and/or slow.

“Do nothing” wasn’t really an alternative, either; to search all the relevant fields, MySQL wanted to do two full table scans.  Searches would be running against the largest tables in the database, which makes even a single full scan prohibitively expensive.

CloudSearch

CloudSearch got a great review in my collection of blurbs about AWS services, but further experience has been somewhat less rosy.

For background, CloudSearch is arranged into one-dimensional domains, with a limited second dimension in the form of array attributes.  To contain costs, I chose to index our customers, attaching their VINs as array attributes, rather than have separate customer and vehicle domains or redundantly index the customer attributes on every vehicle.  This results in a domain with 2.5M records.  (Doing some serious guesswork, that means around 12M contracts in total.  Give or take a couple million.)

Things worked fine with a ‘small’ search instance for a while, but it didn’t handle bursty traffic.  Last month, I resized the instance to ‘medium’, and rebuilt the index… which took an unknown number of hours between 2 and 18, inclusive.

Why don’t I know exactly how long it took?  Well, that’s the next problem: metrics. CloudSearch only keeps metrics for three hours, and doesn’t have an event log.  (They appear to go into CloudWatch, but with a custom 3-hour expiration time.) When did the rebuild finish?  Dunno!  Did the system get overwhelmed overnight?  Too bad; that’s gone! With the basic metrics being so anemic, there’s definitely nothing as useful as RDS’ Performance Insights, which is what I would really want here.

Our instance has managed to survive adequately at medium for a while, but I don’t know when I’ll have to scale it up as we roll out this search to more parts of the system.  We just don’t have the metrics here to plan capacity.

Considering that, and the deprecation of it by AWS, I would love to have an alternative… except it would just be CloudSearch, improved.

How I Use Firefox

A long, long time ago, I firewalled my online and real-world identities.  I have separate email accounts for them.  Those email accounts live in separate Firefox profiles. To throw more chaff into the system, the profiles have different adblockers (commonly uBlock Origin; first runner-up is AdBlocker Ultimate) and may or may not include Privacy Possum.

Within the profile, I’ve separated things further into containers, full name Multi-Account Containers.  Google gets its own container, so that YouTube can’t follow me everywhere online.

For things where I suspect all the defenses are a problem, I also have a profile that runs in “always private browsing” mode, but is otherwise fairly open.  I very rarely need it.  I’d rather bounce out of a site that has too many annoyances, and which hasn’t sold me on its usefulness.  (Will I sign up for your newsletter?  Will I create an account to read this article?  No.)

(There’s also Pale Moon and the Windows XP VM with Firefox 52 ESR on it, for checking compatibility with Quilt Draw when I am working on that.  However, those are well outside of everyday usage.)

At work, I don’t use separate profiles; I ended up with entirely separate browsers instead.  My day-to-day work happens in Firefox, with containers for the AWS console, each of our own sites I am responsible for, and “other things requiring login.”  The theory is that a site outside the container(s) that tries to attack one inside will fail, because the login isn’t valid from outside the container.  Meanwhile, the multiple containers separate our sites and our general service providers, and the cloud against everyone.

Because the Google Panopticon Browser is becoming the new IE6, there’s a copy of Edge to make the corporate site(s) “fully supported.”  Let Microsoft spy upon themselves, and only themselves.  All so-called “AI” “features” are turned off, where possible.

Finally, for browsing work-adjacent things that aren’t actually work, like LWN and various blogs, I have LibreWolf.  [Updated 2025-09-25: formerly Waterfox, which wasn’t integrating Firefox security patches fast enough for my liking.] It doesn’t have containers or profiles, but it does have uMatrix (better security by running less remote code), along with LeechBlock so that I don’t waste the whole day in there.

uMatrix is a great defense system; in fact, too great. I wouldn’t recommend it for most people.  However, it suits my goals for that particular browser.

Boring Code Survives

Over on Wandering Thoughts, Chris writes about some fileserver management tools being fairly unchanged over time by changes to the environment.  There is a Python 2 to 3 conversion, and some changes when the disks being managed are no longer on iSCSI, “but in practice a lot of code really has carried on basically as-is.”

This is completely different than my experience with async/await in Python.  Async was new, so the library I used with it was in 0.x, and in 1.0, the authors inverted the entire control structure. Instead of being able to create an AWS client deep in the stack and return it upwards, clients could only be used as context managers.  It was quite a nasty surprise.

To allow testing for free, my code dynamically instantiated a module to “manage storage,” and whether that was AWS or in-memory was an implementation detail.  Suddenly, one of the clients couldn’t write self.client = c; return anymore.  The top-level had to know about the change.  Other storage clients would have to know about the change, to become context managers themselves, for no reason.

I held onto the 0.x version for a while, until the Python core team felt like “explicit event loop” was a mistake big enough that everyone’s code had to be broken.

Async had been hard to write in the first place, because so much example code out there was for the asyncio module’s decorators, which had preceded the actual async/await syntax.  What the difference between tasks and coroutines even was, and why one should choose one over the other, was never clear.  Why an explicit loop parameter should exist was especially unclear, but it was “best practice” to include it everywhere, so everyone did.  Then Python set it on fire.

(I never liked the Python packaging story, and pipenv didn’t solve it. To pipenv, every Python minor version is an incompatible version?)

I had a rewrite on my hands either way, so I went looking for something else to rewrite in, and v3 is in Go.  The other Python I was using in my VM build pipeline was replaced with a half-dozen lines of shell script.  It’s much less flexible, perhaps, but it’s clear and concise now.

In the end, it seems that boring code survives the changing seasons.  If you’re just making function calls and doing some regular expression work… there’s little that’s likely to change in that space.  If you’re coloring functions and people are inventing brand-new libraries in the space you’re working in, your code will find its environment altered much sooner.  The newer, fancier stuff is inherently closer to the fault-line of future shifts in the language semantics.

The Devil is in the Details

We had an initial vision for canceling a contract: choose the contract, input an effective date, and click Cancel.

Then users wanted to get a preview of the cancellation, with all fully-calculated refund values.  We had the preview write everything to the database, and if the user confirmed it, we would mark the cancellation as “complete.”  A user could also “revoke” cancellation, which would delete the pending cancellation.

Soon, we had a system to recover abandoned cancellations by reminding users they had one pending, and auto-revoke it after a deadline.

Then it became clear that there could be race conditions.  What if the cancellation is processed between receiving the notification and following the link enclosed?  What if someone else was running the cancellation on the same contract simultaneously, and ended up with the same results?  We needed to show the new status.

What if someone made a mistake with the inputs?  We needed an “Edit” button that would go back, as well.  If there’s an Edit button, and someone comes in from the main menu, should we reload the inputs, or skip the input screen entirely?

It wasn’t long before Accounting wanted a “reinstate” button.  Suddenly, the “final” state was no longer final.  In the mean time, we had integrated automatic data pushes to the actual accounting software, which meant a huge mess if they wanted to reinstate one that was on the official record as canceled.

If there’s a moral to this, it’s that any software pipeline should be fully reversible, even into the parts where “we sent money out, and we have to ask for it back as a consequence.”

An Incomplete API: PSR-7 and PSR-18

Consider some code that is using http-factory-discovery to avoid depending directly on specific PSR-7 and PSR-18 implementations. However, the concrete classes may default to an infinite request timeout, in an environment where external forces (AWS ALB) will time out in 60 seconds. Therefore, in order to return a nicer error message, or get a chance to retry, one wishes to set a timeout like 25 seconds on the request.  Can it be done?

Not really!

A PSR-18 HTTP client has an interface consisting of a single defined method: sendRequest(RequestInterface $request): ResponseInterface.  There is no concept of options here.  The only hope would be to carry them as attributes on the request.

Unfortunately, there are no standard request attributes for HTTP options like this.  PSR-7 defines the attribute mechanism, but is silent about its usage. On the “real world usage” side, Guzzle 7 does not define any way to carry HTTP client options on the Request.

This makes the APIs easier to implement, but leaves a gap in them.  The only way out is to use a specific, concrete implementation.  And then what is the point of the discovery library?  If the application must contain its own code to work with underlying libraries, then all discovery does is add the possibility of returning an unrecognized type.  At that point, the app can only choose between running with degraded features, and crashing entirely.

Why I use ext4

Re: Why I (still) use ext4 for my Linux root filesystems

I use ext4 for all of my Linux filesystems.  It’s safety in numbers: Files are fraught with peril notes that it is tricky to support different journal modes on ext4, let alone different filesystems.  btrfs may reorder directory operations, which other filesystems don’t.

So, it seems to be safer to make the expected choice.  For me, using Ubuntu, that’s ext4 in ordered mode.

Does it lock me into the decisions of the past?  I don’t know.  The filesystem developers could have “done it wrong,” but ext4 implemented extent-based allocation tracking, reaching feature parity with other common filesystems of the time.  That was probably the whole reason to raise the version number.

The performance story is pretty much a wash.  Whether one filesystem beats another or not, the numbers are typically pretty close.  In what is hopefully closer to real-world usage, I failed to find a difference in VM boot time using ext4 vs xfs on the file system.  If I’m not going to be running filesystem benchmarks as my primary workload, and the benchmark performance doesn’t translate to a real-world effect, then why bother?

I also don’t take snapshots for backups; I’m interested in surviving a complete disk failure. I’ve lost two disks that way, although one had the decency to limp along just long enough to get the important parts of /home copied from it.  Backups are now “I copy important data to a USB disk.”  One of those disks is only rarely connected, for ransomware resistance.

The sea change to vendoring and containers

Not so long ago, we would install code to the global runtime path, where it would be available machine-wide.  CPAN was perhaps the earliest to do this, but PEAR, rubygems, and probably others followed.  After all, CPAN was seen as a great strength of Perl.

Users of such a library would say something like use Image::EXIF; or include_once "Smarty/Smarty.class.php"; Notice that these are a mechanism to follow some include path to resolve relative filenames.

But as servers got larger faster than applications, co-installing those applications became attractive.  The problem was that they can have different dependency requirements, which complicates the management of a shared include path.

This is about where Bundler happened for Ruby, and its ideas were brought to other ecosystems: Perl got carton, and PHP got composer. These systems build a directory within the project to hold the dependencies, and basically ignore the global include path entirely.

At the cost of disk space, this bypasses the problems of sharing dependencies between projects.  It bypasses all the problems of working with multiple versions of dependencies that may be installed in different hosts’ include paths.  It also makes the concept of the language’s include path obsolete, at least in PHP’s case: the Composer autoloader is a more-capable “include path” written in code.  Finally, it allows piecemeal upgrades—a small service can make a risky upgrade and gain experience, before a larger or more important service on the machine needs to upgrade.

Piecemeal upgrades also enable independent teams to make more decisions on their own.  Since their dependencies are not shared with other users, changes to them do not have to be coordinated with other users.

Containers are the next step in the evolution of this process.  Pulling even more dependencies into a container brings all the advantages of local management of packages into a broader scope.  It allows piecemeal upgrading of entire PHP versions.  And it makes an application cost an order of magnitude more space, once again.

In our non-containerized environment, I can switch the PHP version on a host pretty easily, using co-installable versions and either update-alternatives (for the CLI) or a2disconf/a2enconf (for FPM), but this means the services do not really have a choice about their PHP version.  It has been made before the application's entry point is reached.

Containers over systemd

“Systemd will solve all your problems,” they said.

Having used a number of systemd’s security features to configure a service, I am beginning to suspect everyone uses containers because container runtimes are trying to be secure already.

It's possible to improve the security of a service with systemd, of course.  I’ve worked hard at it.  But in the end, over half the *.service file is consumed with “trying to build my own container out of systemd directives.”  ProtectHome, ProtectSystem, ProtectKernelTunables, Protect This, Protect That, blah blah blah.  The process starts from insecure by default, and then asks me to layer on every individual protection.  This is exactly the sort of thing Linux zealots used to yell at Microsoft about.  ¯\_(ツ)_/¯

But I digress.  I ended up with an excessively long systemd service configuration file, and to apply that to any other service, there’s no option besides copying and pasting those directives.  With every release of systemd, I have to comb the man pages again to see what else is available now, and carefully apply that to every service file.  It’s not easy to tell whether the security posture is up-to-date when the policy is so verbose.

[Updated 2022-03-19: systemd-analyze security foo.service is your friend. This is the best way to get a list of everything systemd thinks about security, and whether it is applied to the unit. It's a little less bad than I thought, but it is still fundamentally the Default Permit and Enumerating Badness approaches.]

Whereas a container has an isolated filesystem (its image) already, so whole classes of configuration (ProtectHome, ProtectSystem, TemporaryFileSystem) become irrelevant.  On top of that, container runtimes start with a more limited set of privileges by default, instead of handing out CAP_SYS_ADMIN and leaving it up to the administrator to carefully disable it.  Escaping from the container runtime is considered a vulnerability; escaping from a poorly-secured systemd service is considered user error.

This is all orthogonal to “containers are interop”, but I think both forces are feeding the containerization craze.  I’m left with the feeling again that systemd should have been the “obvious correct choice,” except they decided usability didn’t matter.

Informative Shell Prompts

Recently, /u/NotSoButFarOtherwise wrote about Starship, a cross-shell prompt generator:

If you really forget your current directory, username, or host, there are commands to remind you. But you'd probably be surprised how often you don't.

I tried it, but I immediately noticed, I like having the context available in the shell history.  If I get interrupted and jump directories a couple of times to run a few brief commands, I rely on my prompt from a few lines or a few screens back to remind me of what I was doing.

Multi-tasking isn’t ideal, yet remains inescapable, so I don’t want to strip away all of my assistive configurations.  It reminds me a lot of my experience with returning to vim after using VS Code for 1-3 months: “You know what, having more information was actually rather useful.”

Deployment May Be Stateful

Our deployment process can technically accept a commit hash or an alternate branch to deploy, but by default, it updates to the currently checked-out branch tip. This default also applies to the auto-update code that brings our pre-baked AMI up-to-date when it launches.

For the most part, this is fine.  We keep master in a deployable state, and that’s always the desired version to deploy.  Thus, the whole system is stateless…

But, it also means that we can’t use our fancy “change branch” or “deploy commit” operations very much.  If we do, then the desired version is no longer what the AMI will auto-deploy when new instances launch from it.  We have to either build a new AMI (for the branch) or restore the deployability of master before any new instances launch.

If we reach the “deploy from tarball” goal, then life would be easier.  Builds could happen from any branch or commit naturally, and we could prevent a broken tarball from auto-deploying by simply deleting it.

systemd: the house built on sand

Once upon a time, supervisord got the service management done, but I never got the logs stored anywhere sensible.  Eventually, I got tired of being tapped to debug anything that had an obvious error, but where the message was only logged by supervisord.

Thus began a quest for running things through the distribution’s init system, which has given me some experience with upstart and a lot of experience with systemd.  Like most software that reaches success, systemd has not been carefully designed and implemented.  It has only accumulated, organically.

This is nowhere more obvious than in the configuration system.  I can’t just read documentation online, write a .service file, and expect it to work; I have to use the online search to find which man page they hid the relevant directives in, and spin up a VM to read it.  Once I find the directives that apply, it’s obvious that we have an INI system crying out to be a more imperative, stateful, and/or macro-driven language.

Those are related; because the configuration is underpowered, new capabilities require new tweaks.  Consider the number of “boolean, or special string value” options like ProtectHome and ProtectSystem: these were clearly designed as booleans and then extended later.

Because the website doesn’t keep a changelog—everything changes so fast, systemd just has a major version and every release is breaking—it’s not easy to build a cross-platform service definition file that takes advantage of the features systemd offers.  You know, the things that make it unique from other init systems.  The things that were supposed to be selling points.

It’s because everything changes at the whim of the developers.  Stable? Backwards-compatible, at least?  In a fundamental system component?

Big nope from the systemd team.  There are at least a few directives that were superseded, and so it’s impossible to make a portable service description for a service that is otherwise portable. And the lack of past-proofing tells us about future-proofing.  What you write today may simply not run tomorrow.

systemd was supposed to be the obvious, easy choice: in theory, it embraced Linux and cgroups so that administrators could use cgroups to provide isolation without a separate containerization layer.  But in practice, the separate layer is looking ever more like a better choice.