A Few More Words about LLMs for Coding

One of the clear and present dangers to LLM coding is to produce a codebase which can only be operated on by LLMs. Shall we be so eager to have the corporations make us an offer we can’t refuse?

Using LLMs Again

They insisted on selling out the future for dubious short-term interests again #capitalism, so about three million tokens in, I have more thoughts on coding with an LLM.

It’s Chaotic

The model has its strengths and weaknesses, but it can be hard to predict how a specific task will fit.

It went great to upgrade an internal site from Bootstrap 3 to 4 to 5.  It only did okay at dark mode for it.  The machine simply does not know what has contrast and what does not.  I spent a long time asking it for updates on a component-by-component basis.

And sometimes, it just outright makes mistakes.  My first-ever fix for an LLM-generated bug was for some text disappearing from the website, because it had transformed something of the form display(error ? err_msg : text) around to if (error) { display(err_msg); } during the process.  It quit displaying text for the normal/success case.

On a different project, I accidentally clipped its wings by not having the vendor directory installed, and it hallucinated some atrocious code.  The model “not knowing what it didn’t know” greatly hampered its ability to proceed… and it didn’t know that, either.  It didn’t stop and ask for the problem to be fixed.  It just slopped some garbage out.

On a third project, it perfectly generated a GitHub workflow for “build an ECR image on push”, and then flopped on its face with a manual workflow for “deploy such an ECR image into ECS”.  It minimized IAM permissions, blissfully unaware that ecs:DescribeTasks does not use a resource tag.  That one action must be given permissions on resource *, even to describe a specific task that is known in advance.  Faced with the error, it shuffled code around to do the same operation a different way, which also failed.  The human had to track it down in the AWS Console and documentation.

(I asked it to store the IAM policies in the repo for reference.  I do not plug the LLM into AWS, GitHub, MySQL, the bastion host, a web browser, or even git fetch.)

It was pretty good at finding differences between an old system and a new one, but less effective at porting the missing features across.  Most of the time was spent on the human working out the tangled mess of the most difficult pieces.  In the aftermath, closer human review observed 27% defective commits had been made.

An equal number of commits were “not how I would code it,” which is also something that bothers me.  It is my name that goes onto the commit, and will show up in git blame later.  Those also got patched up.

Secure Code is an Afterthought

Even with a strong pattern of CSRF and Allow header mitigations (i.e. a couple of function calls in the setup), it was not able to generate code to handle these concerns.  While it probably knows how to set up a popular framework like Symfony or Laravel to do it, it is not able to learn the pattern in our own ancient code.

It might be no better than other developers on the team at XSS, but that is concerning in both directions.  I don’t want either of them introducing div.innerHTML = htmlStr!  String concatenation is a security vulnerability in systems like this.

When generating some code for the GitHub workflows, it produced a command of the form THING=$(...) and then proceeded to use $THING without checking that it actually got any output in there.  For shell, it’s always best to back out as soon as it starts into the weeds.

Good Prompts Take Knowledge

It saves time and money to point the LLM directly at .svcPop instead of describing “the service redemption bubble” and making the machine thrash around, running half a dozen ripgrep commands to try to find it.

On the other hand, making changes with an LLM can quickly erode one’s low-level understanding of the code, making attempted “good” prompts into not-so-good ones.  When I’m not the one making the changes, I lose understanding and effectiveness.  If I’m using it to write using some new libraries, like Pest or AmPHP/Revolt, I am also losing both depth of learning and retention of what’s left.  I can’t ambiently absorb knowledge from documentation I am not looking at.

(And as we saw, even if the prompt is good, the results may not be.)

Narrow Focus is Double Edged

The narrow focus on the task at hand is what makes the LLM useful at what it is doing, but it is also what builds technical debt.  It’s happy to generate all-new CSS for anything it does, without worrying about whether any of it can be a shared concept across the codebase.

Whether I’m reading or writing the code, I’m thinking about this stuff.  It was me who noticed the multiple ‘loading’ spinner images.  When asked to replace all of them, the LLM generated a gif (?) of a non-theme color, that wasn’t animated (???).  Then, it copied that over all four files (including the two unused ones), corrupting the layout where the smaller file had been used, and declared it done.  Oh, and the gif assumed a white background, on a site that already had a dark mode.  I threw up my hands dramatically, tracked down an SVG, and fixed it all myself.

Meanwhile, its chaotic nature makes it somewhat random which CSS features will be used.  This is especially noticeable for things like choosing between repeating selector prefixes, or nesting the blocks.  It’s 2026 and nested CSS is Baseline 2023, so it’s not like this is going to break anything accessing the site, but it immediately raises questions about how to control the CSS feature usage for sites where we don’t have as much latitude in dictating choice of browser.

But When It’s Good, It’s Good

When I’m using the LLM on a task that it is good for, making quick work of some long-delayed upgrade or feature request, the feeling approaches what others describe as manic.  With all the downsides that entails, too: the selfishness, the hubris, and the possibility that it will turn out to be a complete waste after all.

But its siren song is sparkling and effervescent.

This iteration of models is good enough to see why people like it.

Tech Can’t Solve Social Problems

The code I upgraded to Bootstrap 5 had been on Bootstrap 3 because we aren’t spending any time for maintenance in the constant rush toward “more features."  Nothing can be done until the threat becomes existential.  I’m worried that higher levels of the company will soon see the LLMs as a way to continue this business-as-usual approach.  The developers “have this tool to be more productive,” so we can expect more features, sooner.

I also don’t know if management has visibility into the LLM usage, to understand what they’re actually getting for their money.  It’s entirely possible that such information is only available to someone who may or may not be using the corporate account’s resources for personal projects.  Actually worrying about this is above my pay grade, especially since I have no evidence whatsoever, but still: it’s an obvious potential weakness.

The Chaos Demon

Sometimes, the LLM simply doesn’t follow the prompt.  Or accept correction.  The only thing to do is to Ctrl+C and try anew.

And push down the thought of a world with dangerous equipment going rogue like this.  Self-driving cars.  Industrial equipment.  Weapons nominally in the hands of ICE or the police.

The System Is the Authority

This is the first post for Decoded Node that has been written with league/commonmark instead of erusev/parsedown.  This technically aligns the Markdown implementation between this blog and my personal website (where CommonMark was readily integrated with Twig.)  Which is fancy talk for:

• Footnote-style link syntax works on both sites now
• Smart quotes are automated, instead of using platform-dependent input methods, or UniCycle

UniCycle “works,” but it’s not maintained (AFAICS), and the implementation strikes me as a bit of a hack.  It works fine in insert mode, but r' won’t replace with smart quote.  It’s a shame, because it has such a great name.

(Also along the way, I decided that .md won, so this processor no longer accepts anything like .mkd or .mdown.  Makes things easier than asking, “does this file match /\.ma?r?k?do?w?n$/?”)

Didn’t the Title Mention… Authority?

What strikes me about this is that “the format which I author in” is only incompletely described as “Markdown."  This particular move was feasible because league/commonmark is effectively a superset of erusev/parsedown.  If I decided I wanted to go back, I would have to edit all the new features out of newer posts.  Actually, if I want to update an old post and I happened to miss a smart quote in it, the non-smart quotes would also be changed by using the new parser.

What actually defines the output is not only the input, but the exact tool used to create it.

(I’m thrashing around in the direction of “the purpose of a system is what it does” and Hyrum’s Law, but neither of those things fit.)

That wouldn’t be much of an observation if I were saying, “An XLST processor doesn’t turn my Markdown into HTML,” but I didn’t really expect the choice of Markdown tool to matter so much to processing Markdown input.

Maybe it could be phrased as, “Markdown processors are not bug-for-bug compatible.”

Version Note

I use this code to write posts, not to mess around with code, so it’s not always clear whether the Markdown parser is up-to-date.  However, it seems I was using Parsedown 1.7.4 which is, at the time of writing this post, the latest stable release.  1.8.0 is in beta, and 2.0.0 seems to be barely started.

AI Erodes Knowledge

If one has an LLM “do the work,” then one does not actually learn anything.  Taking notes improves recall, and so does trying to remember the answer before looking something up.  In any case, to become more proficient, something has to happen inside the mind. If we skip that part in our rush to produce volume of output, we are trading away our future skills.

“Not gaining proficiency” would be indictment enough, but worse, unused skills decay. There is a reason that C is not on my résumé anymore!  I can’t imagine what would become of my PHP if I didn’t do the coding, and I utterly dread the potential outcome of using an LLM to make changes for an entire month.

An LLM will never produce original work, since it is trying to “predict” based on a corpus of past, public work.  It’s great for writing code like fanboys on the internet write code.  It’s not so great at cost optimization.

Overall, I wonder about how institutional knowledge might be affected by heavier LLM usage.  If we’re not doing the thinking, will we be able to remember anything about our own history?

Early Thoughts on Coding with an LLM

I temporarily displaced my misgivings about the Plagiarism Machine That Also Destroys Earth.  Since I haven’t yet spent a full week working with it, I have some first impressions only:

1. All generators have the highest accuracy on the smallest tasks, where they are the least useful.
2. The tools all seem to prefer the same model, so there’s less differentiation than one might hope for. Nothing is worse… but nothing is better.
3. Writing a good prompt takes a lot of planning.
4. Every change must be tediously reviewed.  This includes every tab-completable inline suggestion in the editor.
5. Junie is not actually integrated very deeply into PHPStorm.
6. Agent mode is full-on Sorcerer’s Apprentice.  One must always be standing by on the “Emergency Stop – Never Use” button.

Due to homogenous model choices, UI is an important differentiator right now.  Using Control+Backslash to generate code in PHPStorm makes for results that are difficult for me to understand, because the diff is in character mode.  Cursor is much better at this, with line diffs.

As for Junie’s lack of integration, it failed to recognize «run the "foo api unit tests"» as an instruction to run the pre-existing test configuration named “foo api unit tests”.  I let it try running foo api unit tests in the terminal to see what it would do.  After getting a command-not-found error, it attempted to find any test that it could run, and try running that instead.  Fortunately, it needed permission to run further off the rails, so I denied it.

Summary

As for the overall experience…

The LLM removes the fun part of programming, the writing of the code, leaving the planning and debugging parts I am less fond of.  The incessant demands for attention from inline suggestions also fundamentally block entering a flow state. Meanwhile, hallucinations are always trying to stab me in the back; there is absolutely no meta-analysis of whether the prompt itself is misguided.

I don’t even see these tools as useful for exploring unfamiliar code.  My IDE already has a set of tools for that; like Find in Files, Go to Definition, and Find References/Usages.  Since these aren’t constrained to a sidebar, the results are also far more usable.

Even if it had no downsides, at first blush, the (paid) systems still rate a solid “meh.”

Thoughts from Trying Generators in PHP

I am late to the party, but I have been playing with Generators in PHP more, and running into the limitations of module boundaries.

Some module might produce a Generator so that iteration can be performed in chunks, reducing peak RAM.  For example, producing results one store at a time, instead of loading up all stores into a giant array.  Code that processes an entire database table, but wants to lower lock contention and memory use can also benefit; it can use a Generator to isolate the fetch-in-pages logic from processing the individual records.  The consumer sees one stream of results, while the Generator fetches more as needed.

In short, there are plenty of use cases.

The problem comes when a caller wants to pass “the data” produced by the Generator to another function or method that specifically takes an array.  Once that happens, either the destination needs to be reworked to accept the broader iterable type, or the efforts toward efficiency are erased by an iterator_to_array() call.

(Of course, back when generators were introduced to PHP, I didn’t use type declarations, so I could have gotten away with throwing a generator at something that assumed it would receive an array or PDOStatement. Dealing with larger teams and beginning to use an IDE were both great reasons to add the type information, and the array type forbids passing a Generator in its place.)

A separate issue is that anything consuming a Generator (thus, anything type-hinted iterable) needs to be aware of its once-only nature.  This only sometimes becomes a problem—for instance, if a template wants to output the data set and also some aggregate statistics over it for display before the main output.

Generators can also produce “return” values, which can be fetched by code that knows it is dealing with a Generator after the regular values are produced.  (I might change my mind later, with more experience, but it doesn’t pass the vibe check.  It feels a lot like requiring methods of a class to be called in a specific order, which is usually best to avoid.)  It implies that the entire system should lean into handling Generators in particular, and not allow them to mix with other iterable types.

These are (mostly) things I was vaguely aware of from reading about Python generators, but they weren’t on my mind while writing PHP.

Some Solutions to a Problem

We have an EC2 instance that has a quickly-produced shell script that runs on boot.  It sets a DNS name to the instance’s public IPv4 address.  Since time was of the essence, it hard-codes everything about this, particularly, the DNS name to use.

This means, if we want to bring up a copy of this instance, based on a snapshot of its root volume, the copied instance will overwrite the DNS record for the production service. We need to stop this.

As a side project, it would be nice to remove the hard-coding of the DNS name.  It would be trivial to “stop DNS name conflicts” if we did not have a DNS name stored on the instance’s disk image to begin with.

What are the options?

A Problem of Semantic Versioning

For a while, we’ve been unable to upgrade to PHPUnit 11 due to a conflict in transitive dependencies.  The crux of the problem is:

1. Psalm (5.25.0) directly requires nikic/php-parser: ^4.16, prohibiting 5.x.
2. PHPUnit (11.3.1) transitively requires nikic/php-parser: ^5.1, prohibiting 4.x.

It is possible in the short term to retain PHPUnit 10.x, but it brings to light a certain limitation of Semantic Versioning: it tells you how to create version numbers for your own code base, but it does not carry information about the dependencies of that code.

When the required PHP runtime version goes up, what kind of change is that?  SemVer prescribes incrementing the major number for “incompatible API changes,” or the patch for “backward compatible bug fixes.”

So, is it a bug fix?  Is it incompatible? Or is the question ill-formed?

It feels wrong to increment the patch version with such a change.  Such a release states, “We are now preventing the installation on {whatever Enterprise Linux versions} and below, and in exchange, you get absolutely nothing. There are no new features.  Or perhaps we fixed bugs, but now you can’t access those fixes.”  That sounds… rude.

Meanwhile, it seems gratuitous to bump the major version on a strict time schedule, merely because an old PHP version is no longer supported upstream every year.  It appears to cause a lot of churn in the API, simply because making a major version change is an opportunity to “break” that API.  PHPUnit is particularly annoying about this, constantly moving the deck chairs around.

In between is the feature release.  I have the same misgivings as with the patch version, although weaker.  Hypothetically, a project could release X.3.0 while continuing to maintain X.2.Y, but I’m not sure how many of them do.  When people have a new shiny thing to chase, they don’t enjoy spending any time on the old, tarnishing one.

What if we take the path of never upgrading the minimum versions of our dependencies?  I have also seen a project try this.  They were starving themselves of contributors, because few volunteers want to make their patch work on PHP 5.2–8.1.  (At the PHP 8.1 release in 2021, PHP 5.2 had reached its “end of life” about 11 years prior, four years after its own release in 2006.) Aside from that issue, they were also either unable to pick up new features in other packages they may use, or they were forever adding run-time feature detection.

As in most things engineering, it comes down to trade-offs… but versions end up being a social question, and projects do not determine their answers in isolation.  The ecosystem as a whole has to work together.  When they don’t, users have to deal with the results, like the nikic/php-parser situation.  And maybe, that means users will migrate away from Psalm, if it’s not moving fast enough to permit use with other popular packages.

The Missing Call

I decided to combine (and minify) some CSS files for our backend administration site, so I wrote the code to load, minify, and output the final stylesheet.  I was very careful to write to a temporary file, check even the fclose() return code, rename it into place, and so on.  I even renamed the original to a backup so that I could attempt to rename it back if the first rename succeeded, but the second failed.

For style points, I updated it to set the last-modified time of the generated file to the timestamp of the latest input, so that If-Modified-Since headers will work correctly.

I tested it, multiple times, with various states of having the main and backup filenames. It looked great.  I pushed it out to production… and that wasn’t so great.

We just had no styles at all. Yikes!  I had some logic in there for “if production and minified CSS exists, use it; else, fall back to the source stylesheets.”  I hastily changed that to if (false) and pushed another deployment, so I could figure out what happened.

It didn’t take long.  The web server log helpfully noted that the site.min.css file wasn’t accessible to the OS user.

I had used tempnam(), which created an output file with mode 600, rw- --- ---.  Per long-standing philosophy, the deployment runs as a separate user from the web server, so a file that’s only readable to the deployer can’t be served by the web server.  Oops.

I had considered the direct failure modes of all of the filesystem calls I was making, but I hadn’t considered the indirect consequences of the actions being performed.  I added a chmod(0o644) call and its error check, and deployed again.  After that, the site worked.

fopen() modes vs. Unix modes

PHP has a function for creating temporary files, tempnam. One limitation is that it only takes a filename prefix, and most often, I want to have a file “type” as the suffix, like “report-20240701-RANDOM.csv”.

new SplTempFileObject() creates a file in memory, which isn’t usable for anything where an actual “file on disk” is required.  The related tmpfile() function does not give access to the file name itself.

Meanwhile, fopen() and new SplFileObject() don’t offer control of the Unix permissions.  We can create files in exclusive-write mode by setting the mode to argument to “x”, but we can’t pass 0o600 (rw- --- ---) at that stage.  We have to create the file, and if it works, call chmod() separately.

fopen() and anything modeled on it offer a context parameter, but there are no context options for the file: scheme, only for other stream wrappers.

Underneath fopen()—at least on Linux—is the open syscall.  That call accepts a mode_t mode argument, to indicate what Unix permissions to use when creating a file, which is exactly what we are after.  But thanks to history and standards, we can’t access that directly from PHP now.

P.S.: there’s actually another possibility: we can rename() the file from tempnam() to add a suffix in the same directory.  If an attacker can observe our original file and create the target file with something unexpected, then the rename() will fail.  If tempnam() didn’t give us the permissions we wanted, though, we’d be out of luck with that, and it’s still a two-step process.

Sorted by What?

Shortened for illustrative purposes, I came across some ancient code of the form:

SELECT DATE_FORMAT(o.created_at, '%c/%e/%Y') AS dt,
    DATE_FORMAT(c.updated_at, '%l:%i %p') as tm,
    …
FROM orders o JOIN customers c … WHERE …
ORDER BY c.last_name, dt DESC, tm;

The ORDER BY dt caught my eye because it’s not an actual column in any table. Its value turns out to be the American-style “6/23/2024” format, which is reasonable to display, but completely wrong to sort on.  Doing that puts October prior to February, as “10” begins with “1”, which is less than “2”.

I cannot guess why it pulls the time from an unrelated column as tiebreaker, sorting it the other direction.  The rest of the issues are likely for the same reasons as the date.

I assume the chaotic arrangement of orders within a customer was never raised as a concern only because duplicating orders would be rare enough—ideally, never happening—that it didn’t matter.

Nonetheless, I queued a change to sort on the full date+time held in created_at, so that records will be fully chronological in the future.

Everything Fails, FCGI::ProcManager::Dynamic Edition

I have been reading a lot of rachelbythebay, which has led me to thinking about the reliability of my own company’s architecture.

It’s top of my mind right now, because an inscrutable race condition caused a half-hour outage of our primary site.  It was a new, never-before-seen condition that slipped right past all our existing defenses.  Using Site as a placeholder for our local namespace, it looked like this:

use Try::Tiny qw(try catch);
try {
  require Site::Response;
  require Site::Utils;
  ...;
} catch {
  # EX_PRELOAD => exit status 1
  exit_manager(EX_PRELOAD, $_);
};
...;
$res = Site::Response->new();

Well, this time—this one time—on both running web servers… it started throwing an error that Method "new" wasn't found in package Site::Response (did you forget to load Site::Response?).  Huh?  Of course I loaded it; I would’ve exited if that had failed.

In response, I added a lot more try/catch, exit_manager() has been improved, and there is a separate site-monitoring service that will issue systemctl restart on the site, if it starts rapidly cycling through workers.

Boring Code Survives

Over on Wandering Thoughts, Chris writes about some fileserver management tools being fairly unchanged over time by changes to the environment.  There is a Python 2 to 3 conversion, and some changes when the disks being managed are no longer on iSCSI, “but in practice a lot of code really has carried on basically as-is.”

This is completely different than my experience with async/await in Python.  Async was new, so the library I used with it was in 0.x, and in 1.0, the authors inverted the entire control structure. Instead of being able to create an AWS client deep in the stack and return it upwards, clients could only be used as context managers.  It was quite a nasty surprise.

To allow testing for free, my code dynamically instantiated a module to “manage storage,” and whether that was AWS or in-memory was an implementation detail.  Suddenly, one of the clients couldn’t write self.client = c; return anymore.  The top-level had to know about the change.  Other storage clients would have to know about the change, to become context managers themselves, for no reason.

I held onto the 0.x version for a while, until the Python core team felt like “explicit event loop” was a mistake big enough that everyone’s code had to be broken.

Async had been hard to write in the first place, because so much example code out there was for the asyncio module’s decorators, which had preceded the actual async/await syntax.  What the difference between tasks and coroutines even was, and why one should choose one over the other, was never clear.  Why an explicit loop parameter should exist was especially unclear, but it was “best practice” to include it everywhere, so everyone did.  Then Python set it on fire.

(I never liked the Python packaging story, and pipenv didn’t solve it. To pipenv, every Python minor version is an incompatible version?)

I had a rewrite on my hands either way, so I went looking for something else to rewrite in, and v3 is in Go.  The other Python I was using in my VM build pipeline was replaced with a half-dozen lines of shell script.  It’s much less flexible, perhaps, but it’s clear and concise now.

In the end, it seems that boring code survives the changing seasons.  If you’re just making function calls and doing some regular expression work… there’s little that’s likely to change in that space.  If you’re coloring functions and people are inventing brand-new libraries in the space you’re working in, your code will find its environment altered much sooner.  The newer, fancier stuff is inherently closer to the fault-line of future shifts in the language semantics.

On Handling Web Forms [2012]

Editor’s Note: I found this in my drafts from 2012.  The login form works as described, but few of the other forms do.  However, the login form has not had glitches, even with 2FA being added to it recently.  The site as a whole retains its multi-page architecture.  Without further ado, the original post follows…

I’ve been working on some fresher development at work, which is generally an opportunity for a lot of design and reflection on that design.

Back in 2006 or so, I did some testing and tediously developed the standard “302 redirect on POST responses” technique for preventing pages that handled inbound form data from showing up in the browser history.  Thus, sites would be Back- and Reload-friendly, as they’d never show that “About to re-submit a form, do you really want to?” box.  (I would bet it was a well-known technique at the time, but NIH.)

That’s pretty much how I’ve written my sites since, but a necessary consequence of the design is that on submission failure, data for the form must be stored “somewhere,” so it can be retrieved for pre-filling the form after the redirection completes.

My recent app built in this style spends a lot of effort on all that, and then throws in extra complexity: when detecting you’re logged out, it decides to minimize the session storage size, so it cleans up all the keys.  Except for the flash, the saved form data, and the redirection URL.

That latter gets stored because I don’t trust Referer as a rule, and if login fails and redirects to the login page for a retry, it won’t be accurate by the time a later attempt succeeds.  So every page that checks login also stores the form URL to return to after a login.

There’s even an extra layer of keys in the form area, so that each form’s data is stored independently in the session, although I don’t think people actually browse multiple tabs concurrently.  All that is serving to do is bloat up the session when a form gets abandoned somehow.

Even then, it still doesn't work if the form was inside a jQueryUI dialog, because I punted on that one.  The backend and page don’t know the dialog was open, and end up losing the user’s data.

Simplify, Young One

That's a whole lot of complexity just to handle a form submission.  Since the advent of GMail, YouTube, and many other sites which are only useful with JavaScript enabled, and since this latest project is a strictly internal app, I've decided to throw all that away and try again.

Now, a form submits as an AJAX POST and the response comes back.  If there was an error, the client-side code can inform the user, and no state needs saved/restored because the page never reloaded. All the state it had is still there.

But while liberating, that much is old news.  “Everyone” builds single-page apps that way, right?

But here’s the thing: if I build out separate pages for each form, then I’m effectively building private sections of code and state from the point of view of “the app” or “the whole site.”  No page is visible from any other, so each one only has to worry about a couple of things going on in the global navbar.

This means changes roll out more quickly, as well, since users do reload when moving from area to area of the app.

The Devil is in the Details

We had an initial vision for canceling a contract: choose the contract, input an effective date, and click Cancel.

Then users wanted to get a preview of the cancellation, with all fully-calculated refund values.  We had the preview write everything to the database, and if the user confirmed it, we would mark the cancellation as “complete.”  A user could also “revoke” cancellation, which would delete the pending cancellation.

Soon, we had a system to recover abandoned cancellations by reminding users they had one pending, and auto-revoke it after a deadline.

Then it became clear that there could be race conditions.  What if the cancellation is processed between receiving the notification and following the link enclosed?  What if someone else was running the cancellation on the same contract simultaneously, and ended up with the same results?  We needed to show the new status.

What if someone made a mistake with the inputs?  We needed an “Edit” button that would go back, as well.  If there’s an Edit button, and someone comes in from the main menu, should we reload the inputs, or skip the input screen entirely?

It wasn’t long before Accounting wanted a “reinstate” button.  Suddenly, the “final” state was no longer final.  In the mean time, we had integrated automatic data pushes to the actual accounting software, which meant a huge mess if they wanted to reinstate one that was on the official record as canceled.

If there’s a moral to this, it’s that any software pipeline should be fully reversible, even into the parts where “we sent money out, and we have to ask for it back as a consequence.”

An Incomplete API: PSR-7 and PSR-18

Consider some code that is using http-factory-discovery to avoid depending directly on specific PSR-7 and PSR-18 implementations. However, the concrete classes may default to an infinite request timeout, in an environment where external forces (AWS ALB) will time out in 60 seconds. Therefore, in order to return a nicer error message, or get a chance to retry, one wishes to set a timeout like 25 seconds on the request.  Can it be done?

Not really!

A PSR-18 HTTP client has an interface consisting of a single defined method: sendRequest(RequestInterface $request): ResponseInterface.  There is no concept of options here.  The only hope would be to carry them as attributes on the request.

Unfortunately, there are no standard request attributes for HTTP options like this.  PSR-7 defines the attribute mechanism, but is silent about its usage. On the “real world usage” side, Guzzle 7 does not define any way to carry HTTP client options on the Request.

This makes the APIs easier to implement, but leaves a gap in them.  The only way out is to use a specific, concrete implementation.  And then what is the point of the discovery library?  If the application must contain its own code to work with underlying libraries, then all discovery does is add the possibility of returning an unrecognized type.  At that point, the app can only choose between running with degraded features, and crashing entirely.

The Pace of Change

I’m not the first, nor the only, person to complain about the pace of technical change.  But what are the actual problems?

We risk losing perspective.  We will forget that the fad of today is just another fad; blockchains and containers are destined to be the next XML, relatively soon in their life, then carried forward for thirty years because changing infrastructure is too risky for the business.

We risk losing the wisdom of the past, assuming even our own younger selves were but naked savages, coding in Perl or PHP. We will not know what made Perl, Perl; we will not bring any of the good ideas forward.

Truly, we risk losing experts. It took me a good 10 or 15 years to really appreciate the sheer amount of knowledge that makes an expert, an expert; if we burn our world down every five years, then we will never come to know anything deeply.  We will have no experts.

Where I used to worry about becoming “a dinosaur,” it now seems that dinosaurs going extinct are the larger problem.

But what is the actual problem?

Pride, perhaps?  Are we too snobby to learn about what came before, to understand our place in history, and to meet the present where it’s at?  Do we think there is nothing to learn about a system in reading its code, in making improvements to it, that we must replace it outright?

Is it ignorance?  Or is it the deep, white-guy need to fall into the pit himself, before he believes it to be there?  Do we really believe that it was jQuery that created the spaghetti, and not ourselves?  Will abandoning one library for another genuinely improve our own capabilities… or is it a convenient deflection?

I am inclined to shout, “just deal with it!” at people.  They probably want to shout it back to me.

The Best Tool for the Job?

I've written 3 generations of memcached-to-DynamoDb server now.  That is, a server that speaks the memcached text protocol to its clients, but stores data in Amazon DynamoDb instead of memory.  (Why?)  Perl is dying, so generation 2 was written in Python, using the system version that was already in our base images.  But cultural issues plagued it, so I began thinking about generation 3.  What language to write in?

Perl is still dying.  PHP doesn't have a great async story; I could use stream_select, but I was hoping not to build even more of the server infrastructure myself.  Outside of that, we don't have any other interpreters or runtime environments pre-installed in our image, and I didn't want to bloat it more.

Of the compiled languages, then:

1. Rust was known to be incomprehensible.
2. C and C++ are not safe, and don't have package managers.
   
3. Go... doesn't have generics?
4. Nothing else in the category seems to have critical mass (e.g. an AWS SDK ready.)
   

We have another project in Go that I wrote circa Go 1.4; since then, it required a tiny bit of understandable work for the massive benefit of migrating to modules (and that became devproxy2.  I value stability.)  If you don't want to write a generic library, then Go is fine enough.

Go is still burdened by a self-isolating inner circle, making it faux-open-source at best.  But on the other hand, they have built a safe, concurrent, stable, popular, compiled language, with a standard package manager.  It even has an official AWS SDK.

Our brief use of systemd's TemporaryFileSystem feature

Late last year, we started having problems where services did not appear to get proper data after a reload.  But I have a hypothesis.

We have a monolithic server build, where several services end up running under one Apache instance.  I thought it would be nice to make each one think it was the only service running.  Instead of enumerating badness, I switched to using TemporaryFileSystem= to eliminate all directories, then allowing access the one needed for the service.

The problem is, I don’t think this gets reset/updated during a reload. Deployment renames the working directory, and renames an all-new directory in at the original name.  It’s possible that even though the reload happens, the processes are not changing working directory, thus still looking in the original one.

Things that have never failed were failing; in particular, Template Toolkit was still using stale template files.  I did some digging and found that it does have a cache by default, but it only holds entries for up to a second. Furthermore, because I was very strict about not having unexpected global state, we create a new Template object per request, and shouldn't be sharing the cache.

Anyway, systemd did not actually document what it does with TemporaryFileSystem on reload, so I can’t be sure.  I ended up abandoning TemporaryFileSystem= entirely.  I only have so much time to mess around, and there’s only so much “risk” that would be mitigated by improved isolation. It’s almost guaranteed that the individual applications have defects, and don’t need a multi-app exploit chain.

(Not in the sense that I know of any specific defects; in the sense that all software has defects, and some proportion of those can be leveraged into a security breach.)

Making Sense of PHP HTTP packages

There are, at the moment, several PSRs (PHP Standard Recommendations) involving HTTP:

1. PSR-7: HTTP messages.  These are the Request, Response, and Uri interfaces.
2. PSR-17: HTTP message factories.  These are the interfaces for “a thing that creates PSR-7 messages.”
3. PSR-18: HTTP clients.  These are the interfaces for sending out a PSR-7 request to an HTTP server, and getting a PSR-7 response in return.
4. PSR-15: HTTP handlers.  These are interfaces for adding “middleware” layers in a server-side framework. The middleware sits “between” application code and the actual Web server, and can process, forward, and/or modify requests and responses.

I will avoid PSR-15 from here on; the inspiration for this post was dealing with the client side of the stack, not the server.

As for the rest, the the two ending in 7 are related: PSR-17 is about creating PSR-7 objects, without the caller knowing the specific PSR-7 objects. However, it’s a kind of recursive problem: how does the caller know the specific PSR-17 object to use?

There's also some confusion caused by having “Guzzle” potentially refer to multiple packages.  There’s the guzzlehttp/guzzle HTTP client, and there’s a separate guzzlehttp/psr7 HTTP message implementation.  Unrelated packages may use guzzle in their name, but not be immediately clear on which exact package they work with.

• php-http/guzzle7-adapter is related to the client.
• http-interop/http-factory-guzzle provides PSR-17 factories for the Guzzle PSR-7 classes, and has nothing to do with the client.

Additionally, whether these packages are needed has changed somewhat over time.  guzzlehttp/psr7 version 2 has PSR-17 support built in, and guzzlehttp/guzzle version 7 is compatible with PSR-18.  Previous major versions of these packages lacked those features.

Discovery

The problem of finding PSR-compatible objects is solved by an important non-PSR library from the HTTPlug (HTTP plug) project: php-http/discovery (and its documentation.)

• It lets code ask for a concrete class by interface, and Discovery takes care of finding which class is available, constructing it, and returning it.
• It includes its own list of PSR-17 factories and PSR-18 clients, and can return those directly, where applicable. When Guzzle 7 is installed, Discovery (of a recent enough version) can return the actual \GuzzleHttp\Client when asked to find a PSR-18 client.
• It has additional interfaces for defining and finding asynchronous HTTP clients, where code is not required to wait for the response before processing continues.

At its most basic, Discovery can find PSR-17 factories and PSR-18 HTTP clients.  These would be loaded through the Psr17FactoryDiscovery and Psr18ClientDiscovery classes.  For the more advanced features like asynchronous clients, the additional adapter packages are required.

For example, to use Guzzle 7 asynchronously, php-http/guzzle7-adapter is required.  At that point, it can be loaded using HttpAsyncClientDiscovery::find().  This method then returns an adapter class, which implements php-http's asynchronous client interface, and passes the actual work to Guzzle.

In any case, library code itself would only require php-http/discovery in its composer.json file; a project making use of the library would need to choose the concrete implementation to use in its composer.json file.

An Important Caveat

Discovery happens at run time.  Since Discovery supports a lot of ways to find a number of packages, it doesn't depend on them all, and it doesn't even have a hard dependency on, say, the PSR-17 interfaces themselves.  This means that Composer MAY install packages, even though requirements aren't fully met to make all of them usable.

To be sure the whole thing will actually work in practice, it's important to make some simple, safe HTTP request.  In my case, I use the Mailgun client to fetch recent log events.

When code using Discovery fails, the error message may suggest installing “one of the packages” from a list, and providing a link to Packagist.  That link may include the name of a package that is installed.  Why doesn’t it work? It’s probably the version that is the culprit.  If guzzlehttp/psr7 version 1 is installed, but not http-interop/http-factory-guzzle, then the error is raised because there is genuinely no PSR-17 implementation available with the installed versions. However, the guzzlehttp/psr7 package will be shown on Packagist as providing the necessary support, because the latest version does, indeed, support PSR-17.

Things Have Changed a Bit Over Time

As noted above, prior to widespread support for PSR-17 and PSR-18, using the php-http adapters was crucial for having a functioning stack.  So was installing http-interop/http-factory-guzzle to get a PSR-17 adapter for the guzzlehttp/psr7 version 1 code.

For code relying only on PSR-17 and PSR-18, and using the specific Discovery classes for those PSRs, the latest Guzzle components should not need any other packages installed to work.

However, things can be different, if there is another library in use that causes the older version of guzzlehttp/psr7 to be used.  This happens for me: the AWS SDK for PHP specifically depends on guzzlehttp/psr7 version 1, so I need to include http-interop/http-factory-guzzle as well, for Mailgun to coexist with it.

One Last Time

If you’re writing a library, use and depend on php-http/discovery.

If you’re writing an application, you must also depend on specific components for Discovery to find, such as guzzlehttp/guzzle.  Depending on how the libraries you are using fit together, you may also need http-interop/http-factory-guzzle for an older guzzlehttp/psr7 version, or a package like php-http/guzzle7-adapter if a simple PSR-18 client isn’t suitable.

There are alternatives to Guzzle, but since my projects are frequently AWS-based, and their SDK depends on Guzzle directly, that’s the one I end up having experience with.  I want all of the libraries to share dependencies, where possible.