The Heart of the Process (2012)

Editor's Note: I found this in my drafts, dated 2012-01-25.  I have opted to retain the content unchanged, merely updating links for ubiquitous HTTPS and replacing broken links as needed.  Please enjoy this work of fiction.

The President saw that things took time to implement in code, and involved programmers, testers, and a deployment.  As someone who liked to make snap decisions and have the results available immediately, this did not sit well in his heart.  Long he meditated, then announced his solution:

Henceforth, the system would be Database-Driven.

A Few More Words about LLMs for Coding

One of the clear and present dangers to LLM coding is to produce a codebase which can only be operated on by LLMs. Shall we be so eager to have the corporations make us an offer we can’t refuse?

Using LLMs Again

They insisted on selling out the future for dubious short-term interests again #capitalism, so about three million tokens in, I have more thoughts on coding with an LLM.

It’s Chaotic

The model has its strengths and weaknesses, but it can be hard to predict how a specific task will fit.

It went great to upgrade an internal site from Bootstrap 3 to 4 to 5.  It only did okay at dark mode for it.  The machine simply does not know what has contrast and what does not.  I spent a long time asking it for updates on a component-by-component basis.

And sometimes, it just outright makes mistakes.  My first-ever fix for an LLM-generated bug was for some text disappearing from the website, because it had transformed something of the form display(error ? err_msg : text) around to if (error) { display(err_msg); } during the process.  It quit displaying text for the normal/success case.

On a different project, I accidentally clipped its wings by not having the vendor directory installed, and it hallucinated some atrocious code.  The model “not knowing what it didn’t know” greatly hampered its ability to proceed… and it didn’t know that, either.  It didn’t stop and ask for the problem to be fixed.  It just slopped some garbage out.

On a third project, it perfectly generated a GitHub workflow for “build an ECR image on push”, and then flopped on its face with a manual workflow for “deploy such an ECR image into ECS”.  It minimized IAM permissions, blissfully unaware that ecs:DescribeTasks does not use a resource tag.  That one action must be given permissions on resource *, even to describe a specific task that is known in advance.  Faced with the error, it shuffled code around to do the same operation a different way, which also failed.  The human had to track it down in the AWS Console and documentation.

(I asked it to store the IAM policies in the repo for reference.  I do not plug the LLM into AWS, GitHub, MySQL, the bastion host, a web browser, or even git fetch.)

It was pretty good at finding differences between an old system and a new one, but less effective at porting the missing features across.  Most of the time was spent on the human working out the tangled mess of the most difficult pieces.  In the aftermath, closer human review observed 27% defective commits had been made.

An equal number of commits were “not how I would code it,” which is also something that bothers me.  It is my name that goes onto the commit, and will show up in git blame later.  Those also got patched up.

Secure Code is an Afterthought

Even with a strong pattern of CSRF and Allow header mitigations (i.e. a couple of function calls in the setup), it was not able to generate code to handle these concerns.  While it probably knows how to set up a popular framework like Symfony or Laravel to do it, it is not able to learn the pattern in our own ancient code.

It might be no better than other developers on the team at XSS, but that is concerning in both directions.  I don’t want either of them introducing div.innerHTML = htmlStr!  String concatenation is a security vulnerability in systems like this.

When generating some code for the GitHub workflows, it produced a command of the form THING=$(...) and then proceeded to use $THING without checking that it actually got any output in there.  For shell, it’s always best to back out as soon as it starts into the weeds.

Good Prompts Take Knowledge

It saves time and money to point the LLM directly at .svcPop instead of describing “the service redemption bubble” and making the machine thrash around, running half a dozen ripgrep commands to try to find it.

On the other hand, making changes with an LLM can quickly erode one’s low-level understanding of the code, making attempted “good” prompts into not-so-good ones.  When I’m not the one making the changes, I lose understanding and effectiveness.  If I’m using it to write using some new libraries, like Pest or AmPHP/Revolt, I am also losing both depth of learning and retention of what’s left.  I can’t ambiently absorb knowledge from documentation I am not looking at.

(And as we saw, even if the prompt is good, the results may not be.)

Narrow Focus is Double Edged

The narrow focus on the task at hand is what makes the LLM useful at what it is doing, but it is also what builds technical debt.  It’s happy to generate all-new CSS for anything it does, without worrying about whether any of it can be a shared concept across the codebase.

Whether I’m reading or writing the code, I’m thinking about this stuff.  It was me who noticed the multiple ‘loading’ spinner images.  When asked to replace all of them, the LLM generated a gif (?) of a non-theme color, that wasn’t animated (???).  Then, it copied that over all four files (including the two unused ones), corrupting the layout where the smaller file had been used, and declared it done.  Oh, and the gif assumed a white background, on a site that already had a dark mode.  I threw up my hands dramatically, tracked down an SVG, and fixed it all myself.

Meanwhile, its chaotic nature makes it somewhat random which CSS features will be used.  This is especially noticeable for things like choosing between repeating selector prefixes, or nesting the blocks.  It’s 2026 and nested CSS is Baseline 2023, so it’s not like this is going to break anything accessing the site, but it immediately raises questions about how to control the CSS feature usage for sites where we don’t have as much latitude in dictating choice of browser.

But When It’s Good, It’s Good

When I’m using the LLM on a task that it is good for, making quick work of some long-delayed upgrade or feature request, the feeling approaches what others describe as manic.  With all the downsides that entails, too: the selfishness, the hubris, and the possibility that it will turn out to be a complete waste after all.

But its siren song is sparkling and effervescent.

This iteration of models is good enough to see why people like it.

Tech Can’t Solve Social Problems

The code I upgraded to Bootstrap 5 had been on Bootstrap 3 because we aren’t spending any time for maintenance in the constant rush toward “more features."  Nothing can be done until the threat becomes existential.  I’m worried that higher levels of the company will soon see the LLMs as a way to continue this business-as-usual approach.  The developers “have this tool to be more productive,” so we can expect more features, sooner.

I also don’t know if management has visibility into the LLM usage, to understand what they’re actually getting for their money.  It’s entirely possible that such information is only available to someone who may or may not be using the corporate account’s resources for personal projects.  Actually worrying about this is above my pay grade, especially since I have no evidence whatsoever, but still: it’s an obvious potential weakness.

The Chaos Demon

Sometimes, the LLM simply doesn’t follow the prompt.  Or accept correction.  The only thing to do is to Ctrl+C and try anew.

And push down the thought of a world with dangerous equipment going rogue like this.  Self-driving cars.  Industrial equipment.  Weapons nominally in the hands of ICE or the police.

Are containers light weight?

I read a thing claiming that containers are “light weight.”  But that’s only compared to a hardware virtual machine!  Containers seem light only through the historical accident of their path to popularity.  They are nearly at the end-point of heavyweight distribution methods.

Once upon a time, we programmers were able to handle a bit of version skew. We’d use libraries like GTK+ which maintained backward compatibility—at the ABI level, even—so that code compiled against 2.4.x would run against 2.4.x or later 2.x releases, without changes.  We’d install something like Smarty to the global PHP include path, and use a single copy of it from all our projects, for space efficiency.  Nothing was vendored!

(We could semi-vendor things in scripting languages by playing with the include path.  Install a major upgrade to, say, “lib-v4”, then in the v4-aware application, prepend the “lib-v4” directory to the include path at runtime. When all the applications were converted, remove the old version from the global path, move the v4 code there, and remove the include-path code from the apps again.  It’s a lot like gradually updating a database column.  It wasn’t a great approach for C code, though.)

Portability across operating systems, even in “POSIX” land, was a mess, but we all learned how to do it.  Virtually all open-source code dealt with it, so we had plenty of examples, and we largely respected the user’s choice of platform to run on.  Even if it were Solaris…

This also produced a pressure for minimal dependencies; the less we required of a user, then the more likely they were to run our code.  I still think that Java largely failed on Linux because every user had to fetch the JRE from Sun’s atrocious website themselves.  (Blackdown and later OpenJDK would change this, long after the ship had sailed.  The Apache Foundation’s Java-based projects are a notable exception from the general attitude, but they are also not desktop software.)

Today’s environment is the complete antithesis.  We pack entire OS distributions, possibly a language interpreter, all of our libraries, and our application code into a gigabyte-plus wrapping (partially shared, but still a minimum).  Then, we call it “lightweight” because it doesn’t have a guest kernel in there.

The old times weren’t perfect; it was an incredibly painful experience to make Linux binaries that worked across distributions, because of variance in the filesystem layout and the need to rely on old libraries to cover the older systems people might run the binary on.  And sometimes, there was no choice but to make multiple builds, because distributions might only package one of incompatible library versions. But largely, to support an app of a few megs, we shipped a few megs, not a few hundred, and we certainly didn’t call “a near-complete disk image” lightweight.

The Pace of Change

I’m not the first, nor the only, person to complain about the pace of technical change.  But what are the actual problems?

We risk losing perspective.  We will forget that the fad of today is just another fad; blockchains and containers are destined to be the next XML, relatively soon in their life, then carried forward for thirty years because changing infrastructure is too risky for the business.

We risk losing the wisdom of the past, assuming even our own younger selves were but naked savages, coding in Perl or PHP. We will not know what made Perl, Perl; we will not bring any of the good ideas forward.

Truly, we risk losing experts. It took me a good 10 or 15 years to really appreciate the sheer amount of knowledge that makes an expert, an expert; if we burn our world down every five years, then we will never come to know anything deeply.  We will have no experts.

Where I used to worry about becoming “a dinosaur,” it now seems that dinosaurs going extinct are the larger problem.

But what is the actual problem?

Pride, perhaps?  Are we too snobby to learn about what came before, to understand our place in history, and to meet the present where it’s at?  Do we think there is nothing to learn about a system in reading its code, in making improvements to it, that we must replace it outright?

Is it ignorance?  Or is it the deep, white-guy need to fall into the pit himself, before he believes it to be there?  Do we really believe that it was jQuery that created the spaghetti, and not ourselves?  Will abandoning one library for another genuinely improve our own capabilities… or is it a convenient deflection?

I am inclined to shout, “just deal with it!” at people.  They probably want to shout it back to me.

Stability vs Churn Culture

I’m working on rewriting memcache-dynamo in Go.  Why?  What was wrong with Python?

The problem is that the development community has diverged from my goals.  I’m looking for a language that’s stable, since this isn’t the company’s primary language.  (memcache-dynamo is utility code.) I want to write the code and then forget about it, more or less.  Python has made that impossible.

A 1-year release cycle with “deprecated for 1 cycle before removal” as the policy means it’s possible for users on Ubuntu LTS to end up in a situation where their previous LTS, 2 versions behind, doesn’t provide an alternative for something that’s been removed in the next LTS / current Python.

But looking closer to home, it’s a trend that’s sweeping the industry as a whole, and fracturing communities in its wake.  Perl wants to do the same thing, if “Perl 7” ever lands.

Also, PHP 5.6 has been unsupported for two years, yet there are still code bases out there that support PHP 5, and people are (presumably) still running 5.6 in production.  With Enterprise Linux distributions, we will see this continue for years; RHEL 7 shipped PHP 5.4, with support through June 2024.

There’s a separate group of people that are moving ahead with the yearly updates.  PHPUnit for example; every year, a new major version comes out, dropping support for the now-unsupported PHP versions, and whatever PHPUnit functionality has been arbitrarily renamed in the meantime.  The people writing for 5.x are still using PHPUnit 4 or 5, which don’t support 8.0; it’s not until PHPUnit 8.5.12 that installation on PHP 8 is allowed, and that still doesn’t support PHP 7.0 or 7.1.

This is creating two ecosystems, and it’s putting pressure on the projects that value stability to stop doing that.  People will make a pull request, and write, “but actually, can you drop php 5 support? i had to work extra because of that.”

The instability of Linux libraries, from glibc on up, made building for Linux excessively complex, and AFAICT few people bother.  Go decided to skip glibc by default when building binaries, apparently because that was the easier path?

Now everyone thinks we should repeat that mistake in most of our other languages.

Python has descended into chaos

At my employer, Python is not our primary language; the main system was built in Perl and is now a Perl/PHP hybrid. The surrounding systems have been built/replaced with PHP, but the core code is resistant to change. Management doesn't want to mess with the part where the actual money flows in.

For seamless session handling—local memcached on dev machines, but shared storage in production—we have memcache-dynamo. It started life in Perl. I rewrote it in Python, as part of the quest to replace all Perl with something more popular in modern times. (I figured async/await syntax would be easier for future developers than React-PHP.) It's been working fine on Ubuntu 18.04 for some time, but then I wanted to update to Ubuntu 20.04.

It quickly turned irritating. The Python 3.6 dependencies didn't work on 3.8. It turns out that the updated dependencies for 3.8 don't work on 3.6, either. I guess this explains why pipenv is so insistent that "requires 3.6" means "exactly 3.6".

Python 3 isn't just a version with some backward-incompatible changes; it's fundamentally a new culture with new values. And honestly, that probably means less Python in my future. I don't want to promote another language into the main systems, and Python no longer seems to be suitable for fire-and-forget background services.

(In the end, I wrote a new layer on our installer that checks the Python version, then copies e.g. Pipfile-3.8 and Pipfile.lock-3.8 into their standard location before starting pipenv. It's a horrible mess and I hate it so much. Please, don't start a Python 3 transition every year!)

pipenv's Surprise

Warning: Python 3.6 was not found on your system…
You can specify specific versions of Python with:
  $ pipenv --python path/to/python

I am left with no clear path to making this project run with the system Python 3 across just two versions of Ubuntu LTS.  It doesn't work on Focal (Python 3.8) as-is, and if I update the Pipfile.lock, it won't run on Bionic (Python 3.6).  It doesn't have shebangs for python3.6, as it expects to run on Python 3.6 or up.  This is how SemVer works!

Maybe the answer is to build my own tools in order to run this in a way that suits me. Which is: I really want a build phase to create a tarball, which can be extracted and run in-place later. All the complexity of vendoring should be finished by the time deployment (from tarball) occurs, for reliability and reproducibility.

I do not want to write this project a third time, probably in a language I hate, just because virtualenv is both wholly inadequate, and central to Python tooling.

(Something like the way the awscli-bundle is packaged is an interesting start, but it still has to grind through a lot of code to "install" itself. It's not unzip-and-go. Also, I've got no idea how they build that in the first place.)

Containers are Interop

I mean this in the same sense as “XML was created to solve the interoperability problem.”

The container craze is about the interoperability problem across environments. By vendoring the entire distribution and communicating only over the network, they essentially provide isolation for all the dependencies of a service. Maybe that part is the same, in essence, as the nix package manager.

But then containers have one more trick: they run anywhere with a “Linux syscall interface” underneath. Any environment with Docker support can run Docker containers, anywhere. (As long as the binaries run on the host, at least.) It’s not entirely simple—orchestration is an issue, and Docker is working on that, too—but the containers themselves become highly portable, since they’re shipped as black boxes. They don’t depend on code outside themselves, and as such, that outside code cannot break them so easily.

And maybe, by so fully entwining a Linux distro to our app, we’re forgetting how to be cross-distro or cross-platform. And the old coder in me wants to grump about that. Yet, that’s also a kind of freedom. Not everyone has to learn how to write cross-platform code if the container environment defines exactly one platform to run on.

Maybe we’re losing something, but we’re also gaining ease-of-use and accessibility in the deal.

Configuration as a Point of Failure

Recently, I happened to restore service for our /giphy Slack command.  I originally had things set up poorly, so we needed a ProxyPass configured in every virtual host that wanted to run PHP.  That changed at some point.

I updated all of the configurations in git, of course.

But our giphy endpoint is a separate project, meant to be shared with the world, which therefore doesn’t follow our standard layout.  Its config is not in its git dir; I missed it in the update pass.

I had also just dealt with all the config file merging and changes in the Ubuntu 18.04.1 LTS update.

It really got me to thinking: anything configurable is a point of failure. Whether it’s a process/knowledge failure like with our giphy endpoint, or merge conflicts in *.lock or webpack.config.js files, or a system package changing its configuration semantics or defaults between versions:

The presence of configuration allows broken configurations.

We should try to avoid unnecessary configurability in a system.  This is also very difficult to achieve—a system that only has defaults absolutely needs to have correct, working defaults.  The problem only gets harder in open source, where a project often serves many diverse users.

Also, one of my greatest weaknesses is throwing in easy-to-build options “just in case.”

Finally, it’s bad practice to bake credentials into a git repository, but how shall they be provided without configuration?  EC2 has IAM Roles, but they don’t work in VirtualBox… it really does seem like some configuration is a necessary evil.

Freezing Layers (static vs dynamic)

Let me warn you right up front: this is a philosophical and rambling post.  I'm trying to pull together a series of observations into a coherent structure and it's quite possible that I failed at that.  In addition, because of its lack of quality, “PHP” throughout refers to the 5.x and earlier lines.  Version 7.0 basically addresses the error complaint.  Now without further ado, here is your ancient, rambling post...

I've written some Perl code using FCGI managed by mod_fcgid, in which I constructed my own request loop and application pre-loader.  (And complained about the way mod_fcgid is set up, since it doesn't pre-fork fastcgi children—whenever concurrency is exceeded, starting from zero after a fresh startup, someone has to wait for the world to load just like the bad old CGI days.)

I've written plenty of PHP as well, where you get to generate a response in response to routing once the server environment gets around to delivering the request to you.  The only things that may persist from request to request are allocated on the C level, not by the PHP script directly: most famously, opcode caches and persistent database connections.

Smart Clients and Dumb Pipes for Web Apps

After something like 8 years of coding for the web, both in new and legacy projects, across four companies and some personal projects, I've come to a philosophy on web application layout:

If you require that the client has JavaScript, then you may as well embrace the intelligence of your client-side code.  Let your AJAX calls return simple messages, to be converted to a visible result for the user by code that shipped alongside that UI.

Labels

I figured out my underlying problem with Yegge's liberal/conservative (libertarian/authoritarian) division of programming cultures.

People like looking down on those considered inferior.  "Conservative" adds another way to do just that.

Plain Old Data

I’m coming to the conclusion that there’s actually no such thing as “plain data;” it always has some metadata attached.  If it doesn’t, it might be displayed incorrectly, and then a human needs to interfere to determine the correct metadata to apply to fix the problem.  (Example: View → Character Encoding in Firefox.)  Pushed to the extreme, even “just numbers” have metadata: they can be encoded as text, a binary integer/float (IEEE 754 or otherwise) of some size/endianness, or an ASN.1 encoding.

Another conclusion I’m reaching is that HTTP conflates all kinds of metadata.  Coupled with the lack of self-contained metadata in file formats and filesystems, things start to accumulate hacks.

Layer 7 Routing: HTTP Ate the Internet

In the beginning was TCP/IP, and the predominant model was that servers would listen for clients using a pre-established port number.  Then came Sun RPC, in which RPC servers were established dynamically, and listened on semi-random ports (still, one port per service provided); the problem was solved by baking the port mapper into the protocol.  The mapper listens on a pre-established port, and the client first connects there to inquire, "On what port shall I find service X?"

Then came HTTP, the layer 6 protocol masquerading as layer 7.

The Chariot

I read "Linking in JSON" the other day.  I knew someone had already gotten started on JSON Schema.  (A quick search shows JSON namespace ideas floating around.)  JSON as the lightweight alternative to XML being turned into XML?  This is beginning to sound familiar.

With lightweight formats, we tend to get a proliferation of variants for different uses.  (Not just images, naturally, and CPAN manages to use more than one.)  Heavier formats tend to have a problem I'm going to call "Accessories Not Included": they get sufficiently large and complex that not all readers support all format options.  If the growth is arrested early enough, you end up with a handful of profiles; if it gets out of hand, you have over ten of them.

I never expected XML to be so widely used for so much stuff, or spawn so many related specs.  After all, it was verbose!  And you could make up just any tag name you wanted!  But it turns out to scale well from a not-quite-simple tree data structure, with annotated nodes, all the way up to unfashionable Enterprisey uses.  But scalability bothers people, because who knows what wacky thing someone else on the team is going to foist on you, so more restricted alternatives rise to popularity.  If this is true, the rise of Java should correlate to (non-game) companies getting burned enough times on C++.  (If you think about Safety, it makes sense.  He wants you to use Java, because he can't hack reversing the polarity of the template stack flow.)

This sort of thing is practically destined to keep happening.  More features generally cost more memory and processing time, or some other inconvenience like a compilation step, which is against the religion of some developers.  Thus, lightweight versions of things spring up in opposition to whatever is perceived to be too heavy.  Sometimes compilation is considered the lightweight alternative, since it's not done on every request.

Though sometimes many similar projects proliferate because they just aren't that hard.  It's easier to write a web framework than to learn one, so there are a lot of them.


Made it out of the link forest and need something more to kill the time?  Maybe you want to subscribe to my feed, or follow me on twitter.