LLMs Are an Inside Threat

There was another “my LLM deleted production” incident (via).  In this particular case, the agent trawled the filesystem to locate another credential with the rights to delete the database and all backups.

This demonstrates an LLM in the role of a “motivated attacker."  When faced with obstacles, it doesn’t simply halt; it inspects the error, adapts its approach, and overcomes the obstacles.  Small comfort that the filesystem outside the workspace is “read only,” if the damage is done beyond the filesystem boundary.

For the foreseeable future, I’ll continue to run the (mandated) LLM tool in a VM.  It’s all alone with the code in a mock production image, and aws-vault, documents, emails, D-Bus, and the GUI are all hidden away on the other side of the hypervisor.

A Few More Words about LLMs for Coding

One of the clear and present dangers to LLM coding is to produce a codebase which can only be operated on by LLMs. Shall we be so eager to have the corporations make us an offer we can’t refuse?

Using LLMs Again

They insisted on selling out the future for dubious short-term interests again #capitalism, so about three million tokens in, I have more thoughts on coding with an LLM.

It’s Chaotic

The model has its strengths and weaknesses, but it can be hard to predict how a specific task will fit.

It went great to upgrade an internal site from Bootstrap 3 to 4 to 5.  It only did okay at dark mode for it.  The machine simply does not know what has contrast and what does not.  I spent a long time asking it for updates on a component-by-component basis.

And sometimes, it just outright makes mistakes.  My first-ever fix for an LLM-generated bug was for some text disappearing from the website, because it had transformed something of the form display(error ? err_msg : text) around to if (error) { display(err_msg); } during the process.  It quit displaying text for the normal/success case.

On a different project, I accidentally clipped its wings by not having the vendor directory installed, and it hallucinated some atrocious code.  The model “not knowing what it didn’t know” greatly hampered its ability to proceed… and it didn’t know that, either.  It didn’t stop and ask for the problem to be fixed.  It just slopped some garbage out.

On a third project, it perfectly generated a GitHub workflow for “build an ECR image on push”, and then flopped on its face with a manual workflow for “deploy such an ECR image into ECS”.  It minimized IAM permissions, blissfully unaware that ecs:DescribeTasks does not use a resource tag.  That one action must be given permissions on resource *, even to describe a specific task that is known in advance.  Faced with the error, it shuffled code around to do the same operation a different way, which also failed.  The human had to track it down in the AWS Console and documentation.

(I asked it to store the IAM policies in the repo for reference.  I do not plug the LLM into AWS, GitHub, MySQL, the bastion host, a web browser, or even git fetch.)

It was pretty good at finding differences between an old system and a new one, but less effective at porting the missing features across.  Most of the time was spent on the human working out the tangled mess of the most difficult pieces.  In the aftermath, closer human review observed 27% defective commits had been made.

An equal number of commits were “not how I would code it,” which is also something that bothers me.  It is my name that goes onto the commit, and will show up in git blame later.  Those also got patched up.

Secure Code is an Afterthought

Even with a strong pattern of CSRF and Allow header mitigations (i.e. a couple of function calls in the setup), it was not able to generate code to handle these concerns.  While it probably knows how to set up a popular framework like Symfony or Laravel to do it, it is not able to learn the pattern in our own ancient code.

It might be no better than other developers on the team at XSS, but that is concerning in both directions.  I don’t want either of them introducing div.innerHTML = htmlStr!  String concatenation is a security vulnerability in systems like this.

When generating some code for the GitHub workflows, it produced a command of the form THING=$(...) and then proceeded to use $THING without checking that it actually got any output in there.  For shell, it’s always best to back out as soon as it starts into the weeds.

Good Prompts Take Knowledge

It saves time and money to point the LLM directly at .svcPop instead of describing “the service redemption bubble” and making the machine thrash around, running half a dozen ripgrep commands to try to find it.

On the other hand, making changes with an LLM can quickly erode one’s low-level understanding of the code, making attempted “good” prompts into not-so-good ones.  When I’m not the one making the changes, I lose understanding and effectiveness.  If I’m using it to write using some new libraries, like Pest or AmPHP/Revolt, I am also losing both depth of learning and retention of what’s left.  I can’t ambiently absorb knowledge from documentation I am not looking at.

(And as we saw, even if the prompt is good, the results may not be.)

Narrow Focus is Double Edged

The narrow focus on the task at hand is what makes the LLM useful at what it is doing, but it is also what builds technical debt.  It’s happy to generate all-new CSS for anything it does, without worrying about whether any of it can be a shared concept across the codebase.

Whether I’m reading or writing the code, I’m thinking about this stuff.  It was me who noticed the multiple ‘loading’ spinner images.  When asked to replace all of them, the LLM generated a gif (?) of a non-theme color, that wasn’t animated (???).  Then, it copied that over all four files (including the two unused ones), corrupting the layout where the smaller file had been used, and declared it done.  Oh, and the gif assumed a white background, on a site that already had a dark mode.  I threw up my hands dramatically, tracked down an SVG, and fixed it all myself.

Meanwhile, its chaotic nature makes it somewhat random which CSS features will be used.  This is especially noticeable for things like choosing between repeating selector prefixes, or nesting the blocks.  It’s 2026 and nested CSS is Baseline 2023, so it’s not like this is going to break anything accessing the site, but it immediately raises questions about how to control the CSS feature usage for sites where we don’t have as much latitude in dictating choice of browser.

But When It’s Good, It’s Good

When I’m using the LLM on a task that it is good for, making quick work of some long-delayed upgrade or feature request, the feeling approaches what others describe as manic.  With all the downsides that entails, too: the selfishness, the hubris, and the possibility that it will turn out to be a complete waste after all.

But its siren song is sparkling and effervescent.

This iteration of models is good enough to see why people like it.

Tech Can’t Solve Social Problems

The code I upgraded to Bootstrap 5 had been on Bootstrap 3 because we aren’t spending any time for maintenance in the constant rush toward “more features."  Nothing can be done until the threat becomes existential.  I’m worried that higher levels of the company will soon see the LLMs as a way to continue this business-as-usual approach.  The developers “have this tool to be more productive,” so we can expect more features, sooner.

I also don’t know if management has visibility into the LLM usage, to understand what they’re actually getting for their money.  It’s entirely possible that such information is only available to someone who may or may not be using the corporate account’s resources for personal projects.  Actually worrying about this is above my pay grade, especially since I have no evidence whatsoever, but still: it’s an obvious potential weakness.

The Chaos Demon

Sometimes, the LLM simply doesn’t follow the prompt.  Or accept correction.  The only thing to do is to Ctrl+C and try anew.

And push down the thought of a world with dangerous equipment going rogue like this.  Self-driving cars.  Industrial equipment.  Weapons nominally in the hands of ICE or the police.

AI Erodes Knowledge

If one has an LLM “do the work,” then one does not actually learn anything.  Taking notes improves recall, and so does trying to remember the answer before looking something up.  In any case, to become more proficient, something has to happen inside the mind. If we skip that part in our rush to produce volume of output, we are trading away our future skills.

“Not gaining proficiency” would be indictment enough, but worse, unused skills decay. There is a reason that C is not on my résumé anymore!  I can’t imagine what would become of my PHP if I didn’t do the coding, and I utterly dread the potential outcome of using an LLM to make changes for an entire month.

An LLM will never produce original work, since it is trying to “predict” based on a corpus of past, public work.  It’s great for writing code like fanboys on the internet write code.  It’s not so great at cost optimization.

Overall, I wonder about how institutional knowledge might be affected by heavier LLM usage.  If we’re not doing the thinking, will we be able to remember anything about our own history?

Early Thoughts on Coding with an LLM

I temporarily displaced my misgivings about the Plagiarism Machine That Also Destroys Earth.  Since I haven’t yet spent a full week working with it, I have some first impressions only:

1. All generators have the highest accuracy on the smallest tasks, where they are the least useful.
2. The tools all seem to prefer the same model, so there’s less differentiation than one might hope for. Nothing is worse… but nothing is better.
3. Writing a good prompt takes a lot of planning.
4. Every change must be tediously reviewed.  This includes every tab-completable inline suggestion in the editor.
5. Junie is not actually integrated very deeply into PHPStorm.
6. Agent mode is full-on Sorcerer’s Apprentice.  One must always be standing by on the “Emergency Stop – Never Use” button.

Due to homogenous model choices, UI is an important differentiator right now.  Using Control+Backslash to generate code in PHPStorm makes for results that are difficult for me to understand, because the diff is in character mode.  Cursor is much better at this, with line diffs.

As for Junie’s lack of integration, it failed to recognize «run the "foo api unit tests"» as an instruction to run the pre-existing test configuration named “foo api unit tests”.  I let it try running foo api unit tests in the terminal to see what it would do.  After getting a command-not-found error, it attempted to find any test that it could run, and try running that instead.  Fortunately, it needed permission to run further off the rails, so I denied it.

Summary

As for the overall experience…

The LLM removes the fun part of programming, the writing of the code, leaving the planning and debugging parts I am less fond of.  The incessant demands for attention from inline suggestions also fundamentally block entering a flow state. Meanwhile, hallucinations are always trying to stab me in the back; there is absolutely no meta-analysis of whether the prompt itself is misguided.

I don’t even see these tools as useful for exploring unfamiliar code.  My IDE already has a set of tools for that; like Find in Files, Go to Definition, and Find References/Usages.  Since these aren’t constrained to a sidebar, the results are also far more usable.

Even if it had no downsides, at first blush, the (paid) systems still rate a solid “meh.”