Using IPv4 for Outbound SMTP in Postfix

I got a rejection notice from Microsoft Office 365.  My email couldn’t be delivered, they said, because it was on the Spamhaus malware-source list.  Sure enough, the /64 was on it… which, I guess, might be the prefix for the entire data center? at my VPS provider.  All I get is a measly /128.

Point being, I wanted to send outbound mail over IPv4 only, while keeping inbound mail available on IPv6.  Thanks to slop, the internet didn’t know how to do it conditionally.  I ended up modifying my /etc/postfix/master.cf to include the configuration on the smtp client only:

smtp    unix    -   -   y   -   -   smtp
    -o inet_protocols=ipv4

The change here is adding the inet_protocols configuration line under the “smtp” UNIX service (first on line), handled by the “smtp” client (last on line).

I worked this out from scattered hints in various documentation, but in a broader search, I managed to find Postfix’s own corroboration of what I did.

I can no longer deliver to IPv6-only MX hosts, but in the short run, that’s far less of a concern than being unable to deliver to dualstack hosts which reject my IPv6 but not my IPv4.  The long-term solution is some sort of smtp_protocol_map that I could specify IPv4-only “for Microsoft” and let the rest of the Internet use IPv6 if it wanted to, but I don’t think Postfix supports that.

The information about my specific server applies to Postfix 3.8 shipped with Ubuntu 24.04 LTS.

Defaults Aren’t Always Good

I am responsible for a b2b system with a few dozen per-client settings, many of which are old and have defaults.  For instance, there is the grace period, which allows a limited self-service time window for clients to enter data that was missed during the normal course of business.  It seemed like a good idea at the time, but now we have a set of invisible values hidden in code holding those defaults.  And we can’t change them.

The main problem is, changing a default is an operational change to an unknown scope of clients.  On top of that, there could be clients that have the current default explicitly configured: would we want to change them, too?  It probably loops in the CEO and takes at least a week of meetings to answer that.

From a b2b business standpoint, it is actually better to require a configuration value (and let the support team view/modify it), even if it introduces the possibility of failure.  Our clients will use the normal support channel (well-oiled and always staffed) to raise the error message to us, if tech doesn’t proactively alert themselves about it.

For an ecommerce or public website where users would just leave, it’s probably more costly to “crash on error” than it is to let an errant process continue.  But if management prefers to halt and catch fire, the defaults are the opposite of that.

The Lock Wait Timeouts: MySQL’s SELECT … FOR UPDATE OF t1

We had an issue with our busiest client, where some updates were failing with a “lock wait timeout."  These were updates of a single row in a single table, and when I ran out of my own ideas, I threw it at an LLM.

The research phase

The main weakness of LLMs is that they’re unliely to say, “I don’t know.  Maybe it’s not here."  Hence, the first thing I did was play along and waste a week on “lock ordering” changes, so that tables would be locked in a consistent order across the main site.  We (someone else with an LLM) identified one query doing a full table scan up front, which would lock everything unintentionally.  But with that fixed, and other ordering changes in place, there was a 0.0% reduction in problems in production.

With my human insight, I looked outside the subsystem, and used the LLM again to analyze the access logs; this found a long-running request in the administrative backend, which was active at the time of the lock-wait errors.  I rediscovered internal timing logs from the admin area, and they confirmed the activity with a narrower time window, but I still didn’t know what was wrong yet.

The final clue came when I had an LLM generate code to instrument the main site, to show lock status when receiving a timeout error.  When the resulting spam was run back through the LLM, it pointed to the contract type table as the source of the failures.  But, the admin page wasn’t planning to write to the type, only the contract itself.  The problem was clearly that it had acquired too strong of a lock.

The actual problem

SELECT … FOR UPDATE in MySQL/InnoDB acquires a write lock on all records scanned. This means means that selecting from t1 and joining two other tables will take write locks on all three tables. For us, blocking reads of contract types would behave the same as locking all contracts of that type, across a much larger number of queries for unrelated contracts.

To limit which tables have write locks acquired, MySQL extended the syntax: SELECT … FOR UPDATE OF t1 will lock only t1 for writing.  Multiple tables can still be locked by comma-separating them.  There’s an additional option which may follow, the NOWAIT (fail the statement if any matched rows are already locked) or SKIP LOCKED (do not lock or return rows which are already locked by another thread.)  Without options, the normal “wait for locks” behavior applies.  Hence, the final statement shape may look something like:

SELECT … FOR UPDATE OF t1, t2 SKIP LOCKED;
SELECT … FOR UPDATE OF t1 NOWAIT;
SELECT … FOR UPDATE OF t1;

We haven’t seen another lock wait timeout since putting this small change into production, which is anywhere from a 0 to 100% decrease in failures.  We will never know exactly, because I couldn’t leave well enough alone.  Next week, we’ll release another change that segments the work in the admin page.  Instead of locking “up to the limit” rows at once, it will lock in blocks of 20, and stream the results.

Fixing My SpamAssassin

Since apparently February, my inbox has been under siege by various spam messages promising free stuff from recognizable brands, except that it’s D1SGUI5ED for old-school spam filter evasion, and the domain names are always alphabet-soup randomness.

Part of the problem: apparently some fool (past me) had set CRON=0 in /etc/default/spamassassin, and also deactivated spamassassin-maintenance.timer, which means the server hadn’t fetched new rules for SpamAssassin in an extremely long time.

Restoring the timer did not help very much… because the other part of the problem is that Bayes auto-learning is on by default.  Amavis feeds emails that result in a pass to SpamAssassin to learn as “ham,” so a spammer who can slip by a few rules can have more luck with their later deliveries.

As a result, spam filter performance had degraded to a <50% block rate, and I was dealing with an overwhelming number of messages.  I reset the Bayes data, moved ~250 emails to my Junk folder, and trained my archives (ham) and Junk (spam.)  Following that, the block rate has been >89%, and the false negatives were sent to Junk for training.

In my particular setup (Postfix smtpd → amavisd-new → Postfix for local delivery), the SpamAssassin processing happens under the amavis system user.  Hence, all the sa-learn commands must be run as that user, and the messages must be accessible to it.

$ cd "$(mktemp -d)"
$ sudo find ~/.maildir/.Junk/cur \
    -maxdepth 1 -type f \
    -exec cp -t . '{}' +
$ sudo chgrp -R amavis .
$ chmod 750 .
$ chmod 440 *
$ sudo -u amavis sa-learn --spam .

Some last, unorganized notes: after a reset, the filter only starts working again when 200+ messages of each type have been learned.  The message IDs are remembered, so mistakes can be corrected by re-sending the same message; this is how training misclassifications can overcome auto-learning.  And finally, I showed training for spam above; training for ham is basically the same process, except changing the source folder and using the --ham flag instead.

This has never been such a problem in the past, because campaigns that succeeded in reaching my inbox kept reusing fixed domain names, which I would configure to accept-but-drop in Postfix ingress.  This kept the messages out of Amavis entirely, and avoided signaling a rejection to spammers.  Unfortunately, the randomized names defeated this old method.

Ubuntu Studio 26.04: We Can’t Have Nice Things

Problem 1: KDE Plasma on Wayland retains the sticky-keys bug, where the latch state (press a modifier twice) is tracked, but doesn’t actually work. Here’s to at least six more months of X11!

Problem 2: the Orchis theme remains deeply, completely broken on KDE.

For science, I created a new user on 26.04 and then used the Wayland session for an as-close-to-default experience as I could manage.  The default experience is the “Ubuntu Studio Dark” theme, with a wallpaper like distressed concrete, a theme choice that died ten years ago.  Now that I had to look at it, we’re both distressed.

I didn’t get to see it very long.  The first thing the system told me was that I needed to reboot to optimize my audio settings, presumably for this new user.  Why Ubuntu Studio can’t get this to work out of the box is unfathomable, but here we are.

The second login, I was able to select the global Orchis-dark theme and apply the desktop layout, which immediately blew everything up.  The panel and wallpaper disappeared, leaving a vast blackness and nothing to interact with.  Except for the Overview hot-corner, anyway.  Fortunately, as someone who has run a FOSS desktop for more than a dozen years, I knew a shortcut to get logged out and back in.

At that point, I had a top bar, with a single visible object: a color picker’s separator and history (black.)  It rather looked like those minimalist corporate logos that are all the rage.  Also, various other parts of the bar had no highlight but would react to clicks, because the problem was actually the lack of icons.

Further research would uncover that Orchis-dark requires in its theme file:

• Telu-circle icon theme
• Vimix-dark cursor theme

I was not able to find these in the Ubuntu repositories.

For further science, or to “keep driving with a blown tire” maybe, I tracked down and installed those themes from the Orchis’ author’s GitHub.  This made the icons show up immediately, and did not fix the cursors.  Why not?  Because the theme installs Vimix and Vimix-white themes!  There is no Vimix-dark.

I began to wonder if I’m the problem, but I checked apt search orchis and there’s no newer/different thing in the repository.  For all the fanfare about the “beautiful new” theme in 24.10 (!), not only did it not work there, but it still does not work in 26.04 LTS.  Embarrassing.

(On top of it all, SDDM still doesn’t remember the last session in a per-user way, so I had to switch it back to X11 to log in as my regular user.)

LLMs Are an Inside Threat

There was another “my LLM deleted production” incident (via).  In this particular case, the agent trawled the filesystem to locate another credential with the rights to delete the database and all backups.

This demonstrates an LLM in the role of a “motivated attacker."  When faced with obstacles, it doesn’t simply halt; it inspects the error, adapts its approach, and overcomes the obstacles.  Small comfort that the filesystem outside the workspace is “read only,” if the damage is done beyond the filesystem boundary.

For the foreseeable future, I’ll continue to run the (mandated) LLM tool in a VM.  It’s all alone with the code in a mock production image, and aws-vault, documents, emails, D-Bus, and the GUI are all hidden away on the other side of the hypervisor.

The Heart of the Process (2012)

Editor's Note: I found this in my drafts, dated 2012-01-25.  I have opted to retain the content unchanged, merely updating links for ubiquitous HTTPS and replacing broken links as needed.  Please enjoy this work of fiction.

The President saw that things took time to implement in code, and involved programmers, testers, and a deployment.  As someone who liked to make snap decisions and have the results available immediately, this did not sit well in his heart.  Long he meditated, then announced his solution:

Henceforth, the system would be Database-Driven.