Using IPv4 for Outbound SMTP in Postfix

I got a rejection notice from Microsoft Office 365.  My email couldn’t be delivered, they said, because it was on the Spamhaus malware-source list.  Sure enough, the /64 was on it… which, I guess, might be the prefix for the entire data center? at my VPS provider.  All I get is a measly /128.

Point being, I wanted to send outbound mail over IPv4 only, while keeping inbound mail available on IPv6.  Thanks to slop, the internet didn’t know how to do it conditionally.  I ended up modifying my /etc/postfix/master.cf to include the configuration on the smtp client only:

smtp    unix    -   -   y   -   -   smtp
    -o inet_protocols=ipv4

The change here is adding the inet_protocols configuration line under the “smtp” UNIX service (first on line), handled by the “smtp” client (last on line).

I worked this out from scattered hints in various documentation, but in a broader search, I managed to find Postfix’s own corroboration of what I did.

I can no longer deliver to IPv6-only MX hosts, but in the short run, that’s far less of a concern than being unable to deliver to dualstack hosts which reject my IPv6 but not my IPv4.  The long-term solution is some sort of smtp_protocol_map that I could specify IPv4-only “for Microsoft” and let the rest of the Internet use IPv6 if it wanted to, but I don’t think Postfix supports that.

The information about my specific server applies to Postfix 3.8 shipped with Ubuntu 24.04 LTS.

Fixing My SpamAssassin

Since apparently February, my inbox has been under siege by various spam messages promising free stuff from recognizable brands, except that it’s D1SGUI5ED for old-school spam filter evasion, and the domain names are always alphabet-soup randomness.

Part of the problem: apparently some fool (past me) had set CRON=0 in /etc/default/spamassassin, and also deactivated spamassassin-maintenance.timer, which means the server hadn’t fetched new rules for SpamAssassin in an extremely long time.

Restoring the timer did not help very much… because the other part of the problem is that Bayes auto-learning is on by default.  Amavis feeds emails that result in a pass to SpamAssassin to learn as “ham,” so a spammer who can slip by a few rules can have more luck with their later deliveries.

As a result, spam filter performance had degraded to a <50% block rate, and I was dealing with an overwhelming number of messages.  I reset the Bayes data, moved ~250 emails to my Junk folder, and trained my archives (ham) and Junk (spam.)  Following that, the block rate has been >89%, and the false negatives were sent to Junk for training.

In my particular setup (Postfix smtpd → amavisd-new → Postfix for local delivery), the SpamAssassin processing happens under the amavis system user.  Hence, all the sa-learn commands must be run as that user, and the messages must be accessible to it.

$ cd "$(mktemp -d)"
$ sudo find ~/.maildir/.Junk/cur \
    -maxdepth 1 -type f \
    -exec cp -t . '{}' +
$ sudo chgrp -R amavis .
$ chmod 750 .
$ chmod 440 *
$ sudo -u amavis sa-learn --spam .

Some last, unorganized notes: after a reset, the filter only starts working again when 200+ messages of each type have been learned.  The message IDs are remembered, so mistakes can be corrected by re-sending the same message; this is how training misclassifications can overcome auto-learning.  And finally, I showed training for spam above; training for ham is basically the same process, except changing the source folder and using the --ham flag instead.

This has never been such a problem in the past, because campaigns that succeeded in reaching my inbox kept reusing fixed domain names, which I would configure to accept-but-drop in Postfix ingress.  This kept the messages out of Amavis entirely, and avoided signaling a rejection to spammers.  Unfortunately, the randomized names defeated this old method.

Kernel Config Must Get Easier

Not to be “old person on main,” but 25 years ago, it was possible to build my own Linux kernel and run it on my hardware.  As usual in development, it would take a few times to get one that would boot nicely, but the process itself didn’t feel difficult.  It was pretty easy to select IDE, VIA chipset support, and the like, and receive a fairly streamlined kernel.

Fast forward to the 2020s, and the opposite has happened.  I’m reliant on distro kernels because trying to configure and build a modern Linux kernel is a huge, interconnected maze of options, with no visibility into why it doesn’t boot.  Options in section 3 may not be visible if required options in section 5 haven’t been selected yet, which make it a nonlinear meta-adventure.  I have repeatedly failed to build a kernel for a VirtualBox guest, something where the hardware should be well known in advance.

However, if I had been able to build my own kernels successfully, Copy Fail 2 and Dirty Frag wouldn’t have been issues.  IPSec is pretty much dead to me, and I had never heard of RxRPC until now, so these things would have naturally been configured out of my own kernel build.

There’s a distinct conflict here between convenience and security: if we autoload the kitchen sink, then nobody needs to build enable-feature ipsec and a GUI for it.  Nobody needs to recompile their kernel for it.  On the other hand, if there’s a bug anywhere in the autoload surface, it can be reachable by any user.

And honestly, the whole reason I fell out of building kernels—other than the journeys into FreeBSD and Windows—is that it was terrible to try to keep up with the influx of kernel updates.  Maybe userspace had a compatibility guarantee, but config did not.  I was soon unwilling to spend so much energy (mine and my PC’s) on rebuilding a kernel so frequently.  Most of the commits would be irrelevant, but trying to filter and judge them all was an even worse proposition.

I know my distro doesn’t want to maintain dozens of packages like linux-modules-ipsec for people (a logical equivalent of enable-feature ipsec) but it’s also difficult to rebuild a distro kernel according to my own config.  Leaving aside the problems with choosing/generating a config, the distro kernel has a lot of features, which makes it take a while to compile.  The cycle to even try a new kernel build is much, much longer than it was when I was building my own, and I think it crosses the threshold between “tolerable” and “too much to bear.”

A Few Notes on Nginx in Debian/Ubuntu

The PPA I had been getting nginx (stable) from decided to delete the whole PPA, with no announcement on Patreon first.  I only found out when unattended-upgrades started sending email that the Release file for the PPA was missing.  I have canceled my support.  However, I learned some things in the process.

nginx-light Is Obsolete

First, the nginx-light package in Debian 13 (Trixie) and Ubuntu 24.04 LTS is a transitional package.  The modern approach is to install nginx and whatever libnginx-mod-* packages are desired.  If you’re making this change on such a system, nginx is already installed, and should be marked manual before removing nginx-light:

$ sudo apt-mark manual nginx
$ sudo apt remove nginx-light

I tried using the nginx apt repository (as published in extrepo because nobody should have to run gpg ever again) but it is packaged against Debian 13, and thus doesn’t work with Ubuntu 24.04’s older OpenSSL.

Getting HTTP/2 Back

I switched back to the version included in the Ubuntu 24.04 repositories, but this downgraded from 1.28 to 1.24, which is before the introduction of a separate http2 on; configuration directive in 1.25.1.  I initially turned it off and proceeded, which meant my server didn’t provide HTTP/2 for a bit.

The solution is, older nginx uses an http2 in the listen directive:

listen 443 ssl http2 default_server;

This “http2” should appear in all “listen 443” directives; certbot renew will leave it alone if it happens to be present.

Getting Brotli Compression Back

The other issue I had was that the servers were configured to support brotli compression.  I got into a state where the Ubuntu nginx couldn’t finish installing itself, because it didn’t recognize the brotli configuration.  Meanwhile, its failure stopped the process of setting up other packages, including the one that would let nginx support brotli, libnginx-mod-http-brotli-filter.

Breaking that logjam required commenting out the brotli configurations, then finishing the package setup, before re-enabling brotli.

$ sudo dpkg --configure -a

This restarts the setup process, and the updated configuration lets it complete.

Looking Forward

I left myself good comments in the nginx configuration, because in a few months, Ubuntu 26.04.1 will be released, and my VPS will be eligible for upgrade.  At that point, I’ll want to know all this again.

Fixing Weblish/SSH Lish showing nothing

I had a fire drill: my SSH host key certificate expired, predictably enough, and I wanted to see if I could get in without simply answering yes at the unknown-host prompt.  The answer was no, but now it’s yes.  What changed?

# systemctl enable --now getty@ttyS0.service

Weblish, and the Lish SSH gateways, use the system’s serial console to provide their service.  If nothing is ‘listening’ on the console, then having access to the console is meaningless.  All I had to do was actually turn on the getty process for that serial console.  Everything worked for me out of the box, without needing me to specify baud/parity/stop bits anywhere.

The ‘problem’ with using Glish instead was that it doesn’t paste; it just prints ^[. on the console when trying to paste.  There is no way that I’m hand-copying 500 bytes of Base64 text into the system, except in a true and dire emergency.  Hence, I accepted the host key, updated the system, and deleted the host keys later.

Reminder: test the serial console/recovery path, before it is needed.

An additional cautionary tale: our EC2 instances at work have serial consoles running, but we don’t have user passwords configured, so they still cannot be logged into.  Fortunately, that problem was curable via reboot, and I didn’t have to restore an EBS backup.

Two Thoughts on Ubuntu Signing Keys

Here’s something I don’t get: why is there a trusted “2012 CD signing key” on my Ubuntu 24.04 machines, when there is also a “2018” signing key?  Shouldn’t this be a transition that could have completed within five years?  Shouldn’t we be able to tie the 2012 key to a specific repository set, instead of all packages?  The latter includes PPAs and I really wish neither of those CD signing keys were valid for that purpose.

The cryptographic domains should be separated:

1. One CD signing key, tied to the CD/DVD packages
2. One online release signing key, tied to the Ubuntu main/security sources
3. One key per PPA, tied to that PPA

Deprecating globally-trusted keys for PPAs is a good step, but the globally-trusted release keys (especially ones that are over a decade old) should be cleaned out immediately as well.

Semi-related pro tip: extrepo

Many packages are supported in extrepo, which handles the keys for you.  There is no need for arcane gpg format-conversion commands, no worrying about whether it goes into /usr (incorrect under Unix philosophy, but widely recommended) or /etc, no manually editing sources files, and especially no cursed curl | bash invocation.

$ sudo apt install extrepo

And then you can do stuff like:

$ extrepo search github
$ sudo extrepo enable github-cli
$ sudo apt install --update gh

This is especially useful for upstreams that distribute an official deb package, outside of PPAs.  I aim to get the code from as close to the source as possible, where the distro itself doesn’t suit my needs.

Vorta’s “No Matter What, Keep All…” Setting

Since switching from Pop!_OS (Gnome) to Kubuntu (KDE) for work, I have also changed my backup GUI.  There is no question that Vorta is more powerful than Pika Backup, but the price of that is the loss of simplicity.

One place I got confused was in the backup-retention rules, referred to as “pruning” by the GUI.  I have learned: when Vorta offers to “keep all backups made within…”, that is internally a separate rule with high priority. Therefore, when I set up my hourly backups to keep “one week” of hourly, two weeks of daily, and so on, but “keep everything from the last six weeks,” I ended up with seven weeks of hourly backups, followed by the two weeks of daily.

I noticed the problem when my laptop fans spun up for a while, which turned out to be Vorta verifying 300+ archives.  The work laptop is on only for work, producing backups for 8–9 hours per weekday, for around 250 extra archives over those first six weeks.

Unrelated, but one nice thing about Vorta is that, like Pika, it is a front-end to Borg.  I gave it the same repository on disk.  Now I have continuous backup history across the two GUIs, and emergency CLI access if necessary.

Online Builds

As a long-time coder and tinkerer, who views computers as deterministic if we understand them properly, modern tooling feels wrong to me.

• Python expects code to be distributed in a form where it has to contact PyPI for dependencies. (You can get around this—like the awscli installer—but I never did figure out how they build that.)
• Python expects code to be distributed in a form where the installation process executes arbitrary code. This transitively happens with all dependencies. 🐝
• composer install (usually) expects to be able to fetch code from GitHub.  Running it (non-interactively and with --no-dev, of course) as part of deployment makes deployment depend on the internet working.
• Containerfile ADD and COPY will happily take URLs as sources, including URLs that are intended to be mutable, like GitHub /latest/ release artifact URLs. Projects may recommend using such URLs.
• curl … | sudo sh also deeply connects the internet to the process, and treats the script itself as ephemera, discarding it as the process completes. If the script makes its own internet connections, the problem with preserving the canonical source is multiplied.

Quite aside from “the internet connection has to be up,” the referenced URLs must keep working over time.  A Containerfile built as recommended for the docker-php-extension-installer inherently requires the up-to-date source of code to remain at the github.com site, and under the mlocati user.

Building reliability and reproducibility into the process is left up to the user.  Those features can only be included if the thought, “what if…?” crosses someone’s mind.

However, saving remote resources into a local build context protects them from loss, but requires the maintainer of that build to update those resources.  Probably manually.  If it can’t be changed out from under me the next time I run podman build, then it also isn’t getting updates to follow changes in the base image.  It takes some discipline to track where these things come from, and sometimes, how to reproduce them.  For instance, when GPG keys for an Ubuntu PPA needed to be converted to binary before use, it wasn’t enough to leave only the URL written down.

Thus, it’s more work, but the result is stable, and that’s important to me.

The fiserv Outage

Editor’s Note: this post was penned offline early Friday evening, before the author had knowledge of the issue being resolved, and fiserv processing the backlog as of 16:45.  We have chosen to simply add some links, now that we are online to retrieve them.  The post follows.

As I write, on Friday, 2025-05-02, fiserv has been offline all day, or substantially all day.  This company acts as a third party to a number of banks, providing wire transfers, ACH, and/or direct deposit services, and possibly even online/mobile banking.  A number of large banks, including Ally Bank, Bank of America, Capital One, and Synchrony have been affected in some way by this outage, as was my regional bank.

I don’t know anything about the root causes yet.  It would be irresponsible to speculate about those causes, so of course I am going to.

Every Change Might Be Breaking

We originally had the “automatic minor version upgrade” option active at Amazon RDS.  This option simply does not work very well.  Sometimes, for no clear reason (and without notification), it would stop applying upgrades, and require manual updates to get moving again.  We mostly lived with it, and then we hit the worst case scenario: it did perform the upgrade, and then one of our scripts stopped working.

Not only that, it managed to break while I was on vacation.

(Obligatory xkcd about spacebar heating.)

Since then, we don’t use that option.  When I’m good and ready, I peruse the changelogs, then schedule the update to happen when I will be in the office to handle unexpected issues.

For their part, AWS recommends testing the app against the new version of the database before performing any upgrades.  This is implicitly a recommendation against using automatic minor upgrades, because there is no automated process to test the upgrade first.

One knows an analysis tool is looking at AWS with a security-first paradigm when it recommends switching the automatic upgrade option back on for the database.  It is technically correct that new releases MAY contain security fixes, but upgrading to them MAY cause an automated denial of service.  It is not a simple, inconsequential task.

Everything Needs Rate Limits

For reasons of “anything else would cost more,” my web server and email MTA are running on the same VPS. Which apparently means, if a known issue in some web-side software fills the disk, email quits flowing in.

Fortunately, I was at liberty to investigate immediately. The root cause was a bot making things up and filling a cache with negative responses. The actual bot was promptly banned. Not only via robots.txt (which used to return text "# 200 OK Have Fun" so that I wouldn’t get 404 errors for it) but via User-Agent in the Web server, since it had obviously already seen the permissive robots.txt contents.

(Also, the only email that failed to be received were spam messages.  “Lucky” me!)

It worried me, though.  What if another bot did this? Am I going to play Whac-A-Mole® with it? (Don’t get me wrong—that’s not a bad game, but this version doesn’t give me any tickets redeemable for prizes at the front counter.)

To give me more runway to respond to future problems, I added per-source rate limits. This should prevent the cache from filling in units of MB/s. Concurrently, there is also a custom disk-usage alarm that will activate if free space falls below a “normal” amount, giving me a chance to catch problems before the MTA starts refusing service again.

A global rate limit for new connections is still being considered. The current bot menace has been beaten for today, and future bots from a single IP will also find themselves running into limits, but someone running a crawler network could still cause plenty of trouble. The problem is that a global rate limit is probably something that would be HTTP-oblivious, responding with an RST packet instead of an application-layer 429 message.

I know the “everything needs rate limits” is common wisdom, at least in some circles, but I ran public sites for at least 14 years without them. Sadly for the nostalgia, it seems those days are gone.

The Ruthless Elimination of Differences

I am excited for image-based Linux.  Yes, I usually complain about people upending things just when they get stable, but I think there’s a real benefit here: the elimination of differences.

Why, exactly, does installing Ubuntu have to unpack a bunch of .deb files inside a system? Thousands or millions of machines will go consume CPU to run maintainer scripts, to hopefully produce identical output, when most of the desired result should have been possible to save as an image in the first place.  Upstream should know what’s in ubuntu-minimal!  Looking through a different lens, Gentoo distributes a stage2 image.

In theory, an installation CD could carry the minimal image, the installer overlay, and the flavor’s overlay.  The installer’s boot loader would bring up the kernel, use the minimal+installer pair as root file system, and the installer would unpack the minimal+flavor images into the new disk partition.

“Image-based Linux” more or less takes this one more step, running the entire system directly from the images (or a singular combined image.)  Everyone gets to use the same pre-made images, and bugs become less dependent on the history of package operations.

If any of this sounds like Puppy Linux, that’s not entirely accidental.

This is also the space where things like ABRoot are being introduced.  Image-based Linux lends itself well to having an integrated rollback/recovery pathway. Even on my non-image systems, having “a recovery partition” has been more valuable than I ever anticipated.  It let me test backups without having to work very hard about simulating a disaster. I also created my own recovery partition when I was still using a RealTek USB WiFi device, to avoid being stranded without internet.  (Word to the wise: use Mediatek instead, or an Intel PCIe card is a good non-USB option.)

Image-based Linux and the tools around it are poised to make real improvements to the repeatability and reliability of the systems.  I don’t know when I, personally, might benefit (as my daily driver is macOS now), but I am very excited about the progress being made here.

Systemd Allows Unknown Units in Before/After

Most of the time, my development virtual machine guest would boot and run perfectly fine.  Sometimes, though, the FastCGI service backing one of the websites would not be up and running.  It had a ConditionPathExists, and if the code to run the service wasn’t mounted, it wouldn’t start.

The intention was to allow colleagues to import a copy of this guest, then set up the mount to share the project from the host as they saw fit.  On their first boot, with no sharing, ConditionPathExists would prevent the FastCGI service from attempting to start, and therefore, systemd would not report that the system was degraded.  Another point about this system is that the sharing mechanism is unspecified: colleagues are free to use NFS (as I do), Plan9 file sharing, or the hypervisor’s shared-files mechanism.  The host paths are also unspecified, so there is no way I can set up the guest to expect specific sharing in advance.

In practice, sometimes NFS wasn’t ready in my guest before systemd was checking conditions for the FastCGI service.  The obvious answer was to add After=remote-fs.target to the FastCGI service.  I quickly added a drop-in to add this directive to my own post-configuration scripts.

However, that’s a local solution to a global problem.  My colleagues can’t benefit from that, and I should minimize the burden on them periodically setting up new guest images.  The fewer things they must remember, the better.

It turns out the answer was even simpler: I could skip the drop-in and add the After= line to the main service file. I added both remote-fs.target and the hypervisor’s guest services to the line, which means:

1. In production, there are no remote filesystems to mount, nor guest services; there is no latency introduced.
2. When using NFS or similar, systemd waits for the remote filesystem before starting the FastCGI service.
3. With the hypervisor’s file sharing, the guest services mount the shared files before starting the FastCGI service.

My guest doesn’t actually have the guest services installed, but the FastCGI service starts up as intended.  Looking at systemctl list-units --all output, the guest services are (now) listed as not-found and inactive, which is pretty much what I would expect from a dangling reference.  systemd knows about it because I listed it in After, but since it’s not required by anything, the missing definition for it doesn’t cause any problems.

Residual Config Without Config Files

apt makes a distinction between “removed” and “purged.” In both, the packages are uninstalled; in the former state, config files remain, and in the latter, those are also removed.  Actually, that’s not quite the whole story.

A package can have no configuration files, yet still be in ”residual config” state when removed.  This happens if a package defines a postrm maintainer script. These can have basically any shell commands in them, so their actions aren’t visible in any list-of-files.

The specific package I was looking into was a library, with a postrm script that ran ldconfig… during removal.  The package was being shown in residual-config state because it had a script.  Although that script would do nothing during purge, apt (and dpkg) can’t know that.

How to list residual-config packages: apt list 2>/dev/null | grep residual-config or dpkg -l | grep ^rc.

Listing configuration files: try one of these answers as this gets real complex, real fast.

Reading a postrm script: look at /var/lib/dpkg/info/{PACKAGE}[:{ARCH}].postrm (the ARCH component may not be present.)

Scattered Notes on Dovecot’s userdb, passdb, and passwd-file

Dovecot can authenticate users using a passwd-like file.  This happens in two phases.  First, users are looked up in the passdb.  If the user is found and authenticated, then the user is looked up again in the userdb to get things like their UID/GID and home directory.

Now, this doesn’t allow for aliasing users in Dovecot.  If the login is user@example.com, then the defaults will lead to trying to find “user@example.com” in the passdb, then the userdb.  Failure to have these configured correctly can result in different errors:

1. User not found in the passdb: authentication fails.  (Beware of fail2ban here.)
2. User not found in the userdb: user can authenticate, but appears to have no mail!

For my own system, the virtual address needs to be resolved to a particular system user (aka Unix account.)  I also want to share the password files with Postfix for outbound email authentication.  This made Dovecot complicated: I want to log in as user@domain, then have that processed as user for both lookups in a file that is specific to the domain. I put the shortened user in the passwd-file, and now I have to configure passdb carefully:

# /etc/dovecot/local.conf snippet
passdb {
    args username_format=%n /local/auth/%d/passwd
    override_fields user=%n
    driver = passwd-file
}
userdb {
    args /local/auth/%d/passwd
    driver = passwd-file
}

This makes passdb do the first lookup using the short username, %n, with the args setting.  Then, that short username is returned by override_fields for use in later lookups.  After that, userdb can continue with no special settings; it will use the overridden user to look up the short name, and nothing special needs to happen.

I believe that the passwd-file can’t return a different username, because there’s only one username field (the first field), and it is also the lookup key.  This is what requires us to use override_fields for this scenario.

Don’t Let HTTP/2 Nest

For some time, I had problems accessing a dev server with HTTP/2.  Asking cURL to use HTTP/1.1 worked fine, so that’s what I did for a long time.

Today, I found the root cause.  I had nginx set up as reverse-proxy/TLS termination (to emulate ALB), proxying requests to apache2.  Both of them had HTTP/2 enabled!  I needed to deactivate support in Apache, and since the system is Debian/Ubuntu based, that meant:

sudo a2dismod http2
sudo systemctl reload apache2

After that, everything worked.

The problem was that the client would connect to nginx with HTTP/2, and then the request would be sent to Apache. Apache's HTTP/2 module would include an Upgrade: h2, h2c header in the response.  Then nginx would dutifully copy this back to the client.  When cURL or PHP streams received this header, they would detect it as invalid: we can’t upgrade to HTTP/2 from inside HTTP/2.

That error-handling resulted in discarding the response body… but not the HTTP 200 status code, which was extremely puzzling.  How could this successful request have failed?  It failed during header processing, after processing the status and before accepting the body.  (I think browsers must ignore it?  Or maybe they don’t use HTTP/2 through a proxy, even with CONNECT requests?  I would have had to figure out the problem much sooner, if they had seen this Upgrade header and treated it as an error.)

The other weird thing about this is that Apache doesn't have TLS configured, but it still provided h2 as an option in its Upgrade header.  I don’t think that’s a reasonable configuration.  It’s especially not a reasonable default, but I’m not sure whether that’s Apache’s problem, Debian’s, or Ubuntu’s.

What I Learned Trying to Install Kubuntu (alongside Pop!_OS)

First and foremost, once again, this is clearly not a supported configuration that I tried to make.  I'm sure that if I wiped the drive and started afresh, things would have gone much better.  I just… wanted to push the envelope a bit.

Pop!_OS installs (with encryption) with the physical partition as a LUKS container, holding an LVM volume group, and the root filesystem is on a logical volume within.  The plan was hatched:

• Create a logical volume for /home and move those files over to it
• Create a logical volume for Kubuntu’s root filesystem
• Install Kubuntu into the new volume, and share /home for easy switching (either direction)

Things immediately got weird.  The Kubuntu installer (calamares) knows how to install into a logical volume, but it doesn’t know how to open the LUKS container.  I quit the installer, unlocked the thing, and restarted the installer.  This let the installation proceed, up to the point where it failed to install grub.

Although that problem can be fixed, the whole installation ended up being irretrievably broken, all because booting Linux is clearly not important enough to get standardized. Oh well!

Unplugging the Network

I ended up finding a use case for removing the network from something.  It goes like this:

I have a virtual machine (guest) set up with nodejs and npm installed, along with @redocly/cli for generating some documentation from an OpenAPI specification.  This machine has two NICs, one in the default NAT configuration, and one attached to a host-only network with a static IP.  The files I want to build are shared via NFS on the host-only network, and I connect over the host-only network to issue the build command.

Meaning, there is no loss of functionality to remove the default NIC (the one configured for NAT), but it does cut npm off from the internet.  That’s an immediate UX improvement: npm can no longer complain that it is out of date! Furthermore, if the software I installed happened to be compromised and running a Bitcoin miner, it has been cut off from its c2 server, and can’t make anyone money.

An interesting side benefit is that it also cuts off everyone’s telemetry, impassively.

I can’t update the OS packages, but I’m not sure that is an actual problem.  If the code installed doesn’t have an exploit payload already, there’s no way to get one later.  The vulnerability remains, but nothing is there to go after it.

Level Up

(Updated 2024-12-19: this section was a P.S. hypothetical on the original post. Later sections are added.)

It is actually possible to deactivate both NICs.  The network was used for only two things: logging in to run commands, and to (re)use the NFS share to get the files.

Getting the files is easy: they can be shared using the hypervisor’s shared-folders system.  Logging in to run commands can be done on the hypervisor’s graphical console.  As a bonus, if the machine has a snapshot when starting, it can be shut down by closing the hypervisor’s window and reverting to snapshot.

Now, we really have a network-less (and stateless) appliance.

Reconfigure

Before I made that first snapshot, I configured the console to boot with the Dvorak layout, because the default of Qwerty is pretty much why I use SSH when available for virtual machines.  But then, after a while, I got tired of being told that the list of packages was more than a week old, so I set out to de-configure some other things.

I cleared out things that would just waste energy on a system that would revert to snapshot: services like rsyslog, cron, and logrotate.  Then I trawled through systemctl list-units --all and cleared a number of timers, such as ones associated with “ua”, apt, dpkg, man-db, and update-notifier.  Any work these tasks do will simply be thrown away every time.

I took the pam_motd modules out of /etc/pam.d/login, too.  If Canonical doesn't want me to clear out the dynamic motd entirely, the next best thing is to completely ignore it.

After a reboot, I went through systemd-analyze critical-chain and its friend, systemd-analyze blame, and turned off more things, like ufw and apport.

With all that out of the way, I rebooted and checked how much memory my actual task consumed; it was apparently a hundred megabytes, so I pared the machine’s memory allocation down from 2,048 MiB to 512 MiB.  The guest runs with neither swap nor earlyoom, so I didn’t want to push it much farther, but 384 MiB is theoretically possible.

NFS

A small, tiny note: besides cutting off the Internet as a whole, sharing files from the hypervisor instead of NFS adds another small bit of security.  The NFS export is a few directories up, and the host has no_subtree_check to improve performance on the other guest that the mount is actually meant for.

Super theoretically, if the guest turned evil, it could possibly look around the entire host filesystem, or at least the entire export.  When using the hypervisor’s file sharing, only the intended directory is accessible to the guest kernel.

Pulling at threads: File Capabilities

For unimportant reasons, on my Ubuntu 24.04 installation, I went looking for things that set file capabilities in /usr/bin and /usr/sbin.  There were three:

• ping: cap_net_raw=ep
• mtr-packet: cap_net_raw=ep
• kwin_wayland: cap_sys_resource=ep

The =ep notation means that only the listed capabilities are set to “effective” and “permitted”, but not “inheritable.”  Processes can and do receive the capability, but cannot pass it to child processes.

ping and mtr-packet are “as expected.”  They want to send unusual network packets, so they need that right.  (This is the sort of thing I would also expect to see on nmap, if it were installed.)

kwin_wayland was a bit more surprising to see.  Why does it want that?  Through reading capabilities(7) and running strings /usr/bin/kwin_xwayland, my best guess is that kwin needs to raise its RLIMIT_NOFILE (max number of open files.)

There’s a separate kwin_wayland_wrapper file.  A quick check showed that it was not a shell script (a common form of wrapper), but an actual executable.  Could it have had the capability, set the limits, and launched the main process?  For that matter, could this whole startup sequence have been structured through systemd, so that none of kwin’s components needed elevated capabilities?

The latter question is easily answered: no.  This clearly isn’t a system service, and if it were run from the user instance, that never had any elevated privileges.  (The goal, as I understand it, is that a systemd user-session bug doesn’t permit privilege escalation, and “not having those privileges” is the surest way to avoid malicious use of them.)

If kwin actually adjusts the limit dynamically, in response to the actual number of clients seen, then the former answer would also be “no.”  To exercise the capability at any time, kwin itself must retain it.

I haven’t read the code to confirm any of this.  Really, it seems like this situation is exactly what capabilities are for; to allow limited actions like raising resource limits, without giving away broad access to the entire system.  Even if I were to engineer a less-privileged alternative, it doesn’t seem like it will measurably improve the “security,” especially not for cap_sys_resource.  It was just a fun little thought experiment.

Some Solutions to a Problem

We have an EC2 instance that has a quickly-produced shell script that runs on boot.  It sets a DNS name to the instance’s public IPv4 address.  Since time was of the essence, it hard-codes everything about this, particularly, the DNS name to use.

This means, if we want to bring up a copy of this instance, based on a snapshot of its root volume, the copied instance will overwrite the DNS record for the production service. We need to stop this.

As a side project, it would be nice to remove the hard-coding of the DNS name.  It would be trivial to “stop DNS name conflicts” if we did not have a DNS name stored on the instance’s disk image to begin with.

What are the options?