No More Realtek WiFi

The current Debian kernel (based on 6.1.66, after the ext4 corruption mess) seems to be locking up with the Realtek USB wireless drivers I use.  Anything that wants the IP address (like agetty or ip addr) hangs, as does shutdown.  It all works fine on the "old" kernel, which is the last version prior to the ext4 issue.

Meanwhile in Ubuntu 23.10, the in-kernel RTW drivers were flaky and bouncing the connection, so I had returned to morrownr’s driver there, as well.  But now that I don’t trust any version of this driver?  Forget this company.  In the future, I will be using any other option:

1. A Fenvi PCIe WiFi card with an Intel chip on board, or the like
2. Using an extra router as a wireless client/media bridge, with its Ethernet connected to the PC
3. If USB were truly necessary, as opposed to simply “convenient,” a Mediatek adapter

Remember that speed testing and studying dmesg output led me to the conclusion that this chipset comes up in USB 2.0 mode, and even the Windows drivers just use it that way.  While morrownr’s driver offers the ability to switch it to USB 3.0 mode under Linux, this prevents it from being connected properly.  I never researched hard enough to find out if there is a way to make that work, short of warm rebooting again so that it is already in USB 3.0 mode.

It’s clearly deficient by design, and adding injury to insult, the drivers aren’t even stable.  Awful experience, one star ★☆☆☆☆, would not recommend. Intel or Mediatek are much better choices.

Addendum, 2024-01-13: I purchased an AX200-based Fenvi card, the FV-AXE3000Pro.  It seemed not to work at all.  In Windows it would fail to start with error code 10, and in Linux it would fail to load RT ucode with error -110.  And then, Linux would report hangs for thermald, and systemd would wait forever for it to shut down.  When the timer ran out at 1m30s, it would just kick up to 3m.

Embarrassingly enough, all problems were solved by plugging it into the correct PCIe slot.  Apparently, despite being physically compatible, graphics card slots (which already had the punch-outs on my case, um, punched out) are for graphics cards only.  (My desktop is sufficiently vintage that it has two PCIe 3.0 x16 slots, one with 16 lanes and one with 4 lanes, and two classic PCI slots between them.)

Result: my WiFi is 93% faster, matching the WAN rate as seen on the Ethernet side of the router.  Good riddance, Realtek!

Diving too deeply into DH Moduli and OpenSSH

tl;dr:

• Debian/Ubuntu use the /etc/ssh/moduli file as distributed by the OpenSSH project at the time of the distribution’s release
• This file is only used for diffie-hellman-group-* KexAlgorithms
• The default KEX algorithm on my setup is the post-quantum sntrup761x25519-sha512@openssh.com instead
• Therefore, you can generate your own moduli, but it is increasingly irrelevant
• Having more moduli listed means that sshd will do more processing during every connection attempt that uses the file

There is also a “fallback” behavior if the server can’t read the moduli file or find a match, which I don’t fully understand.

Viewing the Expiration Date of an SSH Certificate

A while ago, I set up my server with SSH certificates (following a guide like this one), and today, I began to wonder: “When do these things expire?”

Host certificate (option 1)

Among the output of ssh -v (my client is OpenSSH_9.3p1 Ubuntu-1ubuntu3) is this line about the host certificate:

debug1: Server host certificate: ssh-ed25519-cert-v01@openssh.com [...] valid from 2023-04-07T19:58:00 to 2024-04-05T19:59:44

That tells us the server host certificate expiration date, where it says “valid … to 2024-04-05.”  For our local host to continue trusting the server, without using ~/.ssh/known_hosts and the trust-on-first-use (TOFU) model, we must re-sign the server key and install the new signature before that date.

User certificate

I eventually learned that ssh-keygen -L -f id_25519-cert.pub will produce some lovely human-readable output, which includes a line:

Valid: from 2023-04-07T20:14:00 to 2023-05-12T20:15:56

Aha!  I seem to have signed the user for an additional month-ish beyond the host key’s signature.  I will be able to log into the server without my key listed in ~/.ssh/authorized_keys (on the server) until 2023-05-12.

This looks like a clever protection mechanism left by my past self.  As long as I log into my server at least once a month, I'll see an untrusted-host warning before my regular authentication system goes down.  (If that happened, I would probably have to use a recovery image and/or the VPS web console to restore service.)

Host certificate (option 2)

There’s an ssh-keyscan command, which offers a -c option to print certificates instead of keys.  It turns out that we can paste its output to get the certificate validity again.  (Lines shown with $ or > are input, after that prompt; the other lines, including #, are output.)

$ ssh-keyscan -c demo.example.org
# demo.example.org:22 SSH-2.0-OpenSSH_8.9p1
# demo.example.org:22 SSH-2.0-OpenSSH_8.9p1
ssh-ed25519-cert-v01@openssh.com AAAA[.....]mcwo=
# demo.example.org:22 SSH-2.0-OpenSSH_8.9p1

The ssh-ed25519-cert line is the one we need.  We can pass it to ssh-keygen with a filename of - to read standard input, then use the shell’s “heredoc” mechanism to provide the standard input:

$ ssh-keygen -L -f - <<EOF
> ssh-ed25519-cert-v01@openssh.com AAAA[.....]mcwo=
> EOF

Now we have the same information as before, but from the host certificate.  This includes the Valid: from 2023-04-07T19:58:00 to 2024-04-05T19:59:44 line again.

Tips for setting up certificates

Document what you did, and remember the passphrases for the CA keys! This is my second setup, and now I have scripts to do the commands with all of my previous choices.  They’re merely one-line shell scripts with the ssh-keygen command.  But they still effectively record everything like the server name list, identity, validity period, and so forth.

To sign keys for multiple users/servers, it may be convenient to add the CA key to an SSH agent.  Start a separate one to keep things extra clean, then specify the signing key slightly differently:

$ ssh-agent $SHELL
$ ssh-add user-ca
$ ssh-keygen -Us user-ca.pub ...
(repeat to sign other keys)
$ exit

Note the addition of -U (specifying the CA key is in an agent) and the use of the .pub suffix (the public half of the key) in the signing process.

The Logging Tarpit

Back in August, Chris Siebenmann had some thoughts on logging:

• Monitoring your logs is mostly a tarpit
• Programs shouldn't commit to fixed and predictable log messages
• A program's (effective) log messages can have many sources – not as relevant, but linked for completeness

A popular response in the comments was “error numbers solve everything,” possibly along with a list (provided by the vendor) detailing all error numbers.

The first problem is, what if the error number changes?  MySQL changed from key index to name in their duplicate-key error, and consequently changed from error code 1062 to 1586.  Code or tools that were monitoring for 1062 would never hit a match again.  Conversely, if “unknown errors” were being monitored for emergency alerts, the appearance of 1586 might get more attention than it deserves.

In other cases, the error numbers may not capture enough information to provide a useful diagnostic.  MySQL code 1586 may tell us that there was a duplicate value for a unique key, but we need the log message to tell us which value and key.  Unfortunately, that is still missing the schema and table!

Likewise, one day, my Windows 10 PC hit a blue screen, and the only information logged was an error code for machine check 0x3E.  The message “clarified” that this was a machine check exception with code 3e.  No running thread/function, no stack trace, no context.

Finally, in some cases, logging doesn’t always capture an intent fully.  If a log message is generated, is it because of a problem, or is it operationally irrelevant?  Deciding this is the real tar pit of log monitoring, and the presence of an error number doesn’t really make a difference to it.  There’s no avoiding the decisions.

In the wake of Chris’ posts, I changed one of our hacky workaround services to log a message if it decides to take action to fix the problem.  Now we have the opportunity to find out if the service is taking action, not simply being started regularly.  Would allocating an error number (for all time) help with that?

All of this ends up guiding my log monitoring philosophy: look at the logs sometimes, and find ways to clear the highest-frequency messages. I don't want dozens of lines of known uninteresting messages clogging the log during incident response.  For example, “can’t load font A, using fallback B.”  We’d either install font A properly, or mute the message for font A, specifically.  But, I want to avoid trying to categorize every single message, because that way lies madness.

AWS: Requesting gp3 Volumes in SSM Automation Documents

I updated our EC2 instance-building pipeline to use the gp3 volume type, which offers more IOPS at lower costs.

Our initial build runs as an SSM [Systems Manager] Automation Document.  The first-stage build instance is launched from an Ubuntu AMI (with gp2 storage), and produces a “core” image with our standard platform installed.  This includes things like monitoring tools, our language runtime, and so forth.  The core image is then used to build final AMIs that are customized to specific applications.  That is, the IVR system, Drupal, internal accounting platform, and antique monolith all have separate instances and AMIs underlying them.

Our specific SSM document uses the aws:runInstances action, and one of the optional inputs to it is BlockDeviceMappings.  Through some trial and error, I found that the value it requires is the same structure as the AWS CLI uses:

- DeviceName: "/dev/sda1"
  Ebs:
    VolumeType: gp3
    Encrypted: true
    DeleteOnTermination: true

Note 1: this is in YAML format, which requires spaces for indentation.  Be sure “Ebs” is indented two spaces, and the subsequent lines four spaces.  The structure above is a 1-element array, containing a dictionary with two keys, and the “Ebs” key is another dictionary (with 3 items.)

Note 2: the DeviceName I am using comes from the Ubuntu AMI that I am using to start the instance.  DeviceName may vary with different distributions.  Check the AMI you are using for its root device setting.

The last two lines (Encrypted and DeleteOnTermination) may be unnecessary, but I don’t like leaving things to chance.

Doing this in a launch template remains a mystery.  The best I have been able to do, when trying to use the launch template, Amazon warns me that it’s planning to ignore the entire volume as described in the template.  It appears as if it will replace the volume with the one from the AMI, rather than merging the configurations.

I know I have complained about Amazon in the past for not providing a “launch from template” operation in SSM, but in this case, it appears to have worked out in my favor.

Using Cloudflare DNS over TLS system-wide on Ubuntu

My current Linux distributions (Ubuntu 23.04 and the Ubuntu-derived Pop!_OS 22.04) use NetworkManager for managing connections, and systemd-resolved for resolving DNS queries.  I’ve set up Cloudflare’s public DNS service with DoT (DNS over TLS) support twice… and I don’t really have a solid conclusion.  Is one “better?”  🤷🏻

Contents

• Per-connection mode with NetworkManager only
• Globally with systemd-resolved / NetworkManager
• Useful background info

Update on earlyoom

Back in Linux Behavior Without Swap, I noted that the modern Linux kernel will still let the system thrash.  The OOM killer does not come out until it is extremely desperate, long after responsiveness is near zero.

It has been long enough since installing earlyoom on our clouds that I did another stupid thing.  earlyoom was able to terminate the script, and the instance stayed responsive.

I also mentioned “swap on zram” in that post.  It turns out, the ideal use case for zram is when there is no other swap device. When there’s a disk-based swap area (file or partition), one should activate zswap instead.  zswap acts as a front-end buffer to swap, storing the compressible pages, or letting others go to the swap device.

One other note, zswap is compiled into the default Ubuntu kernels, but zram is part of the rather large linux-modules-extra package set.  If there’s no other need for the extra modules, uninstalling them saves a good amount of disk space.

Upgrading a debootstrap'ped Debian Installation

For context, last year, I created a Debian 11 recovery partition using debootstrap for recovering from issues with the 88x2bu driver I was using.

This year, I realized while reading that I have never used anything but Ubuntu’s do-release-upgrade tool to upgrade a non-rolling-release system.  My Debian 12 desktop felt far less polished than Ubuntu Studio, so I reinstalled the latter, and that means I once again don’t need the 88x2bu driver.

Therefore, if I trashed the Debian partition, it wouldn’t be a major loss.  It was time to experiment!

Doing the upgrade

The update process was straightforward, if more low-level than do-release-upgrade.  There are a couple of different procedures online that vary in their details, so I ended up winging it combining them:

1. apt update
2. apt upgrade
3. apt full-upgrade
4. apt autoremove --purge
5. [edit my sources from bullseye to bookworm]
6. apt clean
7. apt update
8. apt upgrade --without-new-pkgs
9. apt full-upgrade
10. reboot
11. apt autoremove

DKMS built the 88x2bu driver for the new kernel, and userspace appeared to be fine.

Fixing the Network

The link came up with an IP, but the internet didn’t work: there was no DNS.  I didn’t have systemd-resolved, named, dnsmasq, nor nscd.  Now, to rescue the rescue partition, I rebooted into Ubuntu, chroot’ed to Debian, and installed systemd-resolved.

Fixing Half of the Races

One of the Debian boots left me confused.  Output to the console appeared to have stopped after some USB devices were initialized.  I thought it had crashed.  I unplugged a keyboard and plugged it in, generating more USB messages on screen, so I experimentally pressed Enter.  What I got was an (initramfs) prompt!  The previous one had been lost in the USB messages printed after it had appeared.

It seems that the kernel had done something different in probing the SATA bus vs. USB this time, and /dev/sdb3 didn’t have the root partition on it.  I ended up rebooting (I don’t know how to get the boot to resume properly if I had managed to mount /root by hand.)

When that worked, I updated the Ubuntu partition’s /boot/grub/custom.cfg to use the UUID instead of the device path for Debian.

It seems that the kernel itself only supports partition UUIDs, but Debian and Ubuntu use initrds (initial RAM disks) that contain the code needed to find the filesystem UUID.  That’s why root=UUID={fs-uuid} has always worked for me!  Including this time.

os-prober (the original source of this entry) has to be more conservative, though, so it put root=/dev/sdb3 on the kernel command line instead.

The Unfixed Race

Sometimes, the wlan0 interface can’t be renamed to wlx{MAC_ADDRESS} because the device is busy.  I tried letting wlan0 be an alias in the configuration for the interface (using systemd-networkd) but it doesn’t seem to take.

I resort to simply rebooting if the login prompt doesn’t reset itself and show a DHCP IP address in the banner within a few seconds.

You have to admire the kernel and systemd teams’ dedication to taking a stable, functional process and replacing it with a complex and fragile mess.

A Brief Flirtation with X11

I installed Cinnamon.  It ran out of space; I ran apt clean and then continued, successfully.  This is my fault; I had “only” given the partition 8 GiB, because I expected it to be a CLI-only environment.

Cinnamon, however, is insistent on NetworkManager, and I was already using systemd-networkd.  It’s very weird to have the desktop showing that there is “no connection” while the internet is actually working fine.

Due to the space issue, I decided to uninstall everything and go back to a minimal CLI.  I would definitely not be able to perform another upgrade to Debian 13, for instance, and it was unclear if I would even be able to do normal updates.

In Conclusion

The Debian upgrade went surprisingly well, considering it was initially installed with debootstrap, and is therefore an unusual configuration.

Losing DNS might have been recoverable by editing /etc/resolv.conf instead, but I wasn’t really in a “fixing this from here is my only option” space.  Actually, one might wonder what happened to the DHCP-provided DNS server?  I don’t know, either.

Trying to add X11 to a partition never designed for it did not work out, but it was largely a whim anyway.

Sound for Firefox Flatpak on Ubuntu 23.04 with PipeWire

I reinstalled Ubuntu Studio recently, excised all Snaps, and installed Firefox from Flatpak.  Afterward, I didn’t have any audio output in Firefox.  Videos would just freeze.

I don’t know how much of this is fully necessary, but I quit Firefox, installed more PipeWire pieces, and maybe signed out and/or rebooted.

sudo apt install pipewire-audio pipewire-alsa

The pipewire-jack and pipewire-pulse packages were already installed.

AIUI, this means that “PipeWire exclusively owns the audio hardware” and provides ALSA, JACK, Pulse, and PipeWire interfaces into it.

It’s not perfect.  Thunderbird (also flatpak; I’d rather have “some” sandbox than “none”) makes a bit of cacophony when emails come in, but at least there’s sound for Firefox.

Boring Code Survives

Over on Wandering Thoughts, Chris writes about some fileserver management tools being fairly unchanged over time by changes to the environment.  There is a Python 2 to 3 conversion, and some changes when the disks being managed are no longer on iSCSI, “but in practice a lot of code really has carried on basically as-is.”

This is completely different than my experience with async/await in Python.  Async was new, so the library I used with it was in 0.x, and in 1.0, the authors inverted the entire control structure. Instead of being able to create an AWS client deep in the stack and return it upwards, clients could only be used as context managers.  It was quite a nasty surprise.

To allow testing for free, my code dynamically instantiated a module to “manage storage,” and whether that was AWS or in-memory was an implementation detail.  Suddenly, one of the clients couldn’t write self.client = c; return anymore.  The top-level had to know about the change.  Other storage clients would have to know about the change, to become context managers themselves, for no reason.

I held onto the 0.x version for a while, until the Python core team felt like “explicit event loop” was a mistake big enough that everyone’s code had to be broken.

Async had been hard to write in the first place, because so much example code out there was for the asyncio module’s decorators, which had preceded the actual async/await syntax.  What the difference between tasks and coroutines even was, and why one should choose one over the other, was never clear.  Why an explicit loop parameter should exist was especially unclear, but it was “best practice” to include it everywhere, so everyone did.  Then Python set it on fire.

(I never liked the Python packaging story, and pipenv didn’t solve it. To pipenv, every Python minor version is an incompatible version?)

I had a rewrite on my hands either way, so I went looking for something else to rewrite in, and v3 is in Go.  The other Python I was using in my VM build pipeline was replaced with a half-dozen lines of shell script.  It’s much less flexible, perhaps, but it’s clear and concise now.

In the end, it seems that boring code survives the changing seasons.  If you’re just making function calls and doing some regular expression work… there’s little that’s likely to change in that space.  If you’re coloring functions and people are inventing brand-new libraries in the space you’re working in, your code will find its environment altered much sooner.  The newer, fancier stuff is inherently closer to the fault-line of future shifts in the language semantics.

Installing Debian 12 and morrownr's Realtek driver (2023-07-03 edit)

Due to deepening dissatisfaction with Canonical, I replaced my Ubuntu Studio installation with Debian 12 “bookworm” recently.

tl;dr:

1. My backups, including the driver source, were compressed with lzip for reasons, but I fell back on a previously-built rescue partition to get the system online.
2. I ended up with an improper grub installation, that couldn’t find /boot/grub/i386-pc/normal.mod. I rebooted the install media in rescue mode, got a shell in the environment, verified the disk structure with fdisk -l, and then ran grub-install /dev/___ to fix it. Replace the blank with your device, but beware: using the wrong device may make the OS on it unbootable.
3. The USB doesn’t work directly with apt-cdrom to install more packages offline. I “got the ISO back” from dd if=/dev/sdd of=./bookworm.iso bs=1M count=4096 status=progress conv=fsync (1M * 4096 = 4G total, which is big enough for the 3.7G bookworm image; you may need to adjust to suit), then made it available with mount -o loop,ro ~+/bookworm.iso /media/cdrom0 (the mount point is the target of /media/cdrom.)
4. Once finished, I found out the DVD had plzip, and if I’d searched for it (lzip), I could have used it (plzip). I didn’t actually need the rescue partition.
5. Once finished, I realized I hadn’t needed to dd the ISO back from the USB stick. The downloaded ISO was on my external drive all along, and I could have loop-mounted that.
6. [Added 2023-07-02]: Letting the swap partition get formatted gave it a new UUID. Ultimately, I would need to update the recovery partition’s /etc/fstab with the new UUID, and follow up with update-initramfs -u to get the recovery partition working smoothly again.

Full, detailed rambling with too much context (as usual) below.

Passing data from AWS EventBridge Scheduler to Lambda

The documentation was lacking images, or even descriptions of some screens ("Choose Next. Choose Next.")  So, I ran a little experiment to test things out.

When creating a new scheduled event in AWS EventBridge Scheduler, then choosing AWS Lambda: Invoke, a field called "Input" will be available.  It's pre-filled with the value of {}, that is, an empty JSON object. This is the value that is passed to the event argument of the Lambda handler:

export async function handler(event, context) {
  // handle the event
}

With an event JSON of {"example":{"one":1,"two":2}}, the handler could read event.example.two to get its value, 2.

It appears that EventBridge Scheduler allows one complete control over this data, and the context argument is only filled with Lambda-related information.  Therefore, AWS provides the ability to include the <aws.scheduler.*> values in this JSON data, to be passed to Lambda (or ignored) as one sees fit, rather than imposing any constraints of its own on the data format.  (Sorry, no examples; I was only testing the basic features.)

Note that the handler example above is written with ES Modules.  This requires the Node 18.x runtime in Lambda, along with a filename of "index.mjs".

DejaDup and PikaBackup, early impressions (Update 2)

I tried a couple of backup programs:

• DejaDup (the one I had heard of), a front-end for Duplicity
• PikaBackup, a front-end for BorgBackup

I installed both of them as Flatpaks, although deja-dup also has a version in the Pop!_OS 22.04 repository.  I have been using DejaDup for four months, and PikaBackup for one month.  This has been long enough for DejaDup to make a second full backup, but not so long for Pika to do anything special.

Speed:

For a weekly incremental backup of my data set…

• DejaDup: about 5 minutes, lots of fan speed changes
• PikaBackup: about 1 minute, fans up the whole time

Part of Pika’s speed is probably the better exclusion rules; I can use patterns of **/node_modules and **/vendor, to exclude those folders, wherever they are in the tree.  With DejaDup, I would apparently have to add each one individually, and I did not want to bother, nor keep the list up-to-date over time.

Part of DejaDup’s slowness might be that it executes thousands of gpg calls as it works.  Watching with top, DejaDup is frequently running, and sometimes there’s a gpg process running with it.  Often, DejaDup is credited with much less than 100% of a single CPU core.

Features:

PikaBackup offers multiple backup configurations.  I keep my main backup as a weekly backup, on an external drive that’s only plugged in for the occasion. I was able to configure an additional hourly backup of my most-often-changed files in Pika.  (This goes into ~/.borg-fast, which I excluded from the weekly backups.) The hourly backups, covering about 2 GB of files, aren’t noticeable at all when using the system.

Noted under “speed,” PikaBackup offers better control of exclusions.  It tracks how long operations took, so I know that it has been exactly 53–57 seconds to make the incremental weekly backups.

On the other hand, Pika appears to always save the backup password.  DejaDup gives the user the option of whether it should be remembered.

There is a DejaDup plugin for Caja (the MATE file manager) in the OS repo, which may be interesting to MATE users.

Space Usage:

PikaBackup did the weekly backup on 2023-04-24 in 46 seconds; it reports a total backup size of 28 GB and 982 MB (0.959 GB = 3.4%) written out.

With scheduled backups, Pika offers control of the number of copies kept.  One can choose from a couple of presets, or provide custom settings.  Of note, these are count-based rather than time-based; if a laptop is only running for 8-9 hours a day, then 24 hourly backups will be able to provide up to 3 days back in time.

For unscheduled backups, it’s not clear that Pika offers any ‘cleanup’ options, because the cleanup is tied to the schedule in the UI.

I do not remember being given many options to control space usage in DejaDup.

Disaster Simulation:

To ensure backups were really encrypted, I rebooted into the OS Recovery environment and tried to access them.  Both programs’ CLI tools (duplicity and borgbackup) from the OS repository were able to verify the data sets. I don’t know what the stability guarantees are, but it’s nice that this worked in practice.

• duplicity verified the DejaDup backup in about 9m40s
• borgbackup verified the PikaBackup backup in 3m23s

This isn’t a benchmark at all; after a while, I got bored of duplicity being credited with 30% of 1 core CPU usage, and started the borgbackup task in parallel.

Both programs required the password to unlock the backup, because my login keychain isn’t available in this environment.

Curiously, borgbackup changed the permissions on a couple of files on the backup during the verification: the config and index files became owned by root.  This made it impossible to access the backups as my normal user, including to take a new one.  I needed to return to my admin user and set the ownership back to my limited account.  The error message made it clear an unexpected exception occurred, but wasn’t very useful beyond that.

Major limitations of this post:

My data set is a few GB, consisting mainly of git repos and related office documents.  The performance of other data sets is likely to vary.

I started running Pika about the same time that DejaDup wanted to make a second backup, so the full-backup date and number of incremental snapshots since should be fairly close to each other.  I expect this to make the verification times comparable.

I haven’t actually done any restores yet.

Final words:

Pika has become my primary backup method.  Together, its speed and its support for multiple configurations made hourly backups reasonable, without compromising the offline weekly backup.

Update History:

This post was updated on 2023-03-31, to add information about multiple backups to “Features,” and about BorgBackup’s file permission change during the verification test.  Links were added to the list above, and a new “Final Words” concluding section was written.

It was updated again on 2023-04-26, to add the “Space Usage” section, and to reduce “I will probably…” statements to reflect the final decisions made.

Using sshuttle with ufw outbound filtering on Linux (Pop!_OS 22.04)

I am using sshuttle and UFW on my Linux system, and I recently set up outbound traffic filtering (instead of default-allow) in ufw.  Immediately, I noticed I couldn’t make connections via sshuttle anymore.

The solution was to add another rule to ufw:

allow out from anywhere to IP 127.0.0.1, TCP port 12300

Note that this is “all interfaces,” not tied to the loopback interface, lo.

Now… why does this work?  Why doesn’t this traffic already match one of the “accept all on loopback” rules?

To receive that sshuttle is responsible for, sshuttle listens at 127.0.0.1:12300 (by default) and creates some NAT rules to redirect traffic for its subnet to that IP and port.  That is, running sshuttle -r example.com 192.168.99.0/24 creates a NAT rule to catch traffic to any host within 192.168.99.0/24.  This is done in netfilter’s nat tables.

UFW has its rules in the filter tables, and the nat tables run first. Therefore, UFW sees a packet that has already been redirected, and this redirection changes the packet’s destination while its interface and source remain the same!

That’s the key to answering the second question: the “allow traffic on loopback” rules are written to allow traffic on interface lo, and these redirected packets have a different interface (Ethernet or Wi-Fi.) The public interfaces are not expected to have traffic for local addresses on them… but if they do, they don’t get to take a shortcut through the firewall.

With this understanding, we can also see what’s going wrong in the filtering rules.  Without a specific rule to allow port 12300 outbound, the packet reaches the default policy, and if that’s “reject” or “deny,” then the traffic is blocked.  sshuttle never receives it.

Now we can construct the proper match rule: we need to allow traffic to IP 127.0.0.1 on TCP port 12300, and use either “all interfaces” or our public (Ethernet/Wi-Fi) interface.  I left mine at “all interfaces,” in case I should ever plug in the Ethernet.

(I admit to a couple of dead-ends along the way.  One, allowing port 3306 out didn’t help.  Due to the NAT redirection, the firewall never sees a packet with port 3306 itself.  This also means that traffic being forwarded by sshuttle can’t be usefully firewalled on the client side.  The other problem was that I accidentally created the rule to allow UDP instead of TCP the first time.  Haha, oops.)

On Handling Web Forms [2012]

Editor’s Note: I found this in my drafts from 2012.  The login form works as described, but few of the other forms do.  However, the login form has not had glitches, even with 2FA being added to it recently.  The site as a whole retains its multi-page architecture.  Without further ado, the original post follows…

I’ve been working on some fresher development at work, which is generally an opportunity for a lot of design and reflection on that design.

Back in 2006 or so, I did some testing and tediously developed the standard “302 redirect on POST responses” technique for preventing pages that handled inbound form data from showing up in the browser history.  Thus, sites would be Back- and Reload-friendly, as they’d never show that “About to re-submit a form, do you really want to?” box.  (I would bet it was a well-known technique at the time, but NIH.)

That’s pretty much how I’ve written my sites since, but a necessary consequence of the design is that on submission failure, data for the form must be stored “somewhere,” so it can be retrieved for pre-filling the form after the redirection completes.

My recent app built in this style spends a lot of effort on all that, and then throws in extra complexity: when detecting you’re logged out, it decides to minimize the session storage size, so it cleans up all the keys.  Except for the flash, the saved form data, and the redirection URL.

That latter gets stored because I don’t trust Referer as a rule, and if login fails and redirects to the login page for a retry, it won’t be accurate by the time a later attempt succeeds.  So every page that checks login also stores the form URL to return to after a login.

There’s even an extra layer of keys in the form area, so that each form’s data is stored independently in the session, although I don’t think people actually browse multiple tabs concurrently.  All that is serving to do is bloat up the session when a form gets abandoned somehow.

Even then, it still doesn't work if the form was inside a jQueryUI dialog, because I punted on that one.  The backend and page don’t know the dialog was open, and end up losing the user’s data.

Simplify, Young One

That's a whole lot of complexity just to handle a form submission.  Since the advent of GMail, YouTube, and many other sites which are only useful with JavaScript enabled, and since this latest project is a strictly internal app, I've decided to throw all that away and try again.

Now, a form submits as an AJAX POST and the response comes back.  If there was an error, the client-side code can inform the user, and no state needs saved/restored because the page never reloaded. All the state it had is still there.

But while liberating, that much is old news.  “Everyone” builds single-page apps that way, right?

But here’s the thing: if I build out separate pages for each form, then I’m effectively building private sections of code and state from the point of view of “the app” or “the whole site.”  No page is visible from any other, so each one only has to worry about a couple of things going on in the global navbar.

This means changes roll out more quickly, as well, since users do reload when moving from area to area of the app.

Argon2id Parameters

There are no specific recommendations in this post. You will need to choose parameters for your specific situation.  However, it is my hope that this post will add deeper understanding to the recommendations that others make.  With that said…

The primary parameter for Argon2 is memory. Increasing memory also increases processing time.

The time cost parameter is intended to make the running time longer when memory usage can’t be increased further.

The threads (or parallelism, or “lanes” when reading the RFC) parameter sub-divides the memory usage.  When the memory is specified as 64 MiB, that is the total amount used, whether threads are 1 or 32.  However, the synchronization overhead causes a sub-linear speedup, and this is more pronounced with smaller memory sizes.  SMT cores offer even less speed improvement than the same number of non-SMT cores, as expected.

I did some tests on my laptop, which has 4 P-cores and 8 E-cores (16 threads / 12 physical cores.) The 256 MiB tests could only push actual CPU usage to about 600% (compared to the 1260% we might expect); it took 1 GiB or more to reach 1000% CPU.  More threads than cores didn’t achieve anything.

Overall then, higher threads allow for using more memory, if enough CPU is available to support the thread count.  If memory and threads are both in limited supply, then time cost is the last resort for extending the operation time until it takes long enough.

Bonus discovery: in PHP, the argon2id memory is separate from the memory limit.  memory_get_peak_usage() reported the same number at the beginning and end of my test script, even for the 1+ GiB tests.

Experiences with AWS

Our core infrastructure is still EC2, RDS, and S3, but we interact with a much larger number of AWS services than we used to.  Following are quick reviews and ratings of them.

CodeDeploy has been mainly a source of irritation.  It works wonderfully to do all the steps involved in a blue/green deployment, but it is never ready for the next Ubuntu LTS after it launches.  As I write, AWS said they planned to get the necessary update out in May, June, September, and September 2022; it is now January 2023 and Ubuntu 22.04 support has not officially been released. Ahem. 0/10 am thinking about writing a Go daemon to manage these deployments instead.  I am more bitter than a Switch game card.

CodeBuild has ‘environments’ thrown over the wall periodically.  We added our scripts to install PHP from Ondřej Surý’s PPA instead of having the environment do it, allowing us to test PHP 8.1 separately from the Ubuntu 22.04 update.  (Both went fine, but it would be easier to find the root cause with the updates separated, if anything had failed.) “Build our own container to route around the damage” is on the list of stuff to do eventually.  Once, the CodeBuild environment had included a buggy version of git that segfaulted unless a config option was set, but AWS did fix that after a while.  9/10 solid service that runs well, complaints are minor.

CodeCommit definitely had some growing pains.  It’s not as bad now, but it remains obviously slower than GitHub.  After a long pause with 0 objects counted, all objects finish counting at once, and then things proceed pretty well.  The other thing of note is that it only accepts RSA keys for SSH access.  6/10 not bad but has clearly needed improvement for a long time.  We are still using it for all of our code, so it’s not terrible.

CodePipeline is great for what it does, but it has limited built-in integrations.  It can use AWS Code services… or Lambda or SNS.  8/10 conceptually sound and easy to use as intended, although I would rather implement my own webhook on an EC2 instance for custom steps.

Lambda has been quarantined to “only used for stuff that has no alternative,” like running code in response to CodeCommit pushes.  It appears that we are charged for the wall time to execute, which is okay, but means that we are literally paying for the latency of every AWS or webhook request that Lambda code needs to make.  3/10 all “serverless” stuff like Lambda and Fargate are significantly more expensive than being server’d.  Would rather implement my own webhook on an EC2 instance.

SNS [Simple Notification Service] once had a habit of dropping subscriptions, so our ALB health checks (formerly ELB health checks) embed a subscription-monitor component that automatically resubscribes if the instance is dropped.  One time, I had a topic deliver to email before the actual app was finished, and the partner ran a load test without warning.  I ended up with 10,000 emails the next day, 0 drops and 0 duplicates.  9/10 has not caused any trouble in a long time, with plenty of usage.

SQS [Simple Queue Service] has been 100% perfectly reliable and intuitive. 10/10 exactly how an AWS service should run.

Secrets Manager has a lot of caching in front of it these days, because it seems to be subject to global limits.  We have observed throttling at rates that are 1% or possibly even less of our account’s stated quota.  The caching also helps with latency, because they are either overloaded (see previous) or doing Serious Crypto that takes time to run (in the vein of bcrypt or argon2i).  8/10 we have made it work, but we might actually want AWS KMS instead.

API Gateway has ended up as a fancy proxy service.  Our older APIs still have an ‘API Definition’ loaded in, complete with stub paths to return 404 instead of the default 403 (which had confused partners quite a bit.) Newer ones are all simple proxies.  We don’t gzip-encode responses to API Gateway because it failed badly in the past. 7/10 not entirely sure what value this provides to us at this point.  We didn’t end up integrating IAM Authentication or anything.

ACM [AWS Certificate Manager] provides all of our certificates in production.  The whole point of the service is to hide private keys, so the development systems (not behind the load balancer) use Let’s Encrypt certificates instead.  10/10 works perfectly and adds security (vs. having a certificate on-instance.)

Route53 Domains is somewhat expensive, as far as registering domains goes, but the API accessibility and integration with plain Route53 are nice.  It is one of the top-3 services on our AWS bill because we have a “vanity domain per client” architecture.  9/10 wish there was a bulk discount.

DynamoDB is perfect for workloads that suit non-queryable data, which is to say, we use it for sessions, and not much else.  It has become usable in far more scenarios with the additions of TTL (expiration times) and secondary indexes, but still remains niche in our architecture.  9/10 fills a clear need, just doesn’t match very closely to our needs.

CloudSearch has been quietly powering “search by name” for months now, without complaints from users.  10/10 this is just what the doctor ordered, plain search with no extra complexity like “you will use this to parse logs, so here are extra tools to manage!”

That’s it for today.  Tune in next time!

FastCGI in Perl, but PHP Style [2012]

Editor's note: I found this in my drafts from 2012. By now, everything that can be reasonably converted to FastCGI has been, and a Perl-PHP bridge has been built to allow new code to be written for the site in PHP instead. However, the conclusion still seems relevant to designers working on frameworks, so without further ado, the original post follows...

The first conversions of CGI scripts to FastCGI have been launched into production. I have both the main login flow and six of the most popular pages converted, and nothing has run away with the CPU or memory in the first 50 hours. It’s been completely worry-free on the memory front, and I owe it to the PHP philosophy.

In PHP, users generally don’t have the option of persistence. Unless something has been carefully allocated in persistent storage in the PHP kernel (the C level code), everything gets cleaned up at the end of the request. Database connections are the famous example.

Perl is obviously different, since data can be trivially kept by using package level variables to stash data, but my handler-modules (e.g. Site::Entry::login) don’t use them. Such handler-modules define one well-known function, which returns an object instance that carries all the necessary state for the dispatch and optional post-dispatch phases. When this object is destroyed in the FastCGI request loop, so too are all its dependencies.

Furthermore, dispatching returns its response, WSGI style, so that if dispatch dies, the FastCGI loop can return a generic error for the browser. Dispatch isn’t allowed to write anything to the output stream directly, including headers, which guarantees a blank slate for the main loop’s error page. (I once wrote a one-pass renderer, then had to grapple with questions like “How do I know whether HTML has been sent?”, “How do I close half-sent HTML?”, and “What if it’s not HTML?” in the error handler.)

PHP’s PDO, Single-Process Testing, and 'Too Many Connections'

Quite some time ago now, I ran into a problem with running a test suite: at some point, it would fail to connect to the database, due to too many connections in use.

Architecturally, each connection sent a PSR-7 Request through the HTTP layer, which caused the back-end code under test to connect to the database in order to fulfill the request.  All of these resources (statement handles and the database handle itself) should have been out of scope be the end of the request.

But every PDOStatement has a reference to its parent PDO object, and apparently each PDO keeps a reference to all of its PDOStatements.  There was no memory pressure (virtually all other allocations were being cleaned up between tests), so PHP wasn’t trying to collect cycles, and the PDO objects were keeping connections open the whole duration of the test suite.

Lowering the connection limit in the database engine (a local, anonymized copy of production data) caused the failure to occur much sooner in testing, proving that it was an environmental factor and not simply “unlucky test ordering” that caused the failure.

Using phpunit’s --process-isolation cured the problem entirely, but at the cost of a lot of time overhead.  This was also expected: with the PHP engine shut down entirely between tests, all of its resources (including open database connections) were cleaned up by the OS.

Fortunately, I already had a database connection helper for other reasons: loading credentials securely, setting up exceptions as the error mode, choosing the character set, and retrying on failure if AWS was in the middle of a failover event (“Host not found”, “connection refused”, etc.) It was a relatively small matter to detect “Too many connections” and, if it was the first such error, issue gc_collect_cycles() before trying again.

(Despite the “phpunit” name, there are functional and integration test suites for the project which are also built on phpunit.  Then, the actual tests to run are chosen using phpunit --testsuite functional, or left at the default for the unit test suite.)