Linux Behavior Without Swap

We had a runaway script clog all of the memory on a micro EC2 Ubuntu instance. Not enough that the kernel OOM killer would do anything, and not enough that the script itself hit the PHP memory limit, but enough to make the instance become unresponsive for 45 minutes.

I have sent Linux into thrashing, back in the old days when typical desktop RAM sizes were less than 1 GB and SSDs weren’t available yet.  What surprised me was just how similar “running out of RAM” was in the modern times, even with the OOM killer.  It let the system bog down instead of killing a process!

We chose to mitigate the issue at work by expanding the instance, so that it has more RAM than memory_limit now.  It will take more than one simultaneous runaway script to bring it down in the future.  (We also fixed the script.  I don’t like throwing resources at problems, in general.)

Then one day, via pure serendipity, I found out about earlyoom.  I have added it to our pet instances, and I’m considering it for the cattle template, but it hasn’t been well-tested due to our previous mitigations.  The instance simply doesn’t run out of RAM anymore.

At home, I first set up swap on zram so that Ubuntu Studio would have a place to “swap out” 2+ GB (out of 12 GB installed), and then recently added a swap partition while I was restructuring things anyway.  It’s not great for realtime audio to swap; but “not having swap” doesn’t appear to change the consequences of memory pressure, so I put some swap in.  With a dedicated swap partition added, I reduced the zram area to 512 MB.  I still want to save the SSD if there’s a small-to-moderate amount of swap usage.

UPDATE: This was imperfect, as it turns out; if you have a swap partition, you should remove zram, and use zswap instead.

debootstrap’ing a Recovery Partition

One of the nicer things about trying Fedora on my work laptop is that when it broke the boot loader, there was a functioning Recovery Mode.

My desktop relies on a particular driver for WiFi, and upgrading the kernel (e.g. from Ubuntu 22.04 to 22.10) requires fully reinstalling it.  But what if the kernel upgrades to a version that isn’t supported by the copy of the driver I happened to have on disk?  And I didn’t want to “just” (disassemble and move the PC to) plug in an Ethernet cable?

I used the Fedora live environment to make a little bit of room for an 8 GiB partition at the end of the disk (and a 2 GiB swap partition, as long as I was there), and then I ran debootstrap to fill it in.  This is about what surprised me doing that.

tl;dr: debootstrap is a lot more aggressively minimalist, more like Arch, than I would have expected.

Container Memory Usage

How efficient is it to run multiple containers with the same code, serving different data?  I am most familiar with a “shared virtual hosting” setup, with a few applications behind a single Web frontend and PHP-FPM pool.  How much would I lose, trying to run each app in its own container?

To come up with a first-order approximation of this, I made a pair of minimal static sites and used the nginx:mainline-alpine image (ID 1e415454686a) to serve them.  The overall question was whether the layer would be shared between multiple containers in the Linux memory system, or whether each image would end up with its own copy of everything.

Updated 2022-12-19: This post has been substantially rewritten and expanded, because Ubuntu would not cleanly reproduce the numbers, requiring deeper investigation.

Failing to Install Fedora 37

Waiting for yet another 80+ MB download of a .deb file, I decided to try to dual-boot Fedora on my Pop!_OS laptop (darp8).  Because drpms exist.  That’s it, that’s the whole reason.

[Update 2022-12-04: Because all resize-related commands use “as much space as possible” by default, I decided to try again, shrinking the filesystem/LV/PV more than was necessary, and then re-expanding them after the partition was changed.  I got Fedora installed, but the system76 extension for the power manager doesn’t work on Gnome 43. Gnome claims it’s not their fault that the system they created frequently breaks extensions in general; I assume they feel the same about the specific case here, where they changed the menu API instantly and completely.]

[Update 2022-12-19: I pretty quickly gave up on Fedora.  It had a tendency to result in the laptop rebooting into recovery mode after using the Fedora install.  Booting Linux is clearly not important enough to get standardized.  Oh well!]