Sunday, May 19, 2024

Everything Fails, FCGI::ProcManager::Dynamic Edition

I have been reading a lot of rachelbythebay, which has led me to thinking about the reliability of my own company’s architecture.

It’s top of my mind right now, because an inscrutable race condition caused a half-hour outage of our primary site.  It was a new, never-before-seen condition that slipped right past all our existing defenses.  Using Site as a placeholder for our local namespace, it looked like this:

use Try::Tiny qw(try catch);
try {
  require Site::Response;
  require Site::Utils;
} catch {
  # EX_PRELOAD => exit status 1
  exit_manager(EX_PRELOAD, $_);
$res = Site::Response->new();

Well, this time—this one time—on both running web servers… it started throwing an error that Method "new" wasn't found in package Site::Response (did you forget to load Site::Response?).  Huh?  Of course I loaded it; I would’ve exited if that had failed.

In response, I added a lot more try/catch, exit_manager() has been improved, and there is a separate site-monitoring service that will issue systemctl restart on the site, if it starts rapidly cycling through workers.

Sunday, May 12, 2024

Using tarlz with GNU tar

I have an old trick that looks something like:

$ ssh HOST tar cf - DIR | lzip -9c >dir.tar.lz

The goal here is to pull a tar from the server, compressing it locally, to trade bandwidth and client CPU for reduced server CPU usage.  I keep this handy for when I don’t want to disturb a small AWS instance too much.

Since then, I learned about tarlz, which can compress an existing tar archive with lzip.  That seemed like what I wanted, but na├»ve usage would result in errors:

$ ssh HOST tar cf - DIR | tarlz -z -o dir.tar.lz
tarlz: (stdin): Corrupt or invalid tar header.

It turned out that tarlz only works on archives in POSIX format, and (modern?) GNU tar produces them in GNU format by default.  Pass it the --posix option to make it all work together:

$ ssh HOST tar cf - --posix DIR | \
    tarlz -z -o dir.tar.lz

(Line broken on my blog for readability.)

Bonus tip: it turns out that GNU tar will auto-detect the compression format on read operations these days.  Running tar xf foo.tar.lz will transparently decompress the archive with lzip.

Tuesday, April 30, 2024

Things I learned Reinstalling My Ubuntu

I did not want to wait for Ubuntu Studio 24.04 to be offered as an update to 23.10, so I got the installer and tried it.  Also, I thought I would try repartitioning the disk as UEFI.

Brief notes:

  • I did not feel in control of manual partitioning
  • I found out one of my USB sticks is bad, thanks to F3…
  • …and no thanks to the Startup Disk Creator!
  • If the X11 window manager crashes/doesn’t start, goofy things happen
  • Wayland+KWin still don’t support sticky keys, smh
  • snap remove pops up the audio device overlay… sometimes repeatedly
  • I depend on a surprising amount of configuration actually

Tuesday, April 23, 2024

Getting fail2ban Working [with my weird choices] on Ubuntu 22.04 (jammy)

To put the tl;dr up front:

  1. The systemd service name may not be correct
  2. The service needs to be logging enough information for fail2ban to process
  3. Unrelatedly, Apple Mail on iPhone is really bad at logging into Dovecot
  4. Extended Research

[2024-04-26: Putting the backend in the DEFAULT section may not actually work on all distributions.  One may need to copy it into each individual jail (sshd, postfix, etc.) for it to take effect.]

A minimalist /etc/fail2ban/jail.local for a few services, based on mine:

backend = systemd
enabled = true
journalmatch = _SYSTEMD_UNIT=ssh.service + _COMM=sshd
enabled = true
journalmatch = _SYSTEMD_UNIT=postfix@-.service
enabled = true
journalmatch = _SYSTEMD_UNIT=pure-ftpd.service

(The journalmatch for pure-ftpd removes the command/_COMM field entirely.)

Sunday, March 3, 2024

vimrc tips

On Debian-family systems, vim.tiny may be providing the vim command, through the alternatives system. If I bring in my dotfiles and haven’t installed a full vim package yet, such as vim-gtk3, then dozens of errors might show up.  vim.tiny really does not support many features.

Other times, I run gvim -ZR for quickly checking some code, to get read-only restricted mode.  In that case, anything that wants to run a shell command will fail.  Restricted mode is also a signal that I don’t trust the files I’m viewing, so I don’t want to process their modelines at all.

To deal with these scenarios, my vimrc is shaped like this (line count heavily reduced for illustration):

set nocompatible ruler laststatus=2 nomodeline modelines=2
if has('eval')
    call plug#begin('~/.vim/plugged')
        call system('true')
        Plug 'dense-analysis/ale'
        Plug 'mhinz/vim-signify' | set updatetime=150
        Plug 'pskpatil/vim-securemodelines'
    catch /E145/
    Plug 'editorconfig/editorconfig-vim'
    Plug 'luochen1990/rainbow'
    Plug 'tpope/vim-sensible'
    Plug 'sapphirecat/garden-vim'
    Plug 'ekalinin/Dockerfile.vim', { 'for': 'Dockerfile' }
    Plug 'rhysd/vim-gfm-syntax', { 'for': 'md' }
    Plug 'wgwoods/vim-systemd-syntax', { 'for': 'service' }
    call plug#end()
    if !has('gui_running') && exists('&termguicolors')
        set termguicolors
    let g:rainbow_active=1
    colorscheme garden

We start off with the universally-supported settings.  Although I use the abbreviated forms in the editor, my vimrc has the full spelling, for self-documentation.

Next is the feature detection of if has('eval') … endif.  This ensures that vim.tiny doesn’t process the block.  Sadly, inverting the test and using the finish command inside didn’t work.

If we have full vim, we start loading plugins, with a try-catch for restricted mode.  If we can’t run the true shell command, due to E145, we cancel the error and proceed without that subset of non-restricted plugins.  Otherwise, ALE and signify would load in restricted mode, but throw errors as soon as we opened files.

After that, it’s pretty straightforward; we’re running in a full vim, loading things that can run in restricted mode.  When the plugins are over, we finish by configuring and activating the ones that need it.

Friday, February 2, 2024

My Issues with Libvirt / Why I Kept VirtualBox

At work, we use VirtualBox to distribute and run development machines.  The primary reasons for this are:

  1. It is free (gratis), at least the portions we require
  2. It has import/export

However, it isn’t developed in the open, and it has a worrying tendency to print sanitizer warnings on the console when I shut down my laptop.

Can I replace it with kvm/libvirt/virt-manager?  Let’s try!

Sunday, December 31, 2023

No More Realtek WiFi

The current Debian kernel (based on 6.1.66, after the ext4 corruption mess) seems to be locking up with the Realtek USB wireless drivers I use.  Anything that wants the IP address (like agetty or ip addr) hangs, as does shutdown.  It all works fine on the "old" kernel, which is the last version prior to the ext4 issue.

Meanwhile in Ubuntu 23.10, the in-kernel RTW drivers were flaky and bouncing the connection, so I had returned to morrownr’s driver there, as well.  But now that I don’t trust any version of this driver?  Forget this company.  In the future, I will be using any other option:

  1. A Fenvi PCIe WiFi card with an Intel chip on board, or the like
  2. Using an extra router as a wireless client/media bridge, with its Ethernet connected to the PC
  3. If USB were truly necessary, as opposed to simply “convenient,” a Mediatek adapter

Remember that speed testing and studying dmesg output led me to the conclusion that this chipset comes up in USB 2.0 mode, and even the Windows drivers just use it that way.  While morrownr’s driver offers the ability to switch it to USB 3.0 mode under Linux, this prevents it from being connected properly.  I never researched hard enough to find out if there is a way to make that work, short of warm rebooting again so that it is already in USB 3.0 mode.

It’s clearly deficient by design, and adding injury to insult, the drivers aren’t even stable.  Awful experience, one star ★☆☆☆☆, would not recommend. Intel or Mediatek are much better choices.

Addendum, 2024-01-13: I purchased an AX200-based Fenvi card, the FV-AXE3000Pro.  It seemed not to work at all.  In Windows it would fail to start with error code 10, and in Linux it would fail to load RT ucode with error -110.  And then, Linux would report hangs for thermald, and systemd would wait forever for it to shut down.  When the timer ran out at 1m30s, it would just kick up to 3m.

Embarrassingly enough, all problems were solved by plugging it into the correct PCIe slot.  Apparently, despite being physically compatible, graphics card slots (which already had the punch-outs on my case, um, punched out) are for graphics cards only.  (My desktop is sufficiently vintage that it has two PCIe 3.0 x16 slots, one with 16 lanes and one with 4 lanes, and two classic PCI slots between them.)

Result: my WiFi is 93% faster, matching the WAN rate as seen on the Ethernet side of the router.  Good riddance, Realtek!