Ubuntu Studio 26.04: We Can’t Have Nice Things

Problem 1: KDE Plasma on Wayland retains the sticky-keys bug, where the latch state (press a modifier twice) is tracked, but doesn’t actually work. Here’s to at least six more months of X11!

Problem 2: the Orchis theme remains deeply, completely broken on KDE.

For science, I created a new user on 26.04 and then used the Wayland session for an as-close-to-default experience as I could manage.  The default experience is the “Ubuntu Studio Dark” theme, with a wallpaper like distressed concrete, a theme choice that died ten years ago.  Now that I had to look at it, we’re both distressed.

I didn’t get to see it very long.  The first thing the system told me was that I needed to reboot to optimize my audio settings, presumably for this new user.  Why Ubuntu Studio can’t get this to work out of the box is unfathomable, but here we are.

The second login, I was able to select the global Orchis-dark theme and apply the desktop layout, which immediately blew everything up.  The panel and wallpaper disappeared, leaving a vast blackness and nothing to interact with.  Except for the Overview hot-corner, anyway.  Fortunately, as someone who has run a FOSS desktop for more than a dozen years, I knew a shortcut to get logged out and back in.

At that point, I had a top bar, with a single visible object: a color picker’s separator and history (black.)  It rather looked like those minimalist corporate logos that are all the rage.  Also, various other parts of the bar had no highlight but would react to clicks, because the problem was actually the lack of icons.

Further research would uncover that Orchis-dark requires in its theme file:

• Telu-circle icon theme
• Vimix-dark cursor theme

I was not able to find these in the Ubuntu repositories.

For further science, or to “keep driving with a blown tire” maybe, I tracked down and installed those themes from the Orchis’ author’s GitHub.  This made the icons show up immediately, and did not fix the cursors.  Why not?  Because the theme installs Vimix and Vimix-white themes!  There is no Vimix-dark.

I began to wonder if I’m the problem, but I checked apt search orchis and there’s no newer/different thing in the repository.  For all the fanfare about the “beautiful new” theme in 24.10 (!), not only did it not work there, but it still does not work in 26.04 LTS.  Embarrassing.

(On top of it all, SDDM still doesn’t remember the last session in a per-user way, so I had to switch it back to X11 to log in as my regular user.)

Kernel Config Must Get Easier

Not to be “old person on main,” but 25 years ago, it was possible to build my own Linux kernel and run it on my hardware.  As usual in development, it would take a few times to get one that would boot nicely, but the process itself didn’t feel difficult.  It was pretty easy to select IDE, VIA chipset support, and the like, and receive a fairly streamlined kernel.

Fast forward to the 2020s, and the opposite has happened.  I’m reliant on distro kernels because trying to configure and build a modern Linux kernel is a huge, interconnected maze of options, with no visibility into why it doesn’t boot.  Options in section 3 may not be visible if required options in section 5 haven’t been selected yet, which make it a nonlinear meta-adventure.  I have repeatedly failed to build a kernel for a VirtualBox guest, something where the hardware should be well known in advance.

However, if I had been able to build my own kernels successfully, Copy Fail 2 and Dirty Frag wouldn’t have been issues.  IPSec is pretty much dead to me, and I had never heard of RxRPC until now, so these things would have naturally been configured out of my own kernel build.

There’s a distinct conflict here between convenience and security: if we autoload the kitchen sink, then nobody needs to build enable-feature ipsec and a GUI for it.  Nobody needs to recompile their kernel for it.  On the other hand, if there’s a bug anywhere in the autoload surface, it can be reachable by any user.

And honestly, the whole reason I fell out of building kernels—other than the journeys into FreeBSD and Windows—is that it was terrible to try to keep up with the influx of kernel updates.  Maybe userspace had a compatibility guarantee, but config did not.  I was soon unwilling to spend so much energy (mine and my PC’s) on rebuilding a kernel so frequently.  Most of the commits would be irrelevant, but trying to filter and judge them all was an even worse proposition.

I know my distro doesn’t want to maintain dozens of packages like linux-modules-ipsec for people (a logical equivalent of enable-feature ipsec) but it’s also difficult to rebuild a distro kernel according to my own config.  Leaving aside the problems with choosing/generating a config, the distro kernel has a lot of features, which makes it take a while to compile.  The cycle to even try a new kernel build is much, much longer than it was when I was building my own, and I think it crosses the threshold between “tolerable” and “too much to bear.”

Two Thoughts on Ubuntu Signing Keys

Here’s something I don’t get: why is there a trusted “2012 CD signing key” on my Ubuntu 24.04 machines, when there is also a “2018” signing key?  Shouldn’t this be a transition that could have completed within five years?  Shouldn’t we be able to tie the 2012 key to a specific repository set, instead of all packages?  The latter includes PPAs and I really wish neither of those CD signing keys were valid for that purpose.

The cryptographic domains should be separated:

1. One CD signing key, tied to the CD/DVD packages
2. One online release signing key, tied to the Ubuntu main/security sources
3. One key per PPA, tied to that PPA

Deprecating globally-trusted keys for PPAs is a good step, but the globally-trusted release keys (especially ones that are over a decade old) should be cleaned out immediately as well.

Semi-related pro tip: extrepo

Many packages are supported in extrepo, which handles the keys for you.  There is no need for arcane gpg format-conversion commands, no worrying about whether it goes into /usr (incorrect under Unix philosophy, but widely recommended) or /etc, no manually editing sources files, and especially no cursed curl | bash invocation.

$ sudo apt install extrepo

And then you can do stuff like:

$ extrepo search github
$ sudo extrepo enable github-cli
$ sudo apt install --update gh

This is especially useful for upstreams that distribute an official deb package, outside of PPAs.  I aim to get the code from as close to the source as possible, where the distro itself doesn’t suit my needs.

Adventures with my old iPod Touch

The iPod Touch (4th Gen) in the car could no longer be detected, so we pulled it out of the console to find it in DFU mode.  Yikes.

I had extremely little hope, but I took it and its USB-A to 30-pin dock cable (the only extant cable of this type in my collection) inside, plugged it into a USB-C to USB-A adapter, and plugged it into the Mac.  It… er… worked. Sure, it appeared to open in Finder and not iTunes (rip), but it actually worked (on the second try; the advice for “unknown error 9” is to try again.  What are we doing as a profession.)  I was able to restore iOS 6.1.6 to it, although I did not have the option to keep my data.

But, I never moved my music onto the Mac.  I figured, with a hardware device that is fifteen years old, and back in factory state, surely, Linux should be able to sync to it?

The first problem was even getting it connected, because Amarok threw an error from ifuse.  Copying the command out of the error message and running it in a terminal worked totally fine.  (I didn’t think of this at the time, but… Amarok logs in the systemd journal.  Maybe its permissions have been stripped down too far.)

Once that was up and running, I restarted Amarok a couple of times, before I found out where it had hidden the iPod.  It’s under “Local Collection.”

I then waited a long time for things to sync.  I waited so long that I wandered off and forgot to set “Don’t Sleep”, so the computer suspended.  The iPod made its ancient, discordant glissando when the computer woke up, and then Amarok—and any process trying to stat() the ifuse mount point—froze.  ifuse sat there burning 100% CPU for a couple of minutes, and then I restarted.

(Apparently the sleep interval was fifteen minutes, the longest time that doesn’t make KDE System Settings complain about ’using more energy.'  Well… I paid for it, one way or the other.)

I got it going again.  Amarok carefully loaded gigabytes of tracks onto the iPod Touch, then started complaining about checksum errors for the database.  That’s the part that makes them useful, instead of having the iPod show “No content” and a button for the iTunes Store.  That ended up being the final boss that I couldn’t beat.  The tracks are still there, apparently, showing up as “Other” data on the Mac.

Yeah.

I plugged the backup drive into the Mac, imported everything, and exported it to the iPod Touch.  The double copy was orders of magnitude faster than Amarok’s unidirectional efforts.  I should never have been so lazy.

• Free Software: 0
• Proprietary OS: 2

I don’t know how we got here.

Observations of Liquid Glass

The effect is very nice.  I don’t use it on my personal phone.  Reduce Motion inhibits the glassiness, and having motion on a phone that I actually use a lot is annoying.  I loathe waiting for an animation to finish before I am Permitted to interact with anything.

Clear icons aren’t… bad.  Unless they’re dark.  Then they’re not clear anymore, which makes me wonder if people in Cupertino actually understand their own language.  I don’t use clear icons on my personal phone, because I like the auto-dark mode.  Ironically, between color filters to greyscale (to make the phone less attention-hogging) and Reduce Motion, normal icons are not that far off their clear counterparts.

On the impersonal phone, clear icons and Liquid Glass look pretty nice together, at least on the iOS 26 wallpapers.  I have them set to “always light” due to their lackluster dark variants, but they look fine that way when the rest of the phone is in dark mode.

Opening an app brings up a solid-background splash screen, and app.  There’s so much 💖new💖 look, until it is touched, and then the glass disappears.  A subconscious wrongness suffuses it.  Apple took it halfway.  They Microsoft’ed it.  I can use it, but it keeps me wondering… why?

Update: [2025-11-15] I experimentally turned off Reduce Motion on my personal phone, and sometimes, the motion gets really confused.  This makes everything jittery with the phone held as still as humanly possible, like it’s had three cups of coffee.  It looks like the effect that I saw sometimes with Reduce Motion on, where the line above the dock would flicker, but with everything at once.

I also found out that icon customization is somehow tied to the screen layout or focus.  I made icons clear in Sleep mode, and they remained non-clear outside of Sleep.  I like this result, but the UI doesn’t make it clear where the setting applies.

Sometimes, the Apple apps in my sleep focus screen turn into empty-app indicators.  No reminders, notes, or music for the night!  But third-party apps are fine.  I cannot fathom how the first-party systems could exhibit such poor results.

I can use it, but there are so many rough edges, I keep wondering… why?

Ubuntu 24.10 First Impressions

I hit the button to upgrade Ubuntu Studio 24.04 to 24.10.  First impressions:

1. The upgrade process was seriously broken.  Not sure if my fault.
2. Sticky Keys is still not correct on Wayland.
3. Orchis has major problems on Ubuntu Studio.

No More Realtek WiFi

The current Debian kernel (based on 6.1.66, after the ext4 corruption mess) seems to be locking up with the Realtek USB wireless drivers I use.  Anything that wants the IP address (like agetty or ip addr) hangs, as does shutdown.  It all works fine on the "old" kernel, which is the last version prior to the ext4 issue.

Meanwhile in Ubuntu 23.10, the in-kernel RTW drivers were flaky and bouncing the connection, so I had returned to morrownr’s driver there, as well.  But now that I don’t trust any version of this driver?  Forget this company.  In the future, I will be using any other option:

1. A Fenvi PCIe WiFi card with an Intel chip on board, or the like
2. Using an extra router as a wireless client/media bridge, with its Ethernet connected to the PC
3. If USB were truly necessary, as opposed to simply “convenient,” a Mediatek adapter

Remember that speed testing and studying dmesg output led me to the conclusion that this chipset comes up in USB 2.0 mode, and even the Windows drivers just use it that way.  While morrownr’s driver offers the ability to switch it to USB 3.0 mode under Linux, this prevents it from being connected properly.  I never researched hard enough to find out if there is a way to make that work, short of warm rebooting again so that it is already in USB 3.0 mode.

It’s clearly deficient by design, and adding injury to insult, the drivers aren’t even stable.  Awful experience, one star ★☆☆☆☆, would not recommend. Intel or Mediatek are much better choices.

Addendum, 2024-01-13: I purchased an AX200-based Fenvi card, the FV-AXE3000Pro.  It seemed not to work at all.  In Windows it would fail to start with error code 10, and in Linux it would fail to load RT ucode with error -110.  And then, Linux would report hangs for thermald, and systemd would wait forever for it to shut down.  When the timer ran out at 1m30s, it would just kick up to 3m.

Embarrassingly enough, all problems were solved by plugging it into the correct PCIe slot.  Apparently, despite being physically compatible, graphics card slots (which already had the punch-outs on my case, um, punched out) are for graphics cards only.  (My desktop is sufficiently vintage that it has two PCIe 3.0 x16 slots, one with 16 lanes and one with 4 lanes, and two classic PCI slots between them.)

Result: my WiFi is 93% faster, matching the WAN rate as seen on the Ethernet side of the router.  Good riddance, Realtek!

Ubuntu Paper Cuts: 22.04

I installed Ubuntu 22.04 “Jammy Jellyfish” as a guest in VirtualBox.  Some observations on the installer itself:

1. The default BIOS boot yields an 800x600 screen that does not fit the installer’s window. The progress bar is hidden from view.
2. EFI boot yields a better 1024x768 screen, but the graphics updating is very bad. (This could be VirtualBox’s fault—it looks a lot like the VirtualBox UI when it forgets to draw stuff until mouseover. Also, the resulting desktop in EFI works fine.)
3. The so-called minimal installation produces a full installation, then removes packages with the package manager, bloating the size of the VDI file backing the VM and placing unnecessary wear on the physical medium.
4. Even normal installation installs “everything,” then removes the other language packs.

Booting into the system reveals the following:

1. [edited to add] The loading animation disappears fairly early during EFI boot, leaving only two static logos (Ubuntu's, and one from the firmware) showing.
2. Asking the installer to download updates didn’t avoid having “Updates Available” immediately appear on the desktop.
3. Installing the updates didn’t avoid having Software show a dot (for 3 more Snap-related packages) on its Updates tab immediately thereafter.
4. Following that, Software had yet another dot on the Updates tab, but no updates.
5. Software doesn’t have an icon for the Health and Fitness category; it looks more like the “Windows default EXE” icon than anything.
6. The main page of Software had, as its top/largest app, an ad for Slack, in which some of the black text of the description overlapped the dark purple background of the Slack icon, rendering it partially unreadable.

All this, and I hadn’t even done anything yet.

snapd

The currently published snap for ripgrep is version 12.1.0; the snap was released 2020-05-17.  The current version, 13.0.0, was released on 2021-06-12. There’s no contact for the publisher of the snap.

The currently published snap for alacritty is version 0.8.0, with the snap released 2021-06-17.  Meanwhile, the current version is 0.10.1; the snap packagers (the “Snapcrafters”) haven’t packaged 0.9.0 released 2021-08-03 or any later version.  The contact information listed for this snap is a Github repository that doesn’t allow issues to be reported; indeed, the contact is the pull request page.  [Updated 2022-07-29: fixed a typo in the snap release date.  Verified that neither snap has been updated since publishing.]

The snap store itself only recognizes legal reasons to “report” a snap, due to violations of either copyright/trademarks, or the terms of the Snap store. There’s no way to signal to the publishers that an update is desired.

It’s clear through their actions where Canonical stands, here: once again, they are interested in taking the work of the FOSS community and siloing it in a place where Canonical can profit, and nobody else can.

The real issue isn’t about snap itself; it’s Canonical’s core strategy. Consider their other initiatives, like upstart, Unity, and Mir.  They wanted to become owners of critical infrastructure, walled off from the whims of the community by a licensing agreement that gave Canonical—especially and only Canonical—the right to profit off the works of others which they did not pay for.

With snaps, Canonical attempts to further erode the community, by letting anyone take over any project’s name, publish under it, and (regardless of intentions) leave it to rot.  Then, they don’t even want to clean up the results, unless they are legally bound to do so.  This helps with the number of “available snap packages,” to be sure, but sinks the utility of the whole system.  Users can’t place their trust in it.

It appears that I may need to shift distributions again, to something that is neither Ubuntu nor downstream from it.

Are containers light weight?

I read a thing claiming that containers are “light weight.”  But that’s only compared to a hardware virtual machine!  Containers seem light only through the historical accident of their path to popularity.  They are nearly at the end-point of heavyweight distribution methods.

Once upon a time, we programmers were able to handle a bit of version skew. We’d use libraries like GTK+ which maintained backward compatibility—at the ABI level, even—so that code compiled against 2.4.x would run against 2.4.x or later 2.x releases, without changes.  We’d install something like Smarty to the global PHP include path, and use a single copy of it from all our projects, for space efficiency.  Nothing was vendored!

(We could semi-vendor things in scripting languages by playing with the include path.  Install a major upgrade to, say, “lib-v4”, then in the v4-aware application, prepend the “lib-v4” directory to the include path at runtime. When all the applications were converted, remove the old version from the global path, move the v4 code there, and remove the include-path code from the apps again.  It’s a lot like gradually updating a database column.  It wasn’t a great approach for C code, though.)

Portability across operating systems, even in “POSIX” land, was a mess, but we all learned how to do it.  Virtually all open-source code dealt with it, so we had plenty of examples, and we largely respected the user’s choice of platform to run on.  Even if it were Solaris…

This also produced a pressure for minimal dependencies; the less we required of a user, then the more likely they were to run our code.  I still think that Java largely failed on Linux because every user had to fetch the JRE from Sun’s atrocious website themselves.  (Blackdown and later OpenJDK would change this, long after the ship had sailed.  The Apache Foundation’s Java-based projects are a notable exception from the general attitude, but they are also not desktop software.)

Today’s environment is the complete antithesis.  We pack entire OS distributions, possibly a language interpreter, all of our libraries, and our application code into a gigabyte-plus wrapping (partially shared, but still a minimum).  Then, we call it “lightweight” because it doesn’t have a guest kernel in there.

The old times weren’t perfect; it was an incredibly painful experience to make Linux binaries that worked across distributions, because of variance in the filesystem layout and the need to rely on old libraries to cover the older systems people might run the binary on.  And sometimes, there was no choice but to make multiple builds, because distributions might only package one of incompatible library versions. But largely, to support an app of a few megs, we shipped a few megs, not a few hundred, and we certainly didn’t call “a near-complete disk image” lightweight.

The Pace of Change

I’m not the first, nor the only, person to complain about the pace of technical change.  But what are the actual problems?

We risk losing perspective.  We will forget that the fad of today is just another fad; blockchains and containers are destined to be the next XML, relatively soon in their life, then carried forward for thirty years because changing infrastructure is too risky for the business.

We risk losing the wisdom of the past, assuming even our own younger selves were but naked savages, coding in Perl or PHP. We will not know what made Perl, Perl; we will not bring any of the good ideas forward.

Truly, we risk losing experts. It took me a good 10 or 15 years to really appreciate the sheer amount of knowledge that makes an expert, an expert; if we burn our world down every five years, then we will never come to know anything deeply.  We will have no experts.

Where I used to worry about becoming “a dinosaur,” it now seems that dinosaurs going extinct are the larger problem.

But what is the actual problem?

Pride, perhaps?  Are we too snobby to learn about what came before, to understand our place in history, and to meet the present where it’s at?  Do we think there is nothing to learn about a system in reading its code, in making improvements to it, that we must replace it outright?

Is it ignorance?  Or is it the deep, white-guy need to fall into the pit himself, before he believes it to be there?  Do we really believe that it was jQuery that created the spaghetti, and not ourselves?  Will abandoning one library for another genuinely improve our own capabilities… or is it a convenient deflection?

I am inclined to shout, “just deal with it!” at people.  They probably want to shout it back to me.

I Don't Know What Grub2 Is Doing

Background: there are a couple of things I do that are far more convenient on Linux, so I decided to replace the USB stick I was using with an SSD.  I bought and installed the SSD, installed Ubuntu Studio 20.10, and let it install Grub to /dev/sda.  Big mistake.

Windows could not be booted.  I wiped the Linux SSD, and then nothing booted; Grub expected to find itself on that drive.  Booting the Linux installer from the USB stick would detect the ghost of Grub in the MBR, and jump into it, instead of starting the stick.  I had to pull out an old copy of SysRescueCd to put something else in Windows' MBR.

But at that point, Windows would still not boot, nor would it repair itself from the repair mode.  I ended up having to reinstall Windows through an unnecessarily confusing and pointless set of menus on the install media.  That's not what the story is about, though.

With a functional Windows on /dev/sda, I reinstalled Linux again, and told it to put the boot loader on /dev/sdb.  This works fine, with the little weird problem that Grub still can't boot Windows 10?

I haven't gone through and tried set root=(hd0) and chain-loading that, but you know, this is 2021? This should work out of the box.

Despite having a "Dual UEFI BIOS" on my motherboard, it boots Windows in legacy/BIOS mode, and not UEFI mode.  So it's not a case of Linux being booted with legacy/BIOS, then not realizing that it needed grub-efi.  I think.

At some level, I don't care.  If it doesn't work, it doesn't work.  The PC is now configured to boot Windows by default, or Linux when I ask for a boot menu and pick the second SSD.  And I'm really just grumpy that the installer/Grub can't figure out how to get Windows up and running properly.

The libsodium rant

Back when the inclusion of libsodium in PHP 7.2 was announced, I was excited.  A modern crypto library, with a modern, easy-to-use API?  Unfortunately, calling the API "modern" meant something completely different to me than the libsodium binding actually delivers to its users.

Rather than a modern, mistake-proof API, it's a direct, low-level replacement for the unmaintained mcrypt library.  It has more modern primitives, obscure names, and very little documentation.  Although the extension was promoted to core years ago, the functions are all "Warning: this is undocumented" to this day.  (Extension documentation is here.)

As for the names, I actually attended a libsodium talk at a PHP conference.  The speaker covered which functions are asymmetric and which are symmetric, but I don't remember which is which without checking my notes.  The docs say "Encrypt a message" for sodium_crypto_box, sodium_crypto_box_seal, and sodium_crypto_secretbox.  They also say that about sodium_crypto_stream_xor, but that definitely sounds wrong.

I had been expecting an interface more like defuse/php-encryption (which I have used) or perhaps soatok/dhole-cryptography (which I have not.)  Even if all the low-level crypto bits were exposed, because interoperability with other systems is important, I still expected a higher level API.  I expected an "encrypt this text with that key" operation that took care of nonces, formatting details, and algorithm choices.

Stability vs Churn Culture

I’m working on rewriting memcache-dynamo in Go.  Why?  What was wrong with Python?

The problem is that the development community has diverged from my goals.  I’m looking for a language that’s stable, since this isn’t the company’s primary language.  (memcache-dynamo is utility code.) I want to write the code and then forget about it, more or less.  Python has made that impossible.

A 1-year release cycle with “deprecated for 1 cycle before removal” as the policy means it’s possible for users on Ubuntu LTS to end up in a situation where their previous LTS, 2 versions behind, doesn’t provide an alternative for something that’s been removed in the next LTS / current Python.

But looking closer to home, it’s a trend that’s sweeping the industry as a whole, and fracturing communities in its wake.  Perl wants to do the same thing, if “Perl 7” ever lands.

Also, PHP 5.6 has been unsupported for two years, yet there are still code bases out there that support PHP 5, and people are (presumably) still running 5.6 in production.  With Enterprise Linux distributions, we will see this continue for years; RHEL 7 shipped PHP 5.4, with support through June 2024.

There’s a separate group of people that are moving ahead with the yearly updates.  PHPUnit for example; every year, a new major version comes out, dropping support for the now-unsupported PHP versions, and whatever PHPUnit functionality has been arbitrarily renamed in the meantime.  The people writing for 5.x are still using PHPUnit 4 or 5, which don’t support 8.0; it’s not until PHPUnit 8.5.12 that installation on PHP 8 is allowed, and that still doesn’t support PHP 7.0 or 7.1.

This is creating two ecosystems, and it’s putting pressure on the projects that value stability to stop doing that.  People will make a pull request, and write, “but actually, can you drop php 5 support? i had to work extra because of that.”

The instability of Linux libraries, from glibc on up, made building for Linux excessively complex, and AFAICT few people bother.  Go decided to skip glibc by default when building binaries, apparently because that was the easier path?

Now everyone thinks we should repeat that mistake in most of our other languages.

Containers over systemd

“Systemd will solve all your problems,” they said.

Having used a number of systemd’s security features to configure a service, I am beginning to suspect everyone uses containers because container runtimes are trying to be secure already.

It's possible to improve the security of a service with systemd, of course.  I’ve worked hard at it.  But in the end, over half the *.service file is consumed with “trying to build my own container out of systemd directives.”  ProtectHome, ProtectSystem, ProtectKernelTunables, Protect This, Protect That, blah blah blah.  The process starts from insecure by default, and then asks me to layer on every individual protection.  This is exactly the sort of thing Linux zealots used to yell at Microsoft about.  ¯\_(ツ)_/¯

But I digress.  I ended up with an excessively long systemd service configuration file, and to apply that to any other service, there’s no option besides copying and pasting those directives.  With every release of systemd, I have to comb the man pages again to see what else is available now, and carefully apply that to every service file.  It’s not easy to tell whether the security posture is up-to-date when the policy is so verbose.

[Updated 2022-03-19: systemd-analyze security foo.service is your friend. This is the best way to get a list of everything systemd thinks about security, and whether it is applied to the unit. It's a little less bad than I thought, but it is still fundamentally the Default Permit and Enumerating Badness approaches.]

Whereas a container has an isolated filesystem (its image) already, so whole classes of configuration (ProtectHome, ProtectSystem, TemporaryFileSystem) become irrelevant.  On top of that, container runtimes start with a more limited set of privileges by default, instead of handing out CAP_SYS_ADMIN and leaving it up to the administrator to carefully disable it.  Escaping from the container runtime is considered a vulnerability; escaping from a poorly-secured systemd service is considered user error.

This is all orthogonal to “containers are interop”, but I think both forces are feeding the containerization craze.  I’m left with the feeling again that systemd should have been the “obvious correct choice,” except they decided usability didn’t matter.

Python has descended into chaos

At my employer, Python is not our primary language; the main system was built in Perl and is now a Perl/PHP hybrid. The surrounding systems have been built/replaced with PHP, but the core code is resistant to change. Management doesn't want to mess with the part where the actual money flows in.

For seamless session handling—local memcached on dev machines, but shared storage in production—we have memcache-dynamo. It started life in Perl. I rewrote it in Python, as part of the quest to replace all Perl with something more popular in modern times. (I figured async/await syntax would be easier for future developers than React-PHP.) It's been working fine on Ubuntu 18.04 for some time, but then I wanted to update to Ubuntu 20.04.

It quickly turned irritating. The Python 3.6 dependencies didn't work on 3.8. It turns out that the updated dependencies for 3.8 don't work on 3.6, either. I guess this explains why pipenv is so insistent that "requires 3.6" means "exactly 3.6".

Python 3 isn't just a version with some backward-incompatible changes; it's fundamentally a new culture with new values. And honestly, that probably means less Python in my future. I don't want to promote another language into the main systems, and Python no longer seems to be suitable for fire-and-forget background services.

(In the end, I wrote a new layer on our installer that checks the Python version, then copies e.g. Pipfile-3.8 and Pipfile.lock-3.8 into their standard location before starting pipenv. It's a horrible mess and I hate it so much. Please, don't start a Python 3 transition every year!)

On "Perl 7"

Previous history:

1. SawyerX: Announcing Perl 7.
2. Grinnz: Perl 7: A Risk-Benefit Analysis.
3. RJBS: On Perl 7 and the Perl Steering Comittee.

My comments at this point:

Hey #perl core. Is there anywhere I can upload a cpanfile, so you can know what's important in production? Like Debian's popularity-contest, but specialized. (We install most everything via Ubuntu repositories, so you don't get downloads/test feedback.)

The perl7 thing is probably accelerating our plans to phase Perl out entirely. I'm not sure what we'll do if Ubuntu 22.04 ships a "Perl 7" that can't run Perl 5 code.

To be honest, ~20% of hits to our client-facing site are served by CGI.pm because of some failures on my part (should have used CGI::Emulate::PSGI?) and on management (50% time savings is great, but let's work on features... forever.)

Anyway, it's going to be real hard to sell "work on Perl7 support" to a management team who have been ramping up the priority on "rewrite the application."

(Lightly edited from my twitter thread.)

Now, I realize I'm nobody, and nobody has to listen to me, but I think the popularity-ranking thing is kind of important, and maybe even a "State of Perl" survey to find out what's even broken. All I know from my vantage point is that Perl looks fairly dead already, despite how much I loved it. The Perl 7 crisis has so far been more anxiety-inducing than inspiring.

Updated 2022-03-19: Perl 7 is basically dead? Business as usual? I still can't help illuminate the DarkPAN for them, and management is inching closer to sunsetting our own Perl code. RIP.

pipenv's Surprise

Warning: Python 3.6 was not found on your system…
You can specify specific versions of Python with:
  $ pipenv --python path/to/python

I am left with no clear path to making this project run with the system Python 3 across just two versions of Ubuntu LTS.  It doesn't work on Focal (Python 3.8) as-is, and if I update the Pipfile.lock, it won't run on Bionic (Python 3.6).  It doesn't have shebangs for python3.6, as it expects to run on Python 3.6 or up.  This is how SemVer works!

Maybe the answer is to build my own tools in order to run this in a way that suits me. Which is: I really want a build phase to create a tarball, which can be extracted and run in-place later. All the complexity of vendoring should be finished by the time deployment (from tarball) occurs, for reliability and reproducibility.

I do not want to write this project a third time, probably in a language I hate, just because virtualenv is both wholly inadequate, and central to Python tooling.

(Something like the way the awscli-bundle is packaged is an interesting start, but it still has to grind through a lot of code to "install" itself. It's not unzip-and-go. Also, I've got no idea how they build that in the first place.)

systemd: the house built on sand

Once upon a time, supervisord got the service management done, but I never got the logs stored anywhere sensible.  Eventually, I got tired of being tapped to debug anything that had an obvious error, but where the message was only logged by supervisord.

Thus began a quest for running things through the distribution’s init system, which has given me some experience with upstart and a lot of experience with systemd.  Like most software that reaches success, systemd has not been carefully designed and implemented.  It has only accumulated, organically.

This is nowhere more obvious than in the configuration system.  I can’t just read documentation online, write a .service file, and expect it to work; I have to use the online search to find which man page they hid the relevant directives in, and spin up a VM to read it.  Once I find the directives that apply, it’s obvious that we have an INI system crying out to be a more imperative, stateful, and/or macro-driven language.

Those are related; because the configuration is underpowered, new capabilities require new tweaks.  Consider the number of “boolean, or special string value” options like ProtectHome and ProtectSystem: these were clearly designed as booleans and then extended later.

Because the website doesn’t keep a changelog—everything changes so fast, systemd just has a major version and every release is breaking—it’s not easy to build a cross-platform service definition file that takes advantage of the features systemd offers.  You know, the things that make it unique from other init systems.  The things that were supposed to be selling points.

It’s because everything changes at the whim of the developers.  Stable? Backwards-compatible, at least?  In a fundamental system component?

Big nope from the systemd team.  There are at least a few directives that were superseded, and so it’s impossible to make a portable service description for a service that is otherwise portable. And the lack of past-proofing tells us about future-proofing.  What you write today may simply not run tomorrow.

systemd was supposed to be the obvious, easy choice: in theory, it embraced Linux and cgroups so that administrators could use cgroups to provide isolation without a separate containerization layer.  But in practice, the separate layer is looking ever more like a better choice.

I still don't understand Python packaging

Since we last talked about this subject, I've tried to use pipenv with PIPENV_VENV_IN_PROJECT=1 for the project in question. Everything was going pretty well, and then… updates!

I'm using a Homebrew-installed version of Python to test, because it's easier and faster on localhost, and the available Python version was upgraded from 3.6 to 3.7. As usual, I ran brew cleanup -s so the Python 3.6 installation is gone.

It turns out that my python_version = "3.6" line doesn't do what I want—pipenv will be unable to do anything because that binary no longer exists—and I haven't been able to figure out a way to ask Pipenv to use "3.6 or above" to both:

1. Express the "minimum version: 3.6" requirement
2. Allow 3.7 to satisfy the requirement

pipenv seems pretty happy to use the system Python when given a version requirement of ">=3.6" but it's also acting like that's a warning only. pipenv check doesn't like this solution, and it's not clear that a system Python 3.5 would cause it to fail as desired.

In PHP, this is just not that hard. We put "php": "^7.1.3" in our composer.json file, and it will install on PHP >=7.1.3,<8.*. It will fail on <7.1.3 or on 8.x or on an 8.0 development version. It's all understood and supported by the tool.

So anyway: right now, we have a deployment process which is more or less "read the internet; build in place for production; swap symlink to make updated code available to the web server."

The end goal is to move the production deployment process to "extract a tarball; swap symlink." To do this, we need to create the tarball with "read the internet; build in place; roll into tarball" prior. And AFAICT, building a virtualenv into a tarball will package everything successfully, similar to Composer, but it will also bake in all the absolute paths to the build process's Python installation.

Pipfile and Pipfile.lock look like what I want (deterministic dependency selection in the build stage, and with the environment variable, in-project vendoring of those dependencies) but it seems like it's fundamentally built on virtualenv, which seems to be a thing that I don't want. I obviously want dependencies like aiobotocore vendored, but I don't necessarily want "the python binary" and everything at that layer. I especially don't want any symlinks pointing outside the build root to be put into the tarball.

Overall, I think pipenv is trying to solve my problem? But it has dragged in virtualenv to do it, which "vendors too much stuff," and it has never been clear to me what benefit I'm supposed to get from having a bloated virtualenv. And also, virtualenv doesn't fully support relocatable environments, which is another problem to overcome. In the past, it has been fairly benign, but now it has turned adversarial.

(We have the technical capability to make the build server and the production server match, file path for file path, exactly. But my devops-senses tell me that tightly coupling these things is a poor management decision, which seems to imply poor design on the part of virtualenv at least. And that contaminates everything built on top of it.)