Sunday, June 20, 2021

What is @system-service?

Because it's subject to change, systemd does not officially publish what makes up "@system-service".  However, as of June 20th, 2021, the git repository defines the system calls allowed in src/shared/seccomp-util.c.  In addition to some more calls, @system-service (SYSCALL_FILTER_SET_SYSTEM_SERVICE) currently includes the following groups:

@aio, @basic-io, @chown, @default, @file-system, @io-event, @ipc, @keyring, @memlock, @network-io, @process, @resources, @setuid, @signal, @sync, and @timer.

It may be more interesting to consider what it doesn't include:

@clock, @cpu-emulation, @debug, @module, @mount, @obsolete, @pkey, @privileged, @raw-io, @reboot, and @swap.

This is not a list of what it "excludes," because system calls may appear on multiple lists; "chown" is permitted by @chown, and the non-inclusion of @privileged doesn't revoke that permission.

Some further observations:

  • @aio includes io_uring.
  • @raw-io is port-based I/O, and configuration of PCI MMIO.
  • @basic-io is read/write and related calls; @file-system is a much larger group that covers access, stat, chdir, open, and so on.
  • @io-event is evented I/O, that is, select/poll/epoll.

Essentially, @system-service is meant to permit a fairly wide range of operations.  This makes it easier to start using (less likely to break things), at the cost of potentially leaving the gates open more than necessary.

As for my own systems, I may start revoking @privileged and @memlock from daemons that run as a non-root user from the start.  Other than that, this exercise didn't turn out to be very informative.

Update on 2021-06-25: The command systemd-analyze syscall-filter will show all of the built-in sets; systemd-analyze syscall-filter @privileged will show the filters for a specific set (in this example, the "@privileged" list of calls.)

Tuesday, May 4, 2021

aws-vault on macOS Catalina

I recently installed aws-vault on my old iMac, for malware hardening.  If any finds me, I don’t want it to be able to pick up ~/.aws/credentials and start mining cryptocurrency.

One problem: there were a lot of password prompts, and the “aws-vault” keychain did not appear in Keychain Access as promised.

But it’s okay; there’s a security command that works with it, if we know the secret internal name of the keychain: aws-vault.keychain-db.  (I found this by looking in my ~/Library directory for the keychains; most of them have several files, but aws-vault has only that one.)

As in:

  1. security show-keychain-info aws-vault.keychain-db
  2. security set-keychain-settings -l -u -t 900 aws-vault.keychain-db
  3. security lock-keychain aws-vault.keychain-db
  4. security set-keychain-password aws-vault.keychain-db (this will prompt for current and new passwords)

Regarding the set-keychain-settings subcommand.  Each option specifies a setting: -l to lock the keychain whenever the screen is locked, -u to lock the keychain after a time period, and -t {seconds} to set that time period. So, in the example, we are setting the keychain to lock again after 900 seconds.  There do not seem to be opposites of these options documented, so I would guess running the command without a flag will remove that option.  I haven’t tested that, though.

I added the lock-keychain subcommand to my logout file [~/.zlogout for zsh, but formerly ~/.bash_logout].  Whenever I exit an iTerm2 tab, everything is locked again.  (I’ve been clearing SSH keys from the agent in that file for a long time.)

I’ve chosen to “Always Allow” aws-vault to access the keychain, which means I need to put in the password only for the unlocking operation, not for every time the binary wants to access information within the keychain.

Thursday, April 8, 2021

I Don't Know What Grub2 Is Doing

Background: there are a couple of things I do that are far more convenient on Linux, so I decided to replace the USB stick I was using with an SSD.  I bought and installed the SSD, installed Ubuntu Studio 20.10, and let it install Grub to /dev/sda.  Big mistake.

Windows could not be booted.  I wiped the Linux SSD, and then nothing booted; Grub expected to find itself on that drive.  Booting the Linux installer from the USB stick would detect the ghost of Grub in the MBR, and jump into it, instead of starting the stick.  I had to pull out an old copy of SysRescueCd to put something else in Windows' MBR.

But at that point, Windows would still not boot, nor would it repair itself from the repair mode.  I ended up having to reinstall Windows through an unnecessarily confusing and pointless set of menus on the install media.  That's not what the story is about, though.

With a functional Windows on /dev/sda, I reinstalled Linux again, and told it to put the boot loader on /dev/sdb.  This works fine, with the little weird problem that Grub still can't boot Windows 10?

I haven't gone through and tried set root=(hd0) and chain-loading that, but you know, this is 2021? This should work out of the box.

Despite having a "Dual UEFI BIOS" on my motherboard, it boots Windows in legacy/BIOS mode, and not UEFI mode.  So it's not a case of Linux being booted with legacy/BIOS, then not realizing that it needed grub-efi.  I think.

At some level, I don't care.  If it doesn't work, it doesn't work.  The PC is now configured to boot Windows by default, or Linux when I ask for a boot menu and pick the second SSD.  And I'm really just grumpy that the installer/Grub can't figure out how to get Windows up and running properly.

Saturday, April 3, 2021

The libsodium rant

Back when the inclusion of libsodium in PHP 7.2 was announced, I was excited.  A modern crypto library, with a modern, easy-to-use API?  Unfortunately, calling the API "modern" meant something completely different to me than the libsodium binding actually delivers to its users.

Rather than a modern, mistake-proof API, it's a direct, low-level replacement for the unmaintained mcrypt library.  It has more modern primitives, obscure names, and very little documentation.  Although the extension was promoted to core years ago, the functions are all "Warning: this is undocumented" to this day.  (Extension documentation is here.)

As for the names, I actually attended a libsodium talk at a PHP conference.  The speaker covered which functions are asymmetric and which are symmetric, but I don't remember which is which without checking my notes.  The docs say "Encrypt a message" for sodium_crypto_box, sodium_crypto_box_seal, and sodium_crypto_secretbox.  They also say that about sodium_crypto_stream_xor, but that definitely sounds wrong.

I had been expecting an interface more like defuse/php-encryption (which I have used) or perhaps soatok/dhole-cryptography (which I have not.)  Even if all the low-level crypto bits were exposed, because interoperability with other systems is important, I still expected a higher level API.  I expected an "encrypt this text with that key" operation that took care of nonces, formatting details, and algorithm choices.

Wednesday, March 31, 2021

The sea change to vendoring and containers

Not so long ago, we would install code to the global runtime path, where it would be available machine-wide.  CPAN was perhaps the earliest to do this, but PEAR, rubygems, and probably others followed.  After all, CPAN was seen as a great strength of Perl.

Users of such a library would say something like use Image::EXIF; or include_once "Smarty/Smarty.class.php"; Notice that these are a mechanism to follow some include path to resolve relative filenames.

But as servers got larger faster than applications, co-installing those applications became attractive.  The problem was that they can have different dependency requirements, which complicates the management of a shared include path.

This is about where Bundler happened for Ruby, and its ideas were brought to other ecosystems: Perl got carton, and PHP got composer. These systems build a directory within the project to hold the dependencies, and basically ignore the global include path entirely.

At the cost of disk space, this bypasses the problems of sharing dependencies between projects.  It bypasses all the problems of working with multiple versions of dependencies that may be installed in different hosts’ include paths.  It also makes the concept of the language’s include path obsolete, at least in PHP’s case: the Composer autoloader is a more-capable “include path” written in code.  Finally, it allows piecemeal upgrades—a small service can make a risky upgrade and gain experience, before a larger or more important service on the machine needs to upgrade.

Piecemeal upgrades also enable independent teams to make more decisions on their own.  Since their dependencies are not shared with other users, changes to them do not have to be coordinated with other users.

Containers are the next step in the evolution of this process.  Pulling even more dependencies into a container brings all the advantages of local management of packages into a broader scope.  It allows piecemeal upgrading of entire PHP versions.  And it makes an application cost an order of magnitude more space, once again.

In our non-containerized environment, I can switch the PHP version on a host pretty easily, using co-installable versions and either update-alternatives (for the CLI) or a2disconf/a2enconf (for FPM), but this means the services do not really have a choice about their PHP version.  It has been made before the application's entry point is reached.

Friday, February 5, 2021

What matters? Experimenting with Ubuntu Desktop VM boot time

It seemed slow to me that the "minimal" installation of Ubuntu Desktop 20.10 in a VirtualBox VM takes 37 seconds to boot up, so I ran some experiments.

The host system is a Late 2012 iMac running macOS Catalina.  It has 24 GB of RAM (2x4 GB + 2x8 GB) installed, and the CPU is the Intel Core i5-3470 (Ivy Bridge, 4 cores, 4 threads, 3.2 GHz base and 3.6 GHz max turbo.)  In all cases, the guests are running with VirtualBox Ubuntu Linux 64-bit defaults, which is to say, one ACHI SATA drive that is dynamically sized.  The backing store is on an APFS-formatted Fusion Drive.

The basic "37 seconds" number was timed with a stopwatch, from the point where the Oracle VM screen is replaced with black, to the point where the wallpaper is displayed.  Auto-login is enabled for consistency in this test.  This number should be considered ±1 second, since I took few samples of each system.  I'm looking for huge improvements.

So what are the results?

  • Baseline: 1 core, ext4, linux-generic, elevator=mq-deadline.  37 seconds.
  • XFS: 1 core, xfs root, linux-generic, elevator=mq-deadline.  37 seconds.
  • linux-virtual: 1 core, xfs root, linux-virtual, elevator=mq-deadline. 37 seconds.
  • No-op scheduler: 1 core, xfs root, linux-virtual, elevator=noop. 37 seconds.
  • Quad core: 4 cores, xfs root, linux-virtual, elevator=noop. 27 seconds!
The GUI status icons showed mostly disk access, with some heavy CPU usage for the last part of the boot, but only adding CPU resources made an appreciable difference.  I also tried the linux-kvm kernel, but it doesn't display any graphics or even any text on the console, so that was a complete failure.

(I've tried optimizing boot times in the past with systemd-analyze critical-chain and systemd-analyze blame, but it mostly tells me what random thing was starved for resources that boot, instead of consistently pointing to one thing causing a lot of delay.  Also, it has a habit of saying "boot complete in seven seconds" or whatever as soon as it has decided to start GDM, leaving a lot of time unaccounted for.  So I didn't bother on these tests.)

Correction: it was Ubuntu 20.10, not 20.04 LTS.  The post has been updated accordingly.

Wednesday, February 3, 2021

Stability vs Churn Culture

I’m working on rewriting memcache-dynamo in Go.  Why?  What was wrong with Python?

The problem is that the development community has diverged from my goals.  I’m looking for a language that’s stable, since this isn’t the company’s primary language.  (memcache-dynamo is utility code.) I want to write the code and then forget about it, more or less.  Python has made that impossible.

A 1-year release cycle with “deprecated for 1 cycle before removal” as the policy means it’s possible for users on Ubuntu LTS to end up in a situation where their previous LTS, 2 versions behind, doesn’t provide an alternative for something that’s been removed in the next LTS / current Python.

But looking closer to home, it’s a trend that’s sweeping the industry as a whole, and fracturing communities in its wake.  Perl wants to do the same thing, if “Perl 7” ever lands.

Also, PHP 5.6 has been unsupported for two years, yet there are still code bases out there that support PHP 5, and people are (presumably) still running 5.6 in production.  With Enterprise Linux distributions, we will see this continue for years; RHEL 7 shipped PHP 5.4, with support through June 2024.

There’s a separate group of people that are moving ahead with the yearly updates.  PHPUnit for example; every year, a new major version comes out, dropping support for the now-unsupported PHP versions, and whatever PHPUnit functionality has been arbitrarily renamed in the meantime.  The people writing for 5.x are still using PHPUnit 4 or 5, which don’t support 8.0; it’s not until PHPUnit 8.5.12 that installation on PHP 8 is allowed, and that still doesn’t support PHP 7.0 or 7.1.

This is creating two ecosystems, and it’s putting pressure on the projects that value stability to stop doing that.  People will make a pull request, and write, “but actually, can you drop php 5 support? i had to work extra because of that.”

The instability of Linux libraries, from glibc on up, made building for Linux excessively complex, and AFAICT few people bother.  Go decided to skip glibc by default when building binaries, apparently because that was the easier path?

Now everyone thinks we should repeat that mistake in most of our other languages.

Friday, January 1, 2021


Growing up, my dad had an old daisy wheel printer hooked up to our computers.  (We had various Commodore hardware; we did not get a PC until 1996, and I'm not sure the Amiga 500 was really decommissioned until the early 2000s.)

The daisy wheel was like a typewriter on a circle.  There were cast metal heads with the letters, numbers, and punctuation on them.  The wheel was spun to position, and then what was essentially an electric hammer punched the letter against the ribbon and the paper.  Then the whole wheel/ribbon/hammer carriage moved to the next position, and the cycle repeated.

This was loud; like a typewriter, amplified by the casing and low-frequency vibrations carried through the furniture.  No printing could be done if anyone in the house had gone to bed, or was napping.

It was also huge. It could have printed on legal paper in landscape mode.

Because of the mechanical similarity to typewriters, the actual print output looked like it was really typewritten.  Teachers would compliment me for my effort on that, and I'd say, "Oh, my dad has a printer that does that."

Nowadays, people send a command language like PostScript, PCL, or even actual PDF to the printer, and it draws on the page.  Everything is graphics; text becomes curves.  But the Daisywriter simply took ASCII in from the parallel port, and printed it out.