Systemd Allows Unknown Units in Before/After

Most of the time, my development virtual machine guest would boot and run perfectly fine.  Sometimes, though, the FastCGI service backing one of the websites would not be up and running.  It had a ConditionPathExists, and if the code to run the service wasn’t mounted, it wouldn’t start.

The intention was to allow colleagues to import a copy of this guest, then set up the mount to share the project from the host as they saw fit.  On their first boot, with no sharing, ConditionPathExists would prevent the FastCGI service from attempting to start, and therefore, systemd would not report that the system was degraded.  Another point about this system is that the sharing mechanism is unspecified: colleagues are free to use NFS (as I do), Plan9 file sharing, or the hypervisor’s shared-files mechanism.  The host paths are also unspecified, so there is no way I can set up the guest to expect specific sharing in advance.

In practice, sometimes NFS wasn’t ready in my guest before systemd was checking conditions for the FastCGI service.  The obvious answer was to add After=remote-fs.target to the FastCGI service.  I quickly added a drop-in to add this directive to my own post-configuration scripts.

However, that’s a local solution to a global problem.  My colleagues can’t benefit from that, and I should minimize the burden on them periodically setting up new guest images.  The fewer things they must remember, the better.

It turns out the answer was even simpler: I could skip the drop-in and add the After= line to the main service file. I added both remote-fs.target and the hypervisor’s guest services to the line, which means:

1. In production, there are no remote filesystems to mount, nor guest services; there is no latency introduced.
2. When using NFS or similar, systemd waits for the remote filesystem before starting the FastCGI service.
3. With the hypervisor’s file sharing, the guest services mount the shared files before starting the FastCGI service.

My guest doesn’t actually have the guest services installed, but the FastCGI service starts up as intended.  Looking at systemctl list-units --all output, the guest services are (now) listed as not-found and inactive, which is pretty much what I would expect from a dangling reference.  systemd knows about it because I listed it in After, but since it’s not required by anything, the missing definition for it doesn’t cause any problems.

Using Cloudflare DNS over TLS system-wide on Ubuntu

My current Linux distributions (Ubuntu 23.04 and the Ubuntu-derived Pop!_OS 22.04) use NetworkManager for managing connections, and systemd-resolved for resolving DNS queries.  I’ve set up Cloudflare’s public DNS service with DoT (DNS over TLS) support twice… and I don’t really have a solid conclusion.  Is one “better?”  🤷🏻

Contents

• Per-connection mode with NetworkManager only
• Globally with systemd-resolved / NetworkManager
• Useful background info

systemd User Service Sandboxing

systemd-analyze security --user SERVICE is handy, but inaccurate.  The analysis (on the version bundled with Ubuntu 22.04 “Jammy”) has a few items that are marked as not applicable to user services, but also recommends setting a few items that apparently cannot be set in a user service. Attempting to do so will fail to start the service, with a log that the capabilities could not be set (regardless of the specific option causing the error.)

This unfortunately restricts the following: CapabilityBoundingSet, PrivateDevices, ProtectClock, ProtectKernelLogs, and ProtectKernelModules.

Using ProtectHostname with RestrictNamespaces=true logs a warning every time the service starts up, because the UTS namespace can’t be configured. It’s only a warning, which I think means that ProtectHostname is redundant in this situation, so I took it out of the service.  It’s more important to have clean logs than a good systemd-analyze security score.

The list of capabilities I was able to set were: KeyringMode=private, LockPersonality=true, MemoryDenyWriteExecute=true, NoNewPrivileges=true, PrivateMounts=true, PrivateTmp=true, ProcSubset=pid, ProtectControlGroups=true, ProtectHome=read-only, ProtectKernelTunables=true, ProtectSystem=strict, RestrictNamespaces=true, RestrictRealtime=true, RestrictSUIDSGID=true, SystemCallArchitectures=native, SystemCallFilter=@system-service, UMask=077, and:

RestrictAddressFamilies=
RestrictAddressFamilies=AF_INET
RestrictAddressFamilies=AF_INET6

Note that this is tailored to the service I was configuring, an HTTP proxy written in Go.  For instance, I wanted to run the binary and load its configuration file from ~/.local, which prevented me from using a stronger setting for ProtectHome.  Likewise, I can’t really turn off networking, because it’s a network program.

Our brief use of systemd's TemporaryFileSystem feature

Late last year, we started having problems where services did not appear to get proper data after a reload.  But I have a hypothesis.

We have a monolithic server build, where several services end up running under one Apache instance.  I thought it would be nice to make each one think it was the only service running.  Instead of enumerating badness, I switched to using TemporaryFileSystem= to eliminate all directories, then allowing access the one needed for the service.

The problem is, I don’t think this gets reset/updated during a reload. Deployment renames the working directory, and renames an all-new directory in at the original name.  It’s possible that even though the reload happens, the processes are not changing working directory, thus still looking in the original one.

Things that have never failed were failing; in particular, Template Toolkit was still using stale template files.  I did some digging and found that it does have a cache by default, but it only holds entries for up to a second. Furthermore, because I was very strict about not having unexpected global state, we create a new Template object per request, and shouldn't be sharing the cache.

Anyway, systemd did not actually document what it does with TemporaryFileSystem on reload, so I can’t be sure.  I ended up abandoning TemporaryFileSystem= entirely.  I only have so much time to mess around, and there’s only so much “risk” that would be mitigated by improved isolation. It’s almost guaranteed that the individual applications have defects, and don’t need a multi-app exploit chain.

(Not in the sense that I know of any specific defects; in the sense that all software has defects, and some proportion of those can be leveraged into a security breach.)

What is @system-service?

Because it's subject to change, systemd does not officially publish what makes up "@system-service".  However, as of June 20th, 2021, the git repository defines the system calls allowed in src/shared/seccomp-util.c.  In addition to some more calls, @system-service (SYSCALL_FILTER_SET_SYSTEM_SERVICE) currently includes the following groups:

@aio, @basic-io, @chown, @default, @file-system, @io-event, @ipc, @keyring, @memlock, @network-io, @process, @resources, @setuid, @signal, @sync, and @timer.

It may be more interesting to consider what it doesn't include:

@clock, @cpu-emulation, @debug, @module, @mount, @obsolete, @pkey, @privileged, @raw-io, @reboot, and @swap.

This is not a list of what it "excludes," because system calls may appear on multiple lists; "chown" is permitted by @chown, and the non-inclusion of @privileged doesn't revoke that permission.

Some further observations:

• @aio includes io_uring.
• @raw-io is port-based I/O, and configuration of PCI MMIO.
• @basic-io is read/write and related calls; @file-system is a much larger group that covers access, stat, chdir, open, and so on.
• @io-event is evented I/O, that is, select/poll/epoll.

Essentially, @system-service is meant to permit a fairly wide range of operations.  This makes it easier to start using (less likely to break things), at the cost of potentially leaving the gates open more than necessary.

As for my own systems, I may start revoking @privileged and @memlock from daemons that run as a non-root user from the start.  Other than that, this exercise didn't turn out to be very informative.

Update on 2021-06-25: The command systemd-analyze syscall-filter will show all of the built-in sets; systemd-analyze syscall-filter @privileged will show the filters for a specific set (in this example, the "@privileged" list of calls.)

Containers over systemd

“Systemd will solve all your problems,” they said.

Having used a number of systemd’s security features to configure a service, I am beginning to suspect everyone uses containers because container runtimes are trying to be secure already.

It's possible to improve the security of a service with systemd, of course.  I’ve worked hard at it.  But in the end, over half the *.service file is consumed with “trying to build my own container out of systemd directives.”  ProtectHome, ProtectSystem, ProtectKernelTunables, Protect This, Protect That, blah blah blah.  The process starts from insecure by default, and then asks me to layer on every individual protection.  This is exactly the sort of thing Linux zealots used to yell at Microsoft about.  ¯\_(ツ)_/¯

But I digress.  I ended up with an excessively long systemd service configuration file, and to apply that to any other service, there’s no option besides copying and pasting those directives.  With every release of systemd, I have to comb the man pages again to see what else is available now, and carefully apply that to every service file.  It’s not easy to tell whether the security posture is up-to-date when the policy is so verbose.

[Updated 2022-03-19: systemd-analyze security foo.service is your friend. This is the best way to get a list of everything systemd thinks about security, and whether it is applied to the unit. It's a little less bad than I thought, but it is still fundamentally the Default Permit and Enumerating Badness approaches.]

Whereas a container has an isolated filesystem (its image) already, so whole classes of configuration (ProtectHome, ProtectSystem, TemporaryFileSystem) become irrelevant.  On top of that, container runtimes start with a more limited set of privileges by default, instead of handing out CAP_SYS_ADMIN and leaving it up to the administrator to carefully disable it.  Escaping from the container runtime is considered a vulnerability; escaping from a poorly-secured systemd service is considered user error.

This is all orthogonal to “containers are interop”, but I think both forces are feeding the containerization craze.  I’m left with the feeling again that systemd should have been the “obvious correct choice,” except they decided usability didn’t matter.

systemd: the house built on sand

Once upon a time, supervisord got the service management done, but I never got the logs stored anywhere sensible.  Eventually, I got tired of being tapped to debug anything that had an obvious error, but where the message was only logged by supervisord.

Thus began a quest for running things through the distribution’s init system, which has given me some experience with upstart and a lot of experience with systemd.  Like most software that reaches success, systemd has not been carefully designed and implemented.  It has only accumulated, organically.

This is nowhere more obvious than in the configuration system.  I can’t just read documentation online, write a .service file, and expect it to work; I have to use the online search to find which man page they hid the relevant directives in, and spin up a VM to read it.  Once I find the directives that apply, it’s obvious that we have an INI system crying out to be a more imperative, stateful, and/or macro-driven language.

Those are related; because the configuration is underpowered, new capabilities require new tweaks.  Consider the number of “boolean, or special string value” options like ProtectHome and ProtectSystem: these were clearly designed as booleans and then extended later.

Because the website doesn’t keep a changelog—everything changes so fast, systemd just has a major version and every release is breaking—it’s not easy to build a cross-platform service definition file that takes advantage of the features systemd offers.  You know, the things that make it unique from other init systems.  The things that were supposed to be selling points.

It’s because everything changes at the whim of the developers.  Stable? Backwards-compatible, at least?  In a fundamental system component?

Big nope from the systemd team.  There are at least a few directives that were superseded, and so it’s impossible to make a portable service description for a service that is otherwise portable. And the lack of past-proofing tells us about future-proofing.  What you write today may simply not run tomorrow.

systemd was supposed to be the obvious, easy choice: in theory, it embraced Linux and cgroups so that administrators could use cgroups to provide isolation without a separate containerization layer.  But in practice, the separate layer is looking ever more like a better choice.