Identity Requires Long Term Secrets

One cannot remove all long-term credentials.  The process of establishing a session is one of identifying a stable entity (such as a user account) and giving that entity temporary access to the resource servers.  In simplest form, this is providing the username (entity) and password (secret that authenticates the username), and receiving a session cookie (temporary credentials) for accessing the service.

Somewhere at the root of trust must be a long-term credential.  Otherwise, if all temporary credentials have expired, how is the user authenticated in order to generate a new one?  What would stop anyone else from going through the same process for the user?

An individual service can outsource user authentication: they can email a code, use SMS or a voice call, or integrate with a third party service like Okta or any OAuth provider.  In those cases, the long-term exists, but the actual location of the key store is externalized.  Then the service is at the mercy of the security of that key store.  Email is probably low risk for normal people, who have an account with multi-factor authentication at a large provider who’s going to notice ‘unusual’ logins, but my quirky personal email isn’t like that.

The other problem with outsourcing is that if the provider changes their mind about account requirements, users can get locked out of both their email and their service account at the same time.  (Ask someone how hard it is to maintain a secondary Google account.)

Everything else is a long-term credential stored by the service.  Passwords need their hash to be checked against.  Passkeys, authenticator app codes, and client certificates are also linked to a user account, so must be stored with it.  The service cannot accept any of these things for the wrong user.

Everything Needs Rate Limits

For reasons of “anything else would cost more,” my web server and email MTA are running on the same VPS. Which apparently means, if a known issue in some web-side software fills the disk, email quits flowing in.

Fortunately, I was at liberty to investigate immediately. The root cause was a bot making things up and filling a cache with negative responses. The actual bot was promptly banned. Not only via robots.txt (which used to return text "# 200 OK Have Fun" so that I wouldn’t get 404 errors for it) but via User-Agent in the Web server, since it had obviously already seen the permissive robots.txt contents.

(Also, the only email that failed to be received were spam messages.  “Lucky” me!)

It worried me, though.  What if another bot did this? Am I going to play Whac-A-Mole® with it? (Don’t get me wrong—that’s not a bad game, but this version doesn’t give me any tickets redeemable for prizes at the front counter.)

To give me more runway to respond to future problems, I added per-source rate limits. This should prevent the cache from filling in units of MB/s. Concurrently, there is also a custom disk-usage alarm that will activate if free space falls below a “normal” amount, giving me a chance to catch problems before the MTA starts refusing service again.

A global rate limit for new connections is still being considered. The current bot menace has been beaten for today, and future bots from a single IP will also find themselves running into limits, but someone running a crawler network could still cause plenty of trouble. The problem is that a global rate limit is probably something that would be HTTP-oblivious, responding with an RST packet instead of an application-layer 429 message.

I know the “everything needs rate limits” is common wisdom, at least in some circles, but I ran public sites for at least 14 years without them. Sadly for the nostalgia, it seems those days are gone.

We Finally Put Up a WAF

Someone sent an awful lot of requests at a system for long enough that management noticed the issue. Working with the responsible admin, I ended up proposing AWS WAF “to see what would happen.”

What happened: WAF blocked 10,000 requests per minute, and someone got the message.  This released the pressure on the DynamoDB table behind the system, allowing it to jump straight from max to min capacity (1/16th) after fifteen minutes.

It seems some automated vulnerability scanner had gotten into an infinite loop.  There were a lot of repeated URLs in the access logs, like it wasn’t clearing pages from its queue if they got an OK response but unexpected data.  The reason “everything” returns OK is because an unknown URL (outside of a specific static-content prefix) returns a page with the React app root, and lets JavaScript worry about rendering whatever should be there.

I went ahead and put the same WAF on my systems, promptly breaking them.  Meanwhile, our automated testing provider started reporting failures, with every request from the original system returning Forbidden.

The testing platform… is a bot.  I had to write them an exception.

Turning my attention back to my systems, I put together a second WAF so I could have different policies.  My system includes an API or two, so I needed to allow HTTP Libraries and non-browsers. I linked in the exception for the testing platform as well.  Things went much more smoothly after that.

I know that the WAF is fundamentally “enumerating badness,” but it is clearly better than zero filtering.  It is also much less effort and risk, which is why this sort of thing persists.

Separate Components Allow “Least Privilege”

At work, our AWS image-building process exists as a set of scripts that are run manually, in-order, if everything looks okay to the human at the terminal.

The problem is, the image auto-installs security updates once it launches; if enough accumulate, instances launched from the image start failing to become ready for traffic before the timeout is up.  The system is guaranteed to decay if I don’t periodically build a fresh image, manually.

One reason it isn’t automated (say, rebuilding itself weekly using AWS infrastructure) is that the process takes more permissions than normal business operations.  It must be able to run and terminate instances, tag things, and reconfigure what image is launched by Auto Scaling.  Those would be scary permissions to give to any instances, which generally only have write access to specific data storage locations.

However, if the various steps (build, test, update configuration) each existed as separate objects visible to AWS, we could give each component unique permissions.  The configuration-update step would be the only one to have access to the image ID in SSM Parameter Store.  Likewise, that step would be forbidden from creating or terminating EC2 instances.

Due to implementation details, for us and our system, we could split it into the following components:

1. Configuration and test script deployment (run by developers off-AWS; stores to S3)
2. Ubuntu base image lookup (read-only)
3. Image build sequence (create/wait/destroy EC2 instances, tag instance and image, read configuration scripts from S3, invoke the AWS-maintained SSM Automation to create the image from the instance)
4. Image test sequence (create/destroy EC2 instances, read the test script from S3)
5. Configuration update (describe the image, update a specific SSM Parameter Store item)
6. Garbage collection (read running configuration to determine “unused” status, deregister images, delete their snapshots)

We only need to update the configuration scripts if we make changes to them; otherwise, doing a rebuild simply pulls in the available security updates.  Nothing inside AWS gets special permission to write these files.

After that, we can run step 2 to find an appropriate base AMI on our cron host.  That requires no write access, so all the cron host really needs is permission to start and monitor the latter tasks.  Tasks 3-5, in particular, are simple enough (once inputs are determined in step 2) to be run via AWS SSM Automation. I imagine there will be a “coordinating” automation that runs those three tasks in order, and the granular tasks exist mainly for ease of debugging each one.  Finally, garbage collection is somewhat compute-heavy but requires no waiting, so AWS Lambda might be the best option for it.

The end result will be a major improvement: each step will actually have minimal privileges, and the union of all privileges is much less than the admin privileges that I am technically granting to the process.  It “works” in the sense that we trust my entire laptop, but what if we didn’t need to?

Side Note: Firefox’s Primary Password is Local

When signing into Firefox Sync to set up a new computer, the primary password is not applied.  I usually forget this, and it takes a couple of runs for me to remember to set it up.

That’s not enough for a post, so here are some additional things about it:

The primary password protects all passwords, but not other data.  If someone can access Firefox data, bookmarks and history are effectively stored in the clear.

The primary password is intended to prevent reading credentials… and the Sync password is one of those credentials.  That’s why a profile with both Sync and a primary password wants that password as soon as Firefox starts; it wants to check for new data.

The same limitation of protections applies to Thunderbird.  If someone has access to the profile, they can read all historic/cached email, but they will not be able to connect and download newly received email without the primary password.

The Primary Password never times out.  As such, it creates a “before/after first unlock” distinction.  After first unlock, the password is in RAM somewhere, and the Passwords UI asking for it again is merely re-authentication.  Firefox obviously has the password saved already, because it can fill form data.

Some time ago, the hash that turns the primary password into an actual encryption key has been strengthened somewhat.  I believe it is now a 10,000-iteration thing, and not just one SHA-1 invocation.  The problem with upgrading it further is that the crypto is always applied; ”no password” is effectively a blank password, and the encryption key still needs to be derived from it to access the storage.  Mozilla understandably doesn’t want to introduce a noticeable startup delay for people who did not set a password.

---

Very recently (2024-10-17), the separate Firefox Sync authentication was upgraded.  Users need to log into Firefox Sync with their password again in order to take advantage of the change.

Pulling at threads: File Capabilities

For unimportant reasons, on my Ubuntu 24.04 installation, I went looking for things that set file capabilities in /usr/bin and /usr/sbin.  There were three:

• ping: cap_net_raw=ep
• mtr-packet: cap_net_raw=ep
• kwin_wayland: cap_sys_resource=ep

The =ep notation means that only the listed capabilities are set to “effective” and “permitted”, but not “inheritable.”  Processes can and do receive the capability, but cannot pass it to child processes.

ping and mtr-packet are “as expected.”  They want to send unusual network packets, so they need that right.  (This is the sort of thing I would also expect to see on nmap, if it were installed.)

kwin_wayland was a bit more surprising to see.  Why does it want that?  Through reading capabilities(7) and running strings /usr/bin/kwin_xwayland, my best guess is that kwin needs to raise its RLIMIT_NOFILE (max number of open files.)

There’s a separate kwin_wayland_wrapper file.  A quick check showed that it was not a shell script (a common form of wrapper), but an actual executable.  Could it have had the capability, set the limits, and launched the main process?  For that matter, could this whole startup sequence have been structured through systemd, so that none of kwin’s components needed elevated capabilities?

The latter question is easily answered: no.  This clearly isn’t a system service, and if it were run from the user instance, that never had any elevated privileges.  (The goal, as I understand it, is that a systemd user-session bug doesn’t permit privilege escalation, and “not having those privileges” is the surest way to avoid malicious use of them.)

If kwin actually adjusts the limit dynamically, in response to the actual number of clients seen, then the former answer would also be “no.”  To exercise the capability at any time, kwin itself must retain it.

I haven’t read the code to confirm any of this.  Really, it seems like this situation is exactly what capabilities are for; to allow limited actions like raising resource limits, without giving away broad access to the entire system.  Even if I were to engineer a less-privileged alternative, it doesn’t seem like it will measurably improve the “security,” especially not for cap_sys_resource.  It was just a fun little thought experiment.

My firewall, as of 2024

On my old Ubuntu installation, I had set up firewall rules to keep me focused on things (and to keep software in line, like blocking plain DNS to require DoT to CloudFlare.)

Before doing a fresh installation, I saved copies of /etc/gufw and /etc/ufw, but they didn’t turn out to be terribly useful.  I don’t know what happened, but some of the rules lost address information.  The ruleset ended up allowing printing to the whole internet, for instance.

I didn’t have a need for profiles (I don’t take my desktop to other networks), so I ended up reconstructing it all as a script that uses ufw, and removing gufw from the system entirely (take that!)

That script looks in part like this:

#!/bin/sh
set -eufC
# -- out --
ufw default reject outgoing
ufw allow out 443/udp comment 'HTTP 3'
ufw allow out 80,443/tcp comment 'Old HTTP'
ufw allow out proto udp \
	to 224.0.0.251 port 5353 \
    comment 'mDNS to LAN for printing'
ufw allow out proto tcp \
    to 192.168.0.251 port 631,9100 \
    comment 'CUPS to Megabrick'
ufw allow out on virbr0 proto tcp \
    to any port 22 comment 'VM SSH'
# -- in --
ufw default deny incoming
ufw allow in 9000:9010/tcp \
    comment 'XDebug listener'

This subset captures all of the syntax I’m using: basic and advanced forms, and all of the shapes of multi-port rules.  One must use the ‘advanced’ form to specify address or interface restrictions.  However, ufw is extremely unhelpful about error messages, usually only giving out “wrong number of arguments.”  The typical recourse is either to look harder at the man page syntax, or to try to roll back conditions until it gets accepted.

For deleting those test rules, the best way is ufw status numbered followed by ufw delete N where N is the desired rule number.  (You can also do ufw reset and start over.)

Note that the ufw port range syntax is “low:high” with a colon, like iptables. For example, 9000:9010 is a range of 11 ports; 9000,9010 is a list of only those two ports.

(I gave the printer a static IP because Windows; thus, the printer’s static IP appears in the ruleset.)

This script, then, only has to be run once per fresh install; after that, ufw will remember these rules and apply them at boot.

Diving too deeply into DH Moduli and OpenSSH

tl;dr:

• Debian/Ubuntu use the /etc/ssh/moduli file as distributed by the OpenSSH project at the time of the distribution’s release
• This file is only used for diffie-hellman-group-* KexAlgorithms
• The default KEX algorithm on my setup is the post-quantum sntrup761x25519-sha512@openssh.com instead
• Therefore, you can generate your own moduli, but it is increasingly irrelevant
• Having more moduli listed means that sshd will do more processing during every connection attempt that uses the file

There is also a “fallback” behavior if the server can’t read the moduli file or find a match, which I don’t fully understand.

Viewing the Expiration Date of an SSH Certificate

A while ago, I set up my server with SSH certificates (following a guide like this one), and today, I began to wonder: “When do these things expire?”

Host certificate (option 1)

Among the output of ssh -v (my client is OpenSSH_9.3p1 Ubuntu-1ubuntu3) is this line about the host certificate:

debug1: Server host certificate: ssh-ed25519-cert-v01@openssh.com [...] valid from 2023-04-07T19:58:00 to 2024-04-05T19:59:44

That tells us the server host certificate expiration date, where it says “valid … to 2024-04-05.”  For our local host to continue trusting the server, without using ~/.ssh/known_hosts and the trust-on-first-use (TOFU) model, we must re-sign the server key and install the new signature before that date.

User certificate

I eventually learned that ssh-keygen -L -f id_25519-cert.pub will produce some lovely human-readable output, which includes a line:

Valid: from 2023-04-07T20:14:00 to 2023-05-12T20:15:56

Aha!  I seem to have signed the user for an additional month-ish beyond the host key’s signature.  I will be able to log into the server without my key listed in ~/.ssh/authorized_keys (on the server) until 2023-05-12.

This looks like a clever protection mechanism left by my past self.  As long as I log into my server at least once a month, I'll see an untrusted-host warning before my regular authentication system goes down.  (If that happened, I would probably have to use a recovery image and/or the VPS web console to restore service.)

Host certificate (option 2)

There’s an ssh-keyscan command, which offers a -c option to print certificates instead of keys.  It turns out that we can paste its output to get the certificate validity again.  (Lines shown with $ or > are input, after that prompt; the other lines, including #, are output.)

$ ssh-keyscan -c demo.example.org
# demo.example.org:22 SSH-2.0-OpenSSH_8.9p1
# demo.example.org:22 SSH-2.0-OpenSSH_8.9p1
ssh-ed25519-cert-v01@openssh.com AAAA[.....]mcwo=
# demo.example.org:22 SSH-2.0-OpenSSH_8.9p1

The ssh-ed25519-cert line is the one we need.  We can pass it to ssh-keygen with a filename of - to read standard input, then use the shell’s “heredoc” mechanism to provide the standard input:

$ ssh-keygen -L -f - <<EOF
> ssh-ed25519-cert-v01@openssh.com AAAA[.....]mcwo=
> EOF

Now we have the same information as before, but from the host certificate.  This includes the Valid: from 2023-04-07T19:58:00 to 2024-04-05T19:59:44 line again.

Tips for setting up certificates

Document what you did, and remember the passphrases for the CA keys! This is my second setup, and now I have scripts to do the commands with all of my previous choices.  They’re merely one-line shell scripts with the ssh-keygen command.  But they still effectively record everything like the server name list, identity, validity period, and so forth.

To sign keys for multiple users/servers, it may be convenient to add the CA key to an SSH agent.  Start a separate one to keep things extra clean, then specify the signing key slightly differently:

$ ssh-agent $SHELL
$ ssh-add user-ca
$ ssh-keygen -Us user-ca.pub ...
(repeat to sign other keys)
$ exit

Note the addition of -U (specifying the CA key is in an agent) and the use of the .pub suffix (the public half of the key) in the signing process.

Using sshuttle with ufw outbound filtering on Linux (Pop!_OS 22.04)

I am using sshuttle and UFW on my Linux system, and I recently set up outbound traffic filtering (instead of default-allow) in ufw.  Immediately, I noticed I couldn’t make connections via sshuttle anymore.

The solution was to add another rule to ufw:

allow out from anywhere to IP 127.0.0.1, TCP port 12300

Note that this is “all interfaces,” not tied to the loopback interface, lo.

Now… why does this work?  Why doesn’t this traffic already match one of the “accept all on loopback” rules?

To receive that sshuttle is responsible for, sshuttle listens at 127.0.0.1:12300 (by default) and creates some NAT rules to redirect traffic for its subnet to that IP and port.  That is, running sshuttle -r example.com 192.168.99.0/24 creates a NAT rule to catch traffic to any host within 192.168.99.0/24.  This is done in netfilter’s nat tables.

UFW has its rules in the filter tables, and the nat tables run first. Therefore, UFW sees a packet that has already been redirected, and this redirection changes the packet’s destination while its interface and source remain the same!

That’s the key to answering the second question: the “allow traffic on loopback” rules are written to allow traffic on interface lo, and these redirected packets have a different interface (Ethernet or Wi-Fi.) The public interfaces are not expected to have traffic for local addresses on them… but if they do, they don’t get to take a shortcut through the firewall.

With this understanding, we can also see what’s going wrong in the filtering rules.  Without a specific rule to allow port 12300 outbound, the packet reaches the default policy, and if that’s “reject” or “deny,” then the traffic is blocked.  sshuttle never receives it.

Now we can construct the proper match rule: we need to allow traffic to IP 127.0.0.1 on TCP port 12300, and use either “all interfaces” or our public (Ethernet/Wi-Fi) interface.  I left mine at “all interfaces,” in case I should ever plug in the Ethernet.

(I admit to a couple of dead-ends along the way.  One, allowing port 3306 out didn’t help.  Due to the NAT redirection, the firewall never sees a packet with port 3306 itself.  This also means that traffic being forwarded by sshuttle can’t be usefully firewalled on the client side.  The other problem was that I accidentally created the rule to allow UDP instead of TCP the first time.  Haha, oops.)

Argon2id Parameters

There are no specific recommendations in this post. You will need to choose parameters for your specific situation.  However, it is my hope that this post will add deeper understanding to the recommendations that others make.  With that said…

The primary parameter for Argon2 is memory. Increasing memory also increases processing time.

The time cost parameter is intended to make the running time longer when memory usage can’t be increased further.

The threads (or parallelism, or “lanes” when reading the RFC) parameter sub-divides the memory usage.  When the memory is specified as 64 MiB, that is the total amount used, whether threads are 1 or 32.  However, the synchronization overhead causes a sub-linear speedup, and this is more pronounced with smaller memory sizes.  SMT cores offer even less speed improvement than the same number of non-SMT cores, as expected.

I did some tests on my laptop, which has 4 P-cores and 8 E-cores (16 threads / 12 physical cores.) The 256 MiB tests could only push actual CPU usage to about 600% (compared to the 1260% we might expect); it took 1 GiB or more to reach 1000% CPU.  More threads than cores didn’t achieve anything.

Overall then, higher threads allow for using more memory, if enough CPU is available to support the thread count.  If memory and threads are both in limited supply, then time cost is the last resort for extending the operation time until it takes long enough.

Bonus discovery: in PHP, the argon2id memory is separate from the memory limit.  memory_get_peak_usage() reported the same number at the beginning and end of my test script, even for the 1+ GiB tests.

systemd User Service Sandboxing

systemd-analyze security --user SERVICE is handy, but inaccurate.  The analysis (on the version bundled with Ubuntu 22.04 “Jammy”) has a few items that are marked as not applicable to user services, but also recommends setting a few items that apparently cannot be set in a user service. Attempting to do so will fail to start the service, with a log that the capabilities could not be set (regardless of the specific option causing the error.)

This unfortunately restricts the following: CapabilityBoundingSet, PrivateDevices, ProtectClock, ProtectKernelLogs, and ProtectKernelModules.

Using ProtectHostname with RestrictNamespaces=true logs a warning every time the service starts up, because the UTS namespace can’t be configured. It’s only a warning, which I think means that ProtectHostname is redundant in this situation, so I took it out of the service.  It’s more important to have clean logs than a good systemd-analyze security score.

The list of capabilities I was able to set were: KeyringMode=private, LockPersonality=true, MemoryDenyWriteExecute=true, NoNewPrivileges=true, PrivateMounts=true, PrivateTmp=true, ProcSubset=pid, ProtectControlGroups=true, ProtectHome=read-only, ProtectKernelTunables=true, ProtectSystem=strict, RestrictNamespaces=true, RestrictRealtime=true, RestrictSUIDSGID=true, SystemCallArchitectures=native, SystemCallFilter=@system-service, UMask=077, and:

RestrictAddressFamilies=
RestrictAddressFamilies=AF_INET
RestrictAddressFamilies=AF_INET6

Note that this is tailored to the service I was configuring, an HTTP proxy written in Go.  For instance, I wanted to run the binary and load its configuration file from ~/.local, which prevented me from using a stronger setting for ProtectHome.  Likewise, I can’t really turn off networking, because it’s a network program.

How does macOS Keychain determine which app is using it?

Thinking about the differences between macOS and Gnome keyrings, I began to wonder… how does macOS know what the app making the access is?  It obviously can’t be by the filesystem path, or malware could steal keys by naming itself aws-vault and sneaking into the keychain.

According to this Stack Overflow answer, Apple is using the app’s bundle ID to determine this, authenticated by code signing.

That’s an infrastructure that doesn’t really exist on Linux.  Each distro signs its packages, but the on-disk artifacts are generally neither signed nor checked.  That turns it into a much larger problem for Gnome Keyring to tie secrets to individual apps, because first, we’d need to build a secure way to identify apps.

And then, culturally, Linux users would reject anything resembling a central authority.  Developers (including me) don’t usually bother even using GPG for decentralized code signing, and that’s not even an option with the fashionable curl|bash style installers.

Gnome Keyring Security

In the ongoing process of moving to a new work laptop, I have been working to protect secrets from hypothetical malware on the device.  I am using Pop!_OS 22.04 at the moment, examining the default environment: Gnome and gnome-keyring[-daemon].

Using Secrets

I had assumed, given the general focus on security in Linux, that it would be broadly similar to macOS.  To illustrate the user experience there, starting iTerm2 requires the login password to open the password manager for the first time.  The login password is also required to view the passwords through Keychain Access, regardless of whether iTerm2 is running.

I found exactly one Linux terminal with a password manager, Tilix, so of course I installed it. Within Tilix, no password is required to use the password manager.  Using Seahorse (the apparent equivalent to Keychain Access), the passwords Tilix has stored appear in the “login” keyring, unlocked by default, and no password is needed to reveal them.

In short, the keyring in Gnome is glorified plain text.  Access to the session bus is unconditional access to all secrets in all unlocked keyrings.

Who’s There?

Another major difference is that macOS seems to associate keyring entries with owners, such that each individual program gets its own settings in the OS about whether it can access particular secrets.  I can “always allow” aws-vault to access secrets in the aws-vault keyring, but I presume if awsthief tried to access them instead, I would get a new prompt.

Furthermore, if I uncheck “remember this password” on the Mac, it stays unchecked the next time the keyring is unlocked.  In Gnome, for the past 8 years, it re-checks itself every time, waiting for a moment of inattention to make the security of the alternative keyring (awsvault, of course) entirely moot.  It may be locked, but you can have D-Bus fetch the key from under the doormat.

Locking Up

I’m not certain yet whether the Gnome keyring can auto-lock collections, either.  My previous post on macOS’ security command includes how to lock the keyring after a timeout, or when the system is locked.  These capabilities are missing from Seahorse, but I haven’t fully analyzed the D-Bus interface.  (Still, I shouldn’t need to do so.)

Copying Microsoft Good Enough?

A cursory Web search suggests that the way Gnome handles the keyring is exactly like Windows.  Not only is Gnome chasing taillights, but it has chased the easiest ones to catch.

Overall, the quality of Gnome (GNU …) keyring lives up to the heuristic for bad cryptography.

Locking one single keyring in gnome-keyring from the terminal

Updated 2022-07-22: The original code didn't work in practice.  The default for --type changes if --print-reply is given, so it stopped working when I removed the latter after testing.  The command below has been updated to provide explicit values for all options: --type to make it work, and --session to future-proof it.  The original text follows.

I'm moving to a new work laptop, where I wanted to lock the aws-vault keyring when I close a shell/terminal window.  Previously, I did this on macOS, using security lock-keychain aws-vault.keychain-db in my ~/.zlogout file.  (I switched to zsh, then discovered zsh-syntax-highlighting, which is immensely useful.)

So anyway, what's the equivalent CLI command in Pop!_OS 22.04 (derived from Ubuntu 22.04)?

dbus-send --session --type=method_call --dest=org.gnome.keyring \
  /org/freedesktop/secrets \
  org.freedesktop.Secret.Service.Lock \
  array:objpath:/org/freedesktop/secrets/collection/awsvault

(Backslash-continuations and line breaks added for readability.  For debugging, adding the --print-reply option may be of use.  Also, the interface can be explored with D-Feet; first, point it to the session bus, then search for secrets, to get the needle out of the haystacks.)

Now, if they ever fix that 8-year-old issue in gnome-keyring, we'll have fully-usable secondary keyrings.  I mean, what is the point to having another keyring, if the option is "No, but ask me again every time?"  Updated 2025-02-09: Gnome 46.4, released 2024-08-11, includes gnome-keyring 46.2 with a fix for this issue.

A New SSH Key Philosophy

I rolled over my SSH keys this morning; but more than that, I generated two additional keys, so now I have four distinct domains:

1. Local systems
2. EC2 systems
3. AWS CodeCommit (read/write)
4. AWS CodeCommit (read only) – this one already existed

Previously, I would load “the SSH key” into ssh-agent, and do everything with one key.  CodeCommit would always be accessed in read/write mode; even systems that only needed read-only access for testing would access the read/write key through the forwarded agent.

Because there was only one skeleton key, which I needed frequently, it would be available to any executing code, any time I was working. All hosts it would work on were clearly indicated in .ssh/config and .git/config files. Any code on a guest VM would also be able to access it, through the forwarded agent.  The guest’s configuration also included the hosts, because I once developed from the guest environment.  Back then, the host was Windows, making it more comfortable to access EC2 and git from the guest.

The first two keys I generated represent different frequencies of access. I access the local systems much more frequently than EC2, to debug code under development running on them.  Splitting those keys means that the EC2 key will be unlocked and available much less frequently.

As for CodeCommit, I decided to separate “shell” keys from “git” keys. Our threat model generally gives up on security if an attacker has shell access, so if I can keep local malware away from remote shell access, it adds a layer to the defense.  In addition, this key is also accessed more frequently than the EC2 key.

Finally, I quit forwarding the SSH agent by default to test systems.  They already have the (sufficient) read-only key installed when their base image is created, so all I had to do was “stop changing the configuration” in my personal setup scripts for those guest VMs.  This reduces unnecessary trust (and coupling) between host and guest.

Of note, I am already guarding the AWS keys with aws-vault, which I wrote about previously. I was sufficiently pleased with aws-vault that I’m using it on my home Linux installation, as well.

Overall, it took me an hour or two to work out the plan and reconfigure everything, but I’m happy with the improved security posture.

Defense in Depth: Read-Only Document Root

One security rule I live by as a sysadmin, which admittedly causes me a lot of trouble with PHP apps, is that the running web server user is never able to write to files within the document root.

Any user-file upload/download mechanism goes somewhere else, so that read access can do a permission check, and be delivered through readfile() or a redirect to a CDN, not passed through a complex mechanism that may decide to execute the file's contents.

Configuration files are similarly locked down.  A project that configures itself with PHP code is a project that offers arbitrary code execution to anyone who can change that configuration file.

I don't let the web user install new code to /bin; we shouldn't let the web user install new code to /srv/www. It's the server, not a system administrator.

Not coincidentally, our document root is never the root of a git repository.  It's typically placed in a public folder, allowing us to have a special writable folder like cache both within the project/deployment, and outside of the document root.

Adventures with kASLR

Things I discovered recently about kASLR:

• Linux added kaslr a while ago, in 3.14.
• It was enabled via kaslr on the kernel command line.
• kaslr wasn’t compatible with hibernation, originally.  This appears to have changed in 4.8.
• It was enabled by default in 4.12, with nokaslr to disable it.
• kaslr support is compiled in by CONFIG_RANDOMIZE_BASE=y.
• Ubuntu 18.04 LTS (bionic) has that setting enabled in the generic and aws kernels, which are based on 4.15.
• /etc/default/grub may be useless on Ubuntu for setting command line flags.

Under the hood, there are “up to” 512 base addresses for the kernel (depending on the specific machine’s memory map), and kaslr decompresses the image into a random available one during bootup.  This puts the base kernel “somewhere” in a 1 GB-sized area, aligned at 2 MB.

The kernel command line is available in /proc/cmdline.  However, it didn’t have my kaslr customization, which sent me on a quest.  I discovered that Debian/Ubuntu configure a bunch of scripts to produce the final grub configuration file, using /etc/default/grub.d/*.cfg.  These are processed after /etc/default/grub.  There turned out to be a “cfg” file that unconditionally replaced GRUB_CMDLINE_LINUX_DEFAULT, which is where I had put our kaslr flag.  This affected both of our instance types: VirtualBox appeared to have one unintentionally left over from the installer, while AWS had one placed there as part of the cloud image build.

But given that kaslr appears to be default, instead of setting up a local configuration file, I ended up removing the code that was trying in vain to set kaslr.

Clean up when closing a terminal

I've taken to clearing my ssh-agent identities and sudo timestamps when I close my shell, by putting in my ~/.bash_logout file:

if [[ -n $SSH_AUTH_SOCK && -z $SSH_CONNECTION && $SHLVL = 1 ]] ; then
 ssh-add -D
 sudo -K
fi

One caveat: the above only works using the shell's exit (or logout or Ctrl + D), not with iTerm2's close button.  However, that can be fixed by using an exit trap in .bash_profile instead, like so:

clearKeys() {
 ssh-add -D
 sudo -K
}
if [[ -n $SSH_AUTH_SOCK && -z $SSH_CONNECTION && $SHLVL = 1 ]] ; then
 trap clearKeys EXIT
fi

To be clear, the latter version requires no changes/additions to .bash_logout.

Rationale: I usually work on a desktop, and keep a copy of my work in sync on a laptop using unison.  Making the SSH connection from the laptop adds the key to the session's ssh-agent, but I don't want that to persist after sync is finished.  I don't want keys to stay active while I'm not planning on using them soon.

Rebuilding Dependencies Frequently

Reddit asked:

how often do you rebuild your OS level dependencies?

In practice, around twice a month, due to accumulation of security updates. Sometimes bit-rot plays a role as well, but mostly, it’s the endless stream of updates. We want images to quickly launch into a secure state.

Waiting until the instance comes up to apply updates has two drawbacks: one, each new instance duplicates the work. Two, if there’s an update that requires a reboot, we’ve found through repeated, painful experience that there’s no way to reliably reboot our instances programatically.

IIRC though, that question was in the context of Docker containers, while our process is to precompile our app and its dependencies into a monolithic AMI that we deploy to instances in AWS. I like to think it’s basically the same, but I haven’t really used Docker much.