My ssh/sshd Configurations

Let’s look at my SSH configurations!

File Layout

Starting with Ubuntu 22.04 LTS and Debian 12, the OpenSSH version in the distribution is new enough that the Include directive is supported, and works properly with Match blocks in included files.  Therefore, most of the global stuff ends up in /etc/ssh/sshd_config.d/01-security.conf and further modifications are made at higher numbers.

Core Security

To minimize surface area, I turn off features I don’t use, if possible:

GSSAPIAuthentication no
HostbasedAuthentication no
PasswordAuthentication no
PermitEmptyPasswords no

AllowTcpForwarding no
X11Forwarding no
Compression no

PermitUserRC no
# Debian and derivatives
DebianBanner no

Some of these are defaults, unless the distribution changes them, which means “explicit is better than implicit” is strongly advised.

Next, I use a group to permit access, allowing me to explicitly add the members to the group without needing to edit the ssh config when things change.  Don’t forget to groupadd ssh-users (once) and gpasswd -a USER ssh-users (for each user.) Then, permit only that group:

AllowGroups ssh-users
# extra paranoia
PermitRootLogin no

Note that all of the above may be overridden in Match blocks, where required. TCP forwarding may also be more finely controlled through PermitListen and PermitOpen directives.

Note also that my systems are essentially single-user.  The group doesn't permit any sharing (and doesn't participate in quotas or anything) that would otherwise be forbidden.

Performance

Machines I use for ssh and sshd are all amd64, so for personal usage, I bump the AES algorithms to the front of the list:

Ciphers ^aes256-gcm@openssh.com,aes256-ctr

SFTP

The biggest trouble is the SFTP subsystem.  I comment that out in the main config, then set it in my own:

# /etc/ssh/sshd_config:
#Subsystem sftp ...

# /etc/ssh/sshd_config/02-sftp.conf:
Subsystem sftp internal-sftp
Match group sftp-only
    # ForceCommand, ChrootDirectory, etc.

I forget the details of what goes in that Match block.  It’s work stuff, set up a while ago now.

Ongoing Hardening

I occasionally run ssh-audit and check out what else shows up.  Note that you may need to run it with the --skip-rate-test option these days, particularly if you have set up fail2ban (guess how I know.)

There are also other hardening guides on the internet; I have definitely updated my moduli to only include 3072-bit and up options.  Incidentally, if you wonder how that works:

awk '$5 >= 3071' ...

The default action for awk is print, so that command prints lines that fulfill the condition.  The fifth field is the length of the modulus, so that’s what we compare to.  The actual bit count is 3071 instead of 3072, because the first digit must be 1 to make a 3072-bit number, so there are only 3071 bits that aren’t predetermined.

Client Config Sample

Host site-admin
  # [HostName, Port, User undisclosed]
  IdentityFile ~/.ssh/id_admin
  IdentitiesOnly yes

Host 192.168.*
  # Allow talking to Dropbear 2022.83+ on this subnet
  KexAlgorithms +curve25519-sha256,curve25519-sha256@libssh.org
  MACs +hmac-sha2-256

Host *
  Ciphers aes256-gcm@openssh.com,aes256-ctr
  KexAlgorithms sntrup761x25519-sha512@openssh.com
  MACs hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com
  GSSAPIAuthentication no

It’s mostly post-quantum, or assigning a very specific private key to the administrative user on my Web server.

Everything Fails, FCGI::ProcManager::Dynamic Edition

I have been reading a lot of rachelbythebay, which has led me to thinking about the reliability of my own company’s architecture.

It’s top of my mind right now, because an inscrutable race condition caused a half-hour outage of our primary site.  It was a new, never-before-seen condition that slipped right past all our existing defenses.  Using Site as a placeholder for our local namespace, it looked like this:

use Try::Tiny qw(try catch);
try {
  require Site::Response;
  require Site::Utils;
  ...;
} catch {
  # EX_PRELOAD => exit status 1
  exit_manager(EX_PRELOAD, $_);
};
...;
$res = Site::Response->new();

Well, this time—this one time—on both running web servers… it started throwing an error that Method "new" wasn't found in package Site::Response (did you forget to load Site::Response?).  Huh?  Of course I loaded it; I would’ve exited if that had failed.

In response, I added a lot more try/catch, exit_manager() has been improved, and there is a separate site-monitoring service that will issue systemctl restart on the site, if it starts rapidly cycling through workers.

Using tarlz with GNU tar

I have an old trick that looks something like:

$ ssh HOST tar cf - DIR | lzip -9c >dir.tar.lz

The goal here is to pull a tar from the server, compressing it locally, to trade bandwidth and client CPU for reduced server CPU usage.  I keep this handy for when I don’t want to disturb a small AWS instance too much.

Since then, I learned about tarlz, which can compress an existing tar archive with lzip.  That seemed like what I wanted, but naïve usage would result in errors:

$ ssh HOST tar cf - DIR | tarlz -z -o dir.tar.lz
tarlz: (stdin): Corrupt or invalid tar header.

It turned out that tarlz only works on archives in POSIX format, and (modern?) GNU tar produces them in GNU format by default.  Pass it the --posix option to make it all work together:

$ ssh HOST tar cf - --posix DIR | \
    tarlz -z -o dir.tar.lz

(Line broken on my blog for readability.)

Bonus tip: it turns out that GNU tar will auto-detect the compression format on read operations these days.  Running tar xf foo.tar.lz will transparently decompress the archive with lzip.