Blogroll Updates

I updated my blogroll project on GitHub.  The real problem is that my feed reader at work is broken, so I got NetNewsWire at home and imported the feeds from my blogroll project.  Consequently:

mjg59’s URL has been updated to follow the blog’s move.

nullprogram was removed for becoming an AI-only feed.  I was reading it to expand my technical horizons, and it is no longer technical.  Something to ponder: if an LLM is so good that “no code” is practical, why would anyone even need a CMake debugger, instead of asking their LLM to go fix it?

The AWS CodeDeploy Agent was removed from the releases because we no longer depend on it.  The agent was always last to support an Ubuntu LTS, often delaying many months when everyone else was ready within weeks.  The whole time, there would be either no communication, or “we’ll get it this month” replies that would be proven extremely false.  This dissatisfaction limited CodeBuild to only one system, which is now mothballed, and if it ever comes back, it will be on ECS/Fargate.

jquery-color was also removed because we no longer depend on it… or if we do, it should not be hard to make a pure CSS replacement.  jquery itself lives on in “old-foo”, which I believe is showing just a couple of admin-only pages under the otherwise-React-based Foo2 intranet site.

The actual problem with the feed reader at work is that Akregator from Flathub has decided that it can’t access its feeds in ~/.var/app/..., which seems to imply some sort of packaging error.  But it’s been going on for enough weeks that I finally took action.  I had chosen Flatpak because the in-repo version regularly deleted its feeds, so going back isn’t an option.  Ubuntu effectively never updates software once the release has been shipped.

Server Management: We Should Be Able to Recreate Pets, Too

For anyone not familiar with the analogy, there are two types of server or private instance: “cattle” exist en masse and have names like atl-prod3-24.  If one has problems, the first solution is replacement.  “Pets,” on the other hand, are unique instances that matter a lot, often with whimsical names.  Problems with pets are approached by working to cure what ails them.

Once upon a time, we had a single pet server which was our bastion host, cron job runner, and SFTP file-exchange point with third parties.  We split those roles apart; the cron jobs get access to the SFTP files (for input or output) by sharing an EFS mount.  The SFTP machine everyone uses can be firewalled off from the rest of our codebase and AWS resources.  (Not that I expected 0days, but there was a notable close call a couple of years after this split was made.  Further detailed in Half a Second.)

After a couple of Debian upgrades, the SFTP host had some network issues.  Investigation found a post putting the blame on ‘the image builder’ and telling the person with the question (my same question, unfortunately) that because of that, they were on their own.

It seems likely, then, that we’ll want to rebuild this pet from scratch at some point.  I fixed everything this time, but who knows what might happen on the next upgrade?  It seems like it would be far more stable if we could re-customize a clean base image, instead of doing a series of in-place upgrades.  This is especially true if each Debian upgrade isn’t prepared for anything that is specific to the Debian AWS cloud images.

A long aside about the networking problems I faced follows.

ScreenZen as an RSI Aid

It hurts my hands to hold a phone for a long time, but apps, games, and the internet in general can be pretty compelling stuff.  I tried using the built-in Screen Time features, but they are fairly bare-bones.  One of the issues was that I would open an app “for five minutes,” and then get the message out of nowhere that there were only five minutes left on the app for the entire day. It meant I had just lost an hour or two, and maybe wouldn’t be able to use the app later.

(For the sake of argument, assume that I never bypass the limit, knowing it only makes everything worse.)

That’s where ScreenZen came in.  I needed a way—ideally, free—to say, “I only want to do these things for so much time, and then I want to be pushed off of them for a while.” When I found ScreenZen, that’s how I configured it.  It took me a bit to sort out the options and settings, so I wanted to walk through what has worked for me.

• The basic settings were pretty obvious: “open each app up to [six] times a day, for [twenty] minutes each.” That’s the number of times the app(s) can be unlocked, and how long they are usable for (in real time) once unlocked.
• I set “Strict block – after daily open goal,” so there’s a real consequence to opening things too much.
• I chose some nice offscreen activities, and set that as the intervention screen.  (Back when I started using ScreenZen, this transformed it from “tap and then wait” to “wait and then tap,” which was much better for mindfulness.)
• The time between the app being locked and being able to unlock it again is under “Advanced – cooldown time.”

I’ve also set up the schedules so that, in the wind-down before bedtime, more things are locked.  ScreenZen has become my “automatic sleep timer” for podcasts.  A short cooldown time there also prevents me from hearing it pause, deciding ‘yeah… sure… i feel awake,’ resuming the podcast, and then immediately falling asleep.  Sometimes, when only Downtime starting would stop playback, I’d find myself seeking half an hour back in an episode to get something familiar, but now, it’s not usually more than ten minutes.

Overall, this has turned out to be much better than trying to control everything through Apple’s Screen Time. If I’m on the phone for a reason, it generally doesn’t take 20 minutes, but if I get into “rat pulling a lever” mode, ScreenZen will interrupt me much sooner than Apple would.  Furthermore, because the stress of holding the phone is nonlinear over time, switching a one-hour daily limit into five half-hour-or-less sessions becomes a reasonable option.

And yes.  I would rather get off the phone than put it in a stand, or add a grip to it.  When it hurts, it’s mainly because I’m unaware of how much time I’m wasting on it.

The main downside to all this is that, if I need a how-to video on something, I have to plan ahead.  I sit down at a non-pocketable computer, watch the video, and take notes if necessary.  The videos cut out the boring parts, making it definitely impossible to follow along in real time.

That, and for other people, well… ScreenZen works hard to be “mindful” and not actually that “controlling.” It is willing to offer a bypass in several places.  I have managed not to touch it for almost an entire year, but I know it would be rather tempting for someone who doesn’t have physical reasons to avoid it.

So to wrap up… that link again is ScreenZen and it’s available for iOS, Android, and macOS.  This is not an ad nor a paid review, that isn’t an affiliate link (unless Blogger has made it one for their benefit), and this post is 100% human-generated.

Spammers Get Confused About Temporary Errors

When I wrote Everything Needs Rate Limits, I mentioned in passing that the disk-full state prevented receiving email.  The MTA was returning a temporary “try again later” code, but the clients weren’t responding.  I got several session transcripts emailed to me that were of the form:

> EHLO some-host-name
< 200 OK + capabilities listing
> MAIL FROM sender-address
< 200 OK
> RCPT TO recipient-address
< 415 Resource unavailable, try again later
> RCPT TO recipient-address
< 415 Resource unavailable, try again later
> RCPT TO recipient-address
< 415 Resource unavailable, try again later
> DATA …
< 500 No recipient given
X Connection lost

The client is recognizing that they’re getting some sort of error, but their idea of “later” was milliseconds later, and the disk space problem was not being cleared at CPU speeds.  After enough rejections of their recipient-address, they YOLO’d it and sent the email body, to no avail.

(I also think it’s pretty interesting that the MTA is happy to tell everyone else ENOSPC, yet deliver these error emails through to me.  I suppose it purposely stops accepting email with some reserve disk space, so that it can continue to deliver critical errors for a while.)

Mac Mini (M4/2024) First Impressions

I bought an M4 Mac Mini (2024) to replace my Ivy Bridge (2012) PC.

It was difficult to choose a configuration, because of the need to see a decade into the future, and the cost of upgrades.  It is hard to believe that an additional half terabyte (internal) would cost more than a whole terabyte external drive (with board, USB electronics/port, case, cable, and retail box.)

It feels pretty fast.  Apps open unexpectedly quickly.  Which is to say, on par with native apps on my 12C/16T Alder Lake work laptop.  Apparently, my expectations have been lowered by heavy use of Flatpaks.

It is quiet.  When I ejected the old USB drive I was using for file transfer, it spun down, and that was the noise I had been hearing all along.  The Mac itself is generally too quiet to hear.

It is efficient.  I have a power strip that detects when the main device is on, and powers an extra set of outlets for other devices.  Even with the strip moved from “PC” to “Netbook,” the Mini does not normally draw enough power to keep the other outlets on.  (I put the power strip on the desk and plugged it into the desk power, then turned off the Mac’s wake-on-sleep feature.  Now I can unplug the whole strip when not in use.)

It has been weird getting used to the Mac keyboard shortcuts again.  For two years, I haven’t needed to think about which computer I’m in front of; Windows and Linux share the basic structure for app shortcuts and cursor movement.  I don’t know how many times I have pressed Ctrl+T in Firefox on the Mac and waited a beat for the tab to open, before pressing Cmd+T instead.

It is extremely weird to me that the PC Home/End keys do nothing by default on the Mac.  It’s not like they do something better, or even different; they just don’t do anything. Why?

I also had to search the web to find out why an NTFS external drive couldn’t put things in the trash after I had copied them onto the Mac.  It seems the whole volume is read-only; macOS doesn’t have built-in support for writing to NTFS.  Meanwhile, I didn’t notice anything in the UI to suggest that the volume is read-only; some operations just don’t work (quietly, in the case of keyboard shortcuts.)

There was one time where I tried to wake the Mac up, and it didn't want to talk to the keyboard. I plugged and unplugged the USB (both the keyboard from the C-to-A adapter, and the adapter from the Mac) and tried it with a different keyboard, but to no avail.  I couldn’t find any way to open an on-screen keyboard with the trackpad alone.  I had to hard power off, but it has been fine ever since.

I guess that’s about it!  It doesn’t feel like “coming home” or anything, it just feels like a new computer to be set up.

How I Use Firefox

A long, long time ago, I firewalled my online and real-world identities.  I have separate email accounts for them.  Those email accounts live in separate Firefox profiles. To throw more chaff into the system, the profiles have different adblockers (commonly uBlock Origin; first runner-up is AdBlocker Ultimate) and may or may not include Privacy Possum.

Within the profile, I’ve separated things further into containers, full name Multi-Account Containers.  Google gets its own container, so that YouTube can’t follow me everywhere online.

For things where I suspect all the defenses are a problem, I also have a profile that runs in “always private browsing” mode, but is otherwise fairly open.  I very rarely need it.  I’d rather bounce out of a site that has too many annoyances, and which hasn’t sold me on its usefulness.  (Will I sign up for your newsletter?  Will I create an account to read this article?  No.)

(There’s also Pale Moon and the Windows XP VM with Firefox 52 ESR on it, for checking compatibility with Quilt Draw when I am working on that.  However, those are well outside of everyday usage.)

At work, I don’t use separate profiles; I ended up with entirely separate browsers instead.  My day-to-day work happens in Firefox, with containers for the AWS console, each of our own sites I am responsible for, and “other things requiring login.”  The theory is that a site outside the container(s) that tries to attack one inside will fail, because the login isn’t valid from outside the container.  Meanwhile, the multiple containers separate our sites and our general service providers, and the cloud against everyone.

Because the Google Panopticon Browser is becoming the new IE6, there’s a copy of Edge to make the corporate site(s) “fully supported.”  Let Microsoft spy upon themselves, and only themselves.  All so-called “AI” “features” are turned off, where possible.

Finally, for browsing work-adjacent things that aren’t actually work, like LWN and various blogs, I have LibreWolf.  [Updated 2025-09-25: formerly Waterfox, which wasn’t integrating Firefox security patches fast enough for my liking.] It doesn’t have containers or profiles, but it does have uMatrix (better security by running less remote code), along with LeechBlock so that I don’t waste the whole day in there.

uMatrix is a great defense system; in fact, too great. I wouldn’t recommend it for most people.  However, it suits my goals for that particular browser.

Containers over systemd

“Systemd will solve all your problems,” they said.

Having used a number of systemd’s security features to configure a service, I am beginning to suspect everyone uses containers because container runtimes are trying to be secure already.

It's possible to improve the security of a service with systemd, of course.  I’ve worked hard at it.  But in the end, over half the *.service file is consumed with “trying to build my own container out of systemd directives.”  ProtectHome, ProtectSystem, ProtectKernelTunables, Protect This, Protect That, blah blah blah.  The process starts from insecure by default, and then asks me to layer on every individual protection.  This is exactly the sort of thing Linux zealots used to yell at Microsoft about.  ¯\_(ツ)_/¯

But I digress.  I ended up with an excessively long systemd service configuration file, and to apply that to any other service, there’s no option besides copying and pasting those directives.  With every release of systemd, I have to comb the man pages again to see what else is available now, and carefully apply that to every service file.  It’s not easy to tell whether the security posture is up-to-date when the policy is so verbose.

[Updated 2022-03-19: systemd-analyze security foo.service is your friend. This is the best way to get a list of everything systemd thinks about security, and whether it is applied to the unit. It's a little less bad than I thought, but it is still fundamentally the Default Permit and Enumerating Badness approaches.]

Whereas a container has an isolated filesystem (its image) already, so whole classes of configuration (ProtectHome, ProtectSystem, TemporaryFileSystem) become irrelevant.  On top of that, container runtimes start with a more limited set of privileges by default, instead of handing out CAP_SYS_ADMIN and leaving it up to the administrator to carefully disable it.  Escaping from the container runtime is considered a vulnerability; escaping from a poorly-secured systemd service is considered user error.

This is all orthogonal to “containers are interop”, but I think both forces are feeding the containerization craze.  I’m left with the feeling again that systemd should have been the “obvious correct choice,” except they decided usability didn’t matter.

Containers are Interop

I mean this in the same sense as “XML was created to solve the interoperability problem.”

The container craze is about the interoperability problem across environments. By vendoring the entire distribution and communicating only over the network, they essentially provide isolation for all the dependencies of a service. Maybe that part is the same, in essence, as the nix package manager.

But then containers have one more trick: they run anywhere with a “Linux syscall interface” underneath. Any environment with Docker support can run Docker containers, anywhere. (As long as the binaries run on the host, at least.) It’s not entirely simple—orchestration is an issue, and Docker is working on that, too—but the containers themselves become highly portable, since they’re shipped as black boxes. They don’t depend on code outside themselves, and as such, that outside code cannot break them so easily.

And maybe, by so fully entwining a Linux distro to our app, we’re forgetting how to be cross-distro or cross-platform. And the old coder in me wants to grump about that. Yet, that’s also a kind of freedom. Not everyone has to learn how to write cross-platform code if the container environment defines exactly one platform to run on.

Maybe we’re losing something, but we’re also gaining ease-of-use and accessibility in the deal.

Apache Config Merging

(I found this post I drafted a year ago. I don't know why it isn't posted, so here it is...)

The Apache documentation tells you how to order directives in the same scopes, but neglects to remind you about the way different scopes merge.  So, as of 2.4, here's an overview of how it all works.

When the server first receives a URL, it searches for matching containers to determine the active configuration.  The following containers are checked at this stage, highest priority first ("last to first" or "first to last" refers to the order that the sections or directives are listed in the configuration file):

1. <If>, from last to first.
2. <Location> and <LocationMatch>, last to first.

Once the server has a physical path, directives are applied, highest priority first:

1. <Files> and <FilesMatch>, last to first.
2. <DirectoryMatch> sections matching the directory, last to first.
3. From most- to least-specific directory path (e.g. /foo/bar before /), regardless of order:
   1. Directives in .htaccess files, if allowed and present.
   2. <Directory> section.  I suspect, but haven't verified, that multiple Directory sections on the same path (e.g. <Directory /foo></Directory> ... <Directory /foo></Directory>) will apply last to first.

Directives are also merged with VirtualHost and server-wide (outside of any other section) contexts, with priority again being given to VirtualHost over the server-wide directives.  That is, a ProxyPass in the server-wide section will apply to all virtual hosts by default, but a ProxyPass within a <VirtualHost> section will be able to replace the server-wide rule.

The ordering of directives within sections, and what happens with duplicates (at multiple priority levels) in general, is defined by each individual module.

1. RewriteRules execute first-to-last when the section containing them is processed.  Stopping is subject to flags and things (mod_rewrite is a powerful beast): the [L] flag and any that imply it end processing of rewrite rules in that section; for example, rules in <VirtualHost> cannot stop rules in <Directory> from applying.
2. ProxyPass and ProxyPassMatch rules execute first-to-last, stopping when any match is found.  Thus the "longest match first" rule given in mod_proxy's documentation.
3. Alias, AliasMatch, Redirect, and RedirectMatch rules execute first-to-last, stopping when any match is found.  Likewise, this produces a "longest match first" rule that is given in the mod_alias documentation.
4. Whether a URL is tried with other modules (like mod_proxy or mod_alias) after RewriteRules have taken effect depends on how the RewriteRule is written and where it is placed.  I am not sure I understand the finer points of this, but the pass-through flag [PT] exists to force mod_rewrite to treat it as a URL and let other modules have a chance to handle it.

The main takeaway here is to remember what level you're working at, all the time.  If you place two ProxyPassMatch directives each in a separate <LocationMatch> block, then their order of application is defined by the rules for <LocationMatch>.  It is only when ProxyPass/Match directives are sharing the same section (say, both within <VirtualHost>) that they use the longest-match-first rule from mod_proxy.

One other quirk of laying out mod_proxy directives is that they're not actually valid within <Directory> or <Files> sections.  mod_proxy itself deals only with the URL space, and if another module has decided on a file-system path, then Apache is implicitly serving the content directly, as origin server.  It is then too late for pure URL-to-URL manipulations, which is the level that mod_proxy works at.

Rebuilding Dependencies Frequently

Reddit asked:

how often do you rebuild your OS level dependencies?

In practice, around twice a month, due to accumulation of security updates. Sometimes bit-rot plays a role as well, but mostly, it’s the endless stream of updates. We want images to quickly launch into a secure state.

Waiting until the instance comes up to apply updates has two drawbacks: one, each new instance duplicates the work. Two, if there’s an update that requires a reboot, we’ve found through repeated, painful experience that there’s no way to reliably reboot our instances programatically.

IIRC though, that question was in the context of Docker containers, while our process is to precompile our app and its dependencies into a monolithic AMI that we deploy to instances in AWS. I like to think it’s basically the same, but I haven’t really used Docker much.

Linode KVM: not recommended

Looks like KVM has been a de facto downgrade. I never had constant small dropouts on Xen, and I had certainly never had my server go silent for hours. (Linode 2048, btw. It started as a 512.)

It might be time to go looking for a new VPS provider...?

Apache, FastCGI, and the Authorization header

I couldn’t find much about why Apache (up to and including 2.4.x) doesn’t pass the HTTP Authorization header to FastCGI by searching the Internet, so I fished through their source code.

There’s a function that sets up the default variables to pass to CGI programs, called ap_add_common_vars. This function expects that other users on the server can see these variables with ps -e, so to hide usernames and passwords from appearing in there, it specifically avoids passing Authorization through to CGI.

FastCGI uses the same ap_add_common_vars function, and mod_proxy_fcgi doesn’t take any special care to put the Authorization back in for requests that are sent over a socket. Thus, by default, no FastCGI requests will receive usernames and passwords.

The standard workaround is to use mod_rewrite to set an environment flag if the request has authorization. Because nothing unsets the Authorization—ap_add_common_vars simply skips setting a default—that explicitly-set environment variable will be included in the FastCGI request.

RewriteCond %{HTTP:Authorization} .
RewriteRule / - [E:HTTP_AUTHORIZATION=%{HTTP:Authorization}]

That’s fine if you’re already using mod_rewrite. Can it be done without that module? I haven’t benchmarked the difference, but mod_setenvif can do it via the rule:

SetEnvIf Authorization (.+) HTTP_AUTHORIZATION=$1

tl;dr: Authorization isn’t passed to FastCGI because of the default protection from exposing passwords with slow CGI. It can safely be passed to a FastCGI server (requests are sent over a socket, not through world-readable process attributes) but it must be explicitly passed.

Updated 28 Jul 2015: I finally tested the SetEnvIf rule. I also mentioned what Apache line I'm working with.

Hacking my Habits: screen vim → gvim

I spend much of my time using the screen command inside a VM, opening vim in a new window by running screen vim.  When I'm on bare metal, then, sometimes I accidentally start a whole screen for a quick edit, and I don't realize it until I exit vim and it says, [screen is terminating].

So I built a little function into my ~/.bash_aliases (sourced from the default .bashrc) on the host machine:

# if I run `screen vim` outside of screen, invoke gvim instead.
screen() {
 if [[ $1 = vim && -z $STY ]] ; then
  shift
  gvim ${1+"$@"}
 else
  command screen ${1+"$@"}
 fi
}

Now "screen <anything but vim>" will run screen, and "screen vim" will invoke gvim.  The latter may surprise me, but it'll be what I actually want.

Update: I put similar code inside the VM to convert "gvim" into "screen vim" or regular "vim", as appropriate. Now I abuse this all the time to save three keystrokes ("gvim" vs "scr<TAB>vim").

Pointless metrics: Uptime

Back in the day when I was on Slashdot, a popular pasttime was making fun of Windows. The 9x line had a time counter that rolled over after 30-45 days (I forget and can’t dredge the real number up off the Internet now), which would crash the system. Linux could stay up, like, forever dude, because Unix is just that stable.

So I spent a while thinking that ‘high uptime’ was a good thing. I was annoyed, once upon a time, at a VPS provider that regularly rebooted my system for security upgrades approximately monthly, because they were using Virtuozzo and needed to really, seriously, and truly block privilege escalations.

About monthly. As in 30-45 days…

I thought that was bad, but nowadays, the public-facing servers my employer runs live for less than a week. Maybe a whole week if they get abnormally old before Auto Scaling gets around to culling them. And I’m cool with this!

I try to rebuild an image monthly even if “nothing” has happened, and definitely whenever upstream releases a new base image, and sometimes just because I know “major” updates were made to our repos (e.g. I just did composer update everywhere) and it’ll save some time in the self-update scripts when the image relaunches.

It turns out that the ‘uptime’ that the Slashdot crowd were so proud of was basically counterproductive. I do not want to trade security, agility, or anything else just to make that number larger. There is no benefit from it! Nothing improves solely because the kernel has been running longer, and if it does, then the kernel needs fixed to provide that improvement instantly.

And if the business is structured around one physical system that Must Stay Running, then the business on that server is crucial enough to have redundancy, backups, failovers… and regular testing of the failover by switching onto fresh hardware with fresh uptimes.

AppArmor and the problems of LSM

AppArmor is a pretty excellent framework.  It clearly works according to its design.  There are only two major problems, one of which is: apps can't give out useful error messages when an AppArmor check fails.

Since LSMs insert themselves inside the kernel, the only thing they can really do if an access check fails is force the system call to return EPERM, "permission denied."  An app can't actually tell whether it has been denied access because of filesystem permissions, LSM interference, or the phase of the moon, because the return code can't carry any metadata like that.  Besides which, it's considered bad practice to give an attacker detail about a failing security check, thus the ubiquitous and uninformative "We emailed you a password reset link if you gave us a valid email" message in password reset flows.
Thus, the hapless administrator does something perfectly reasonable, AppArmor denies it, and it becomes an adventure to find the real error message.  AppArmor tends to hide its messages away in a special log file, so that the normal channels and the app's log (if different) only show a useless EPERM message for a file that the app would seem to have access to upon inspection of the filesystem.

Adding more trickiness, AppArmor itself doesn't always apply to an application, so testing a command from a root shell will generally work without issue.  Those root shells are typically "unconfined" and don't apply the profiles to commands run from them.

The other main problem is that it requires profile developers to be near-omniscient.  It's nice that tcpdump has a generic mechanism to run a command on log rotation, but the default AppArmor profile only sets up access for gzip or bzip2... and even then, if it's a *.pcap file outside of $HOME, it can't be compressed because the AppArmor profile doesn't support creating a file with the compressed extension.  (That can be fixed.)

It's nice that charon (part of strongswan) has a mechanism to include /etc/ipsec.*.secrets so that I could store everyone's password in /etc/ipsec.$USER.secrets ... but the profile doesn't let charon list what those files are even though it grants access to the files in question.  So using the include command straight out of the example/documentation will, by default, allow the strongswan service to start... but it won't be able to handle any connections.

I had SELinux issues in the past (which share all the drawbacks of AppArmor since it's just another LSM) when I put an Apache DocumentRoot outside of /var/www.  In that case, though, I disabled SELinux entirely instead of trying to learn about it.

tl;dr: it's pretty much a recipe for Guide Dang It.

Add permissions to an AppArmor profile

Background: I tested something in the shell as root, then deployed it as an upstart script, and it failed.  Checking the logs told me that AppArmor had denied something perfectly reasonable, because the developers of the profile didn't cover my use case.

Fortunately, AppArmor comes with a local override facility.  Let's use it to fix this error (which may be in /var/log/syslog or /var/log/audit.log depending on how the system is set up):

kernel: [ 7226.358630] type=1400 audit(1416403573.247:17): apparmor="DENIED" operation="mknod" profile="/usr/sbin/tcpdump" name="/var/log/tcpdump/memcache-20141119-072608.pcap.gz" pid=2438 comm="gzip" requested_mask="c" denied_mask="c" fsuid=0 ouid=0

What happened?  requested_mask="c" means the command (comm="gzip") tried to create a file (given by the name value) and AppArmor denied it.  The profile tells us, indirectly, which AppArmor file caused the denial; substitute the first '/' with the AppArmor config dir (/etc/apparmor.d/) and the rest with dots.  Thus, if we look in /etc/apparmor.d/usr.sbin.tcpdump, we find the system policy.  It has granted access to *.pcap case-insensitively anywhere in the system, with the rule /**.[pP][cC][aA][pP] rw.  I used that rule to choose my filename outside of $HOME, but now the -z /bin/gzip parameter I used isn't able to do its work because it can't create the *.gz version there.

Incidentally, the command differs from the profile in this case, because tcpdump executed gzip.  It's allowed to do that by the system profile, which uses ix permissions—that stands for inherit-execute, and means that the gzip command is run under the current AppArmor profile.  All the permissions defined for tcpdump continue to affect the gzip command.

Anyway, back to that fix I promised.  AppArmor provides /etc/apparmor.d/local/ for rules to add to the main ones.  (Although this can't be used to override an explicit deny like tcpdump's ban on using files in $HOME/bin.)  We just need to add a rule for the *.gz, and while we're there, why not the *.bz2 version as well?

/**.[pP][cC][aA][pP].[gG][zZ] rw,
/**.[pP][cC][aA][pP].[bB][zZ]2 rw,

The trailing comma does not seem to be an issue for me.  Note also that we don't need to specify the binary and braces, since the #include line in the system profile is already inside the braces.

Ubuntu ships some files in the local directory already; we should be able to run sudo -e /etc/apparmor.d/local/usr.sbin.tcpdump and add the lines above to the existing file.  Once the file is ready, we need to reload that profile to the kernel.  Note that we use the system profile here, not the one we just edited:

sudo apparmor_parser -r /etc/apparmor.d/usr.sbin.tcpdump

I'm not clear enough on how AppArmor works to know if we need to restart the service now (I'm purposely leaving my tcpdump service file/discussion out of this post) because I restarted mine just to be safe.

Web project repository layout

I used to stuff all my web files under the root of the project repository when I was developing "a web site", and include lots of tricky protections (is random constant defined and equal known value?) to 'protect' my includes from being directly accessed over the web.  I was inspired at the time by a lot of open-source projects that seemed to work this way, including my blog software at the time (serendipity); when one downloaded the project, all their files were meant to be FTP'd to your server's document root.

However, these days I'm mostly developing for servers my company or I administer, so I don't need to make the assumption that there is "no place outside the DocumentRoot."  Instead, I designate the top level of the repo as a free-for-all, and specify one folder within as the DocumentRoot.

This means that all sorts of housekeeping can go on up in the repo root, like having composer files and its vendor tree.  Like having the .git folder and files safely tucked away.  Like having a build script that pre-compresses a bunch of JavaScript/CSS... or a Makefile.  Or an extra document root with a static "Closed for maintenance" page.  The possibilities are almost endless.

Some of our sites rely on RewriteRules, and that configuration also lands outside the DocumentRoot, to be included from the system Apache configuration.  This lets us perform rewrites at the more-efficient server/URL level instead of filesystem/directory level, while allowing all the code to be updated in the repo.  When we change rewriting rules, that goes right into the history with everything else.

To give a concrete example, a website could look like this on the server:

• public/
• index.php
• login.php
• pdf-builder.php
• pdf-loader.php
• css/
• js/
• img/
• tcpdf/
• (the TCPDF library code)
• built-pdf/
• 20141011/
• 1f0acb7623a40cfa.pdf
• cron/
• expire-built-pdfs.php
• conf/
• rewrite.conf
• composer.lock
• composer.json
• vendor/
• aws/
• aws-sdk-php/
• guzzle/
• ...

In this case, DocumentRoot is the public folder, it uses TCPDF to generate private PDFs into built-pdf and lets the user download them, doing access control in pdf-loader.php.  It divides the built pdfs up by day so they can be expired conveniently by the cron script, it has some rewrite rules, and it uses Composer.  (Disclaimer: I pretty much made up all the details here, but the theory is broadly applicable.)

Again, this doesn't really work for letting other people publish on a shared host (rewrite.conf could conceivably do anything allowable in VirtualHost context, not just rewrite) but we own the whole host.

An AWS VPN: Part III, Routing

The last two posts in this series (part I, IPSec and part II, L2TP) covered enough to get an L2TP/IPSec VPN connection up, to the point where arbitrary traffic can be exchanged between the client and the server.  But, there's a missing feature yet.  Remember this picture from part II?

If we assigned a third network for the VPN, how does the client know that the protected network is even back there to send traffic to?  It's not the remote end of the VPN link (172.16.0.1) so it'll get routed via the gateway and fail at some point.

The answer is that someone conveniently threw in an extra hack for us: the VPN client sends a DHCPINFORM message over the L2TP connection, and the server just has to respond to it with a few vital options.

An AWS VPN: Part II, L2TP/PPP

In part I of this series, we covered the IPSec layer.  With IPSec up and running, we can move on to configuring L2TP; routing will be covered in part III.

In the "usual" VPN setup, we'd cut a hole in the existing DHCP pool in the protected network to reserve as addresses to be given to VPN clients.  In a /24 subnet, network gear might get addresses up to 30, the DHCP pool could run from 31-150, and other static IPs (hi, printers!) might live at 200-254, with 151-199 open for VPN access.

Unfortunately, my protected network in this scenario is AWS, which I can't actually carve random IPs out of to assign to connecting clients.  I have to assign one in a different netblock and hope none of the three (the client, AWS, and my new L2TP-only address pool) conflict.

Luckily, there are three private network spaces defined, and everyone always forgets about the weird one.  Let's set up the connection something like this:

Of course I didn't actually use 172.16.0.x myself, that's asking for almost as much trouble as 192.168.0.x.  (Also, I guess I could have colored the NAT-to-server link green as well, because that's the same network space in my AWS setup.  Too late, we're going!)

An AWS VPN: Part I, IPSec

I recently set up an IPSec/L2TP VPN.  This post is about how I did it, and how I debugged each individual part.  In honor of how long it took me, this will be a three-part series, one for each day of work I spent on it.  Part II will cover L2TP/PPP, and part III will get into routing and DNS.

First things first: IP addresses have been changed.  Some details will apply to Linux, OpenSWAN, xl2tpd, pppd, iproute2, dnsmasq, or road-warrior configurations in particular, but the theory is applicable cross-platform.  I want to cover both, because the theory is invaluable for debugging the practice.

This also covers a few details about putting the VPN server on an AWS instance in EC2 Classic with an elastic IP.  But first, let's take a look at the general structure we have to work with:

A client connecting from a network we don't control, over networks that are possibly evil (hence the need for a VPN), to a server that provides access to vague "other resources" in non-globally-routable space behind that server.  We don't know where the client is going to be, and/or the corporate net is sufficiently large, that we can't use dirty NAT tricks to move each network "into" the other.  (But, spoiler, we will take some inspiration from there.)

Without further ado, here's part I: IPSec.  Part II and III are forthcoming.