AppArmor and the problems of LSM

AppArmor is a pretty excellent framework.  It clearly works according to its design.  There are only two major problems, one of which is: apps can't give out useful error messages when an AppArmor check fails.

Since LSMs insert themselves inside the kernel, the only thing they can really do if an access check fails is force the system call to return EPERM, "permission denied."  An app can't actually tell whether it has been denied access because of filesystem permissions, LSM interference, or the phase of the moon, because the return code can't carry any metadata like that.  Besides which, it's considered bad practice to give an attacker detail about a failing security check, thus the ubiquitous and uninformative "We emailed you a password reset link if you gave us a valid email" message in password reset flows.
Thus, the hapless administrator does something perfectly reasonable, AppArmor denies it, and it becomes an adventure to find the real error message.  AppArmor tends to hide its messages away in a special log file, so that the normal channels and the app's log (if different) only show a useless EPERM message for a file that the app would seem to have access to upon inspection of the filesystem.

Adding more trickiness, AppArmor itself doesn't always apply to an application, so testing a command from a root shell will generally work without issue.  Those root shells are typically "unconfined" and don't apply the profiles to commands run from them.

The other main problem is that it requires profile developers to be near-omniscient.  It's nice that tcpdump has a generic mechanism to run a command on log rotation, but the default AppArmor profile only sets up access for gzip or bzip2... and even then, if it's a *.pcap file outside of $HOME, it can't be compressed because the AppArmor profile doesn't support creating a file with the compressed extension.  (That can be fixed.)

It's nice that charon (part of strongswan) has a mechanism to include /etc/ipsec.*.secrets so that I could store everyone's password in /etc/ipsec.$USER.secrets ... but the profile doesn't let charon list what those files are even though it grants access to the files in question.  So using the include command straight out of the example/documentation will, by default, allow the strongswan service to start... but it won't be able to handle any connections.

I had SELinux issues in the past (which share all the drawbacks of AppArmor since it's just another LSM) when I put an Apache DocumentRoot outside of /var/www.  In that case, though, I disabled SELinux entirely instead of trying to learn about it.

tl;dr: it's pretty much a recipe for Guide Dang It.

Add permissions to an AppArmor profile

Background: I tested something in the shell as root, then deployed it as an upstart script, and it failed.  Checking the logs told me that AppArmor had denied something perfectly reasonable, because the developers of the profile didn't cover my use case.

Fortunately, AppArmor comes with a local override facility.  Let's use it to fix this error (which may be in /var/log/syslog or /var/log/audit.log depending on how the system is set up):

kernel: [ 7226.358630] type=1400 audit(1416403573.247:17): apparmor="DENIED" operation="mknod" profile="/usr/sbin/tcpdump" name="/var/log/tcpdump/memcache-20141119-072608.pcap.gz" pid=2438 comm="gzip" requested_mask="c" denied_mask="c" fsuid=0 ouid=0

What happened?  requested_mask="c" means the command (comm="gzip") tried to create a file (given by the name value) and AppArmor denied it.  The profile tells us, indirectly, which AppArmor file caused the denial; substitute the first '/' with the AppArmor config dir (/etc/apparmor.d/) and the rest with dots.  Thus, if we look in /etc/apparmor.d/usr.sbin.tcpdump, we find the system policy.  It has granted access to *.pcap case-insensitively anywhere in the system, with the rule /**.[pP][cC][aA][pP] rw.  I used that rule to choose my filename outside of $HOME, but now the -z /bin/gzip parameter I used isn't able to do its work because it can't create the *.gz version there.

Incidentally, the command differs from the profile in this case, because tcpdump executed gzip.  It's allowed to do that by the system profile, which uses ix permissions—that stands for inherit-execute, and means that the gzip command is run under the current AppArmor profile.  All the permissions defined for tcpdump continue to affect the gzip command.

Anyway, back to that fix I promised.  AppArmor provides /etc/apparmor.d/local/ for rules to add to the main ones.  (Although this can't be used to override an explicit deny like tcpdump's ban on using files in $HOME/bin.)  We just need to add a rule for the *.gz, and while we're there, why not the *.bz2 version as well?

/**.[pP][cC][aA][pP].[gG][zZ] rw,
/**.[pP][cC][aA][pP].[bB][zZ]2 rw,

The trailing comma does not seem to be an issue for me.  Note also that we don't need to specify the binary and braces, since the #include line in the system profile is already inside the braces.

Ubuntu ships some files in the local directory already; we should be able to run sudo -e /etc/apparmor.d/local/usr.sbin.tcpdump and add the lines above to the existing file.  Once the file is ready, we need to reload that profile to the kernel.  Note that we use the system profile here, not the one we just edited:

sudo apparmor_parser -r /etc/apparmor.d/usr.sbin.tcpdump

I'm not clear enough on how AppArmor works to know if we need to restart the service now (I'm purposely leaving my tcpdump service file/discussion out of this post) because I restarted mine just to be safe.

Unison fatal error: lost connection to server

My unison sync started dying this week with an abrupt loss of connection.  It turns out, when I scrolled back up past the status spam, that there was an additional error message: Uncaught exception Failure("input_value: bad bigarray kind")

This turns out to be caused by a change of serializer between ocaml 4.01 and 4.02.  Unison compiled with 4.02 can't talk to pre-4.02 unisons successfully, and the reason mine failed this week is because I did some reinstalls following an upgrade to OS X Yosemite.  So I had homebrew's unison compiled with 4.02+ and Ubuntu's unison on my server, compiled with 4.01.

Even if I updated the server to Ubuntu utopic unicorn, it wouldn't solve the problem.  ocaml 4.02 didn't make it in because it wasn't released soon enough.

So, time to build unison from source against ocaml built from source!

One problem: ocaml 4.01 doesn't build against clang... and on Yosemite, that's all there is.  I ended up hacking the ./configure script to delete all occurrences of the troublesome -fno-defer-pop flag, then redoing the install.  Per the linked bug, it was added in the 01990s to avoid tickling a code generation bug in gcc-1.x.  Newer clangs won't complain about this particular flag, and newer ocamls won't use it, but I chose to stick with the old (semi-current) versions of  both.

Happily, that seemed to be the only problem, and the rest of the unison build (namely, the text UI; I didn't want a heap of dependencies) went fine.

(And this, friends, is why you should define a real file format, instead of just dumping your internal state out through your native serializer.  For things you control, you can have much broader cross-compatibility.)

Web project repository layout

I used to stuff all my web files under the root of the project repository when I was developing "a web site", and include lots of tricky protections (is random constant defined and equal known value?) to 'protect' my includes from being directly accessed over the web.  I was inspired at the time by a lot of open-source projects that seemed to work this way, including my blog software at the time (serendipity); when one downloaded the project, all their files were meant to be FTP'd to your server's document root.

However, these days I'm mostly developing for servers my company or I administer, so I don't need to make the assumption that there is "no place outside the DocumentRoot."  Instead, I designate the top level of the repo as a free-for-all, and specify one folder within as the DocumentRoot.

This means that all sorts of housekeeping can go on up in the repo root, like having composer files and its vendor tree.  Like having the .git folder and files safely tucked away.  Like having a build script that pre-compresses a bunch of JavaScript/CSS... or a Makefile.  Or an extra document root with a static "Closed for maintenance" page.  The possibilities are almost endless.

Some of our sites rely on RewriteRules, and that configuration also lands outside the DocumentRoot, to be included from the system Apache configuration.  This lets us perform rewrites at the more-efficient server/URL level instead of filesystem/directory level, while allowing all the code to be updated in the repo.  When we change rewriting rules, that goes right into the history with everything else.

To give a concrete example, a website could look like this on the server:

• public/
• index.php
• login.php
• pdf-builder.php
• pdf-loader.php
• css/
• js/
• img/
• tcpdf/
• (the TCPDF library code)
• built-pdf/
• 20141011/
• 1f0acb7623a40cfa.pdf
• cron/
• expire-built-pdfs.php
• conf/
• rewrite.conf
• composer.lock
• composer.json
• vendor/
• aws/
• aws-sdk-php/
• guzzle/
• ...

In this case, DocumentRoot is the public folder, it uses TCPDF to generate private PDFs into built-pdf and lets the user download them, doing access control in pdf-loader.php.  It divides the built pdfs up by day so they can be expired conveniently by the cron script, it has some rewrite rules, and it uses Composer.  (Disclaimer: I pretty much made up all the details here, but the theory is broadly applicable.)

Again, this doesn't really work for letting other people publish on a shared host (rewrite.conf could conceivably do anything allowable in VirtualHost context, not just rewrite) but we own the whole host.

MySQL 5.6 TIMESTAMP changes

So you upgraded to MySQL 5.6 and there's a crazy warning in your error log about TIMESTAMP columns doing stuff, or not, when explicit_defaults_for_timestamp is enabled, or not?

It's actually pretty simple: TIMESTAMP columns without any DEFAULT nor ON UPDATE clause are going to change behavior in the future, and MySQL 5.6 has an option to allow choosing whether to opt-in to that future behavior at present.

MySQL 5.6 without the explicit_defaults_for_timestamp option set, which is default, will continue treating a column defined as simply TIMESTAMP (possibly also NOT NULL) as if it were defined TIMESTAMP [NOT NULL] DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP.

MySQL 5.6 with the explicit_defaults_for_timestamp option set will behave like the future planned versions of MySQL, where TIMESTAMP will be treated as TIMESTAMP DEFAULT NULL and TIMESTAMP NOT NULL will behave as TIMESTAMP NOT NULL DEFAULT 0.  Implied automatic updates will be no more.

That's all.

All of the other historic TIMESTAMP behaviors, such as assigning NULL to a column declared with NOT NULL actually assigning CURRENT_TIMESTAMP, remain unchanged by this update.  There are some brand-new capabilities such as fractional seconds, applying default/update clauses to DATETIME columns, and setting those clauses on more than one column in a table.  However, those features aren't a change of meaning for existing definitions, so they're unaffected by the option.

PHP-FPM error logging (where FastCGI errors go)

Let's start with the catchy summary graphic:

There are a lot of streams in the worker because historically, CGI directly connected a worker's standard output and error streams (file descriptors 1 and 2 on Unix) to the web server.  FastCGI kept the notions, but established new, separate FastCGI output and error streams for FastCGI-aware applications to use.  It also specified that the standard output/error streams should be discarded.

So by default, that's exactly what the FPM master does: it arranges for workers' standard streams to be discarded.  That's actually fine with PHP because the error logs go through the SAPI layer, and that layer writes them to the FastCGI error stream when FastCGI is active.

By default, that stream goes through to the web server, where Apache logs it to whatever error log is in effect.  (Typically, I've worked on sites that have per-virtualhost logs.  Especially for error logs, it's a lot easier to find the relevant messages when it's known that Host X is down.)

PHP-FPM, however, also has the option of setting per-pool php.ini values in the pool configuration.  If a value is provided there (e.g. /etc/php-fpm-5.5.d/www.conf) for the error_log setting, then that pool will log to the specified destination instead of the Web server's error log.  In other words, the php_value[error_log] set in the pool configuration overrides the default behavior of logging to the FastCGI error stream.

These options turn out to be entirely independent of the master FastCGI configuration setting named catch_workers_output.  This option, in the php-fpm configuration (e.g. /etc/php-fpm-5.5.conf) controls whether the standard streams are discarded as specified, or if they will appear as lines in the master process's error log.  Unless the workers are dying unexpectedly, this probably doesn't contain anything interesting.

jQuery event handlers and Content-Security-Policy

I wrote a while back about the efficiency angle of using .on() to set up event handlers, but I recently discovered a surprising additional feature: when an AJAX request to the server returns HTML, jQuery (or bootstrap?) carefully lifts out the embedded <script> elements, and runs their content via eval().  This doesn't work with a Content-Security-Policy in place, because eval is forbidden.

In my case, the script just wanted to bind a handler to all the newly-added elements that were also provided in the response.

Switching to binding the handler to the modal itself and filtering the interesting children meant that not only did I get the memory and CPU improvements of binding a single handler one time, but it eliminated the script—and the need to eval it—from the response.

Before, event handlers were bound with this code in the response, via jQuery secretly calling eval():

<script type="text/javascript">
$('#clientlist a').click(activateClient);
</script>

Afterwards, the response contained no script, and the event handlers were bound natively in my page initialization code:

$('#clientmodal').on('click', '#clientlist a', activateClient);

The content of #clientmodal was replaced whenever the modal was opened, but the modal itself (and its event handler) remained in place.