Using Cloudflare DNS over TLS system-wide on Ubuntu

My current Linux distributions (Ubuntu 23.04 and the Ubuntu-derived Pop!_OS 22.04) use NetworkManager for managing connections, and systemd-resolved for resolving DNS queries.  I’ve set up Cloudflare’s public DNS service with DoT (DNS over TLS) support twice… and I don’t really have a solid conclusion.  Is one “better?”  🤷🏻

Contents

• Per-connection mode with NetworkManager only
• Globally with systemd-resolved / NetworkManager
• Useful background info

Using sshuttle with ufw outbound filtering on Linux (Pop!_OS 22.04)

I am using sshuttle and UFW on my Linux system, and I recently set up outbound traffic filtering (instead of default-allow) in ufw.  Immediately, I noticed I couldn’t make connections via sshuttle anymore.

The solution was to add another rule to ufw:

allow out from anywhere to IP 127.0.0.1, TCP port 12300

Note that this is “all interfaces,” not tied to the loopback interface, lo.

Now… why does this work?  Why doesn’t this traffic already match one of the “accept all on loopback” rules?

To receive that sshuttle is responsible for, sshuttle listens at 127.0.0.1:12300 (by default) and creates some NAT rules to redirect traffic for its subnet to that IP and port.  That is, running sshuttle -r example.com 192.168.99.0/24 creates a NAT rule to catch traffic to any host within 192.168.99.0/24.  This is done in netfilter’s nat tables.

UFW has its rules in the filter tables, and the nat tables run first. Therefore, UFW sees a packet that has already been redirected, and this redirection changes the packet’s destination while its interface and source remain the same!

That’s the key to answering the second question: the “allow traffic on loopback” rules are written to allow traffic on interface lo, and these redirected packets have a different interface (Ethernet or Wi-Fi.) The public interfaces are not expected to have traffic for local addresses on them… but if they do, they don’t get to take a shortcut through the firewall.

With this understanding, we can also see what’s going wrong in the filtering rules.  Without a specific rule to allow port 12300 outbound, the packet reaches the default policy, and if that’s “reject” or “deny,” then the traffic is blocked.  sshuttle never receives it.

Now we can construct the proper match rule: we need to allow traffic to IP 127.0.0.1 on TCP port 12300, and use either “all interfaces” or our public (Ethernet/Wi-Fi) interface.  I left mine at “all interfaces,” in case I should ever plug in the Ethernet.

(I admit to a couple of dead-ends along the way.  One, allowing port 3306 out didn’t help.  Due to the NAT redirection, the firewall never sees a packet with port 3306 itself.  This also means that traffic being forwarded by sshuttle can’t be usefully firewalled on the client side.  The other problem was that I accidentally created the rule to allow UDP instead of TCP the first time.  Haha, oops.)

Argon2id Parameters

There are no specific recommendations in this post. You will need to choose parameters for your specific situation.  However, it is my hope that this post will add deeper understanding to the recommendations that others make.  With that said…

The primary parameter for Argon2 is memory. Increasing memory also increases processing time.

The time cost parameter is intended to make the running time longer when memory usage can’t be increased further.

The threads (or parallelism, or “lanes” when reading the RFC) parameter sub-divides the memory usage.  When the memory is specified as 64 MiB, that is the total amount used, whether threads are 1 or 32.  However, the synchronization overhead causes a sub-linear speedup, and this is more pronounced with smaller memory sizes.  SMT cores offer even less speed improvement than the same number of non-SMT cores, as expected.

I did some tests on my laptop, which has 4 P-cores and 8 E-cores (16 threads / 12 physical cores.) The 256 MiB tests could only push actual CPU usage to about 600% (compared to the 1260% we might expect); it took 1 GiB or more to reach 1000% CPU.  More threads than cores didn’t achieve anything.

Overall then, higher threads allow for using more memory, if enough CPU is available to support the thread count.  If memory and threads are both in limited supply, then time cost is the last resort for extending the operation time until it takes long enough.

Bonus discovery: in PHP, the argon2id memory is separate from the memory limit.  memory_get_peak_usage() reported the same number at the beginning and end of my test script, even for the 1+ GiB tests.

Effectively Using Future in Perl

I’ve been working a lot with the Future library and IO::Async in Perl recently.  There was a bug in our memcache/dynamo proxy, so I ended up doing a lot of investigation about Futures in order to simulate the bug and verify the fix.  So, I want to talk about how Futures work and why it was so difficult for me.

Unikernels

I learned a new term recently: unikernel.  Wikipedia defines it as a “single address space machine image” or else uses the “library operating system” term, but the gist of it is:

Unikernel: A system where a single application runs in unprotected mode.

Instead of a “normal” operating system managing multiple processes, a unikernel has the OS services arranged as a library, and runs one process with that library in the kernel space of the host hardware.

This used to be fairly impractical, as you’d have to dedicate physical hardware to the unikernel, and it would still need drivers for that hardware. It was a lot of work and not very portable.  But with virtualization, a unikernel can build to the hypervisors’ interfaces, then run in any environment that supports the hypervisor.

Here’s the difference from other approaches once virtualization gets in the mix: a unikernel uses the hypervisor only to provide isolation and virtio devices, then runs one process with no further hardware protections inside the guest.  Networking, filesystems, and anything else normally provided by the OS through the syscall interface are, instead, built as function calls living in the same address space with the main application.

A container uses a host OS kernel in much the same way, to run each container as a host process.  While this requires a specific syscall interface, it also means the container environment can take advantage of the host OS’s drivers, filesystems, and networking.  The container and hypervisor are fully unaware of each other, which also enables them to run on a host OS that’s running on bare metal.

A traditional guest system, of course, runs a full OS inside the guest and ordinary processes inside the OS.  Like the unikernel, it boots anywhere with the requisite CPU/virtio support, regardless of underlying host OS, but like a container, it continues to provide OS services like networking, filesystems, and process management through the syscall interface.

(Plenty of other ink has been spilled about the security of all this, but the tl;dr is, hypervisors more-or-less secure multi-tenant hardware.  Running in the cloud means being virtualized, even with containers.)

Here’s a rough drawing of two virtual-machine architectures (KVM and Xen), and how they differ from containers and unikernels:

Obviously, unikernels are also a bigger change in application architecture from containers (a single package and its dependencies, minus a kernel) or a full OS, but they’re well-suited for single-language microservices.  Once a language has the “OS library” to provide networking and filesystems (if needed), any app written to that library can be compiled to a unikernel.

The unikernel, essentially, gives up composability in the pursuit of speed. Additional process can’t be added in, because the unikernel doesn’t support processes or process isolation without losing its essential difference from an ordinary OS or container.  (Threads are possible, but not processes.)

Perhaps, again, the difference is best described with a picture:

In the unikernel, what are ordinarily “operating system services” and shared library calls are compiled into a single address space as ordinary function calls.  Getting rid of the user/system split is of course their whole point, and shared libraries also disappear because, as one process, there is no “other” process to share with.  All the mechanics involved in doing so become unnecessary.

As noted above, hypervisors have made unikernels much more interesting and feasible lately.  There’s a full list at unikernel.org, particularly their projects page, but the two that look most interesting to me are:

• OS^v
• rumprun

I don’t know when—or even if—I’ll get around to doing anything with them, but those are likely to be the easiest-to-use unikernels, unless someone makes node.js into one.

Apache Config Merging

(I found this post I drafted a year ago. I don't know why it isn't posted, so here it is...)

The Apache documentation tells you how to order directives in the same scopes, but neglects to remind you about the way different scopes merge.  So, as of 2.4, here's an overview of how it all works.

When the server first receives a URL, it searches for matching containers to determine the active configuration.  The following containers are checked at this stage, highest priority first ("last to first" or "first to last" refers to the order that the sections or directives are listed in the configuration file):

1. <If>, from last to first.
2. <Location> and <LocationMatch>, last to first.

Once the server has a physical path, directives are applied, highest priority first:

1. <Files> and <FilesMatch>, last to first.
2. <DirectoryMatch> sections matching the directory, last to first.
3. From most- to least-specific directory path (e.g. /foo/bar before /), regardless of order:
   1. Directives in .htaccess files, if allowed and present.
   2. <Directory> section.  I suspect, but haven't verified, that multiple Directory sections on the same path (e.g. <Directory /foo></Directory> ... <Directory /foo></Directory>) will apply last to first.

Directives are also merged with VirtualHost and server-wide (outside of any other section) contexts, with priority again being given to VirtualHost over the server-wide directives.  That is, a ProxyPass in the server-wide section will apply to all virtual hosts by default, but a ProxyPass within a <VirtualHost> section will be able to replace the server-wide rule.

The ordering of directives within sections, and what happens with duplicates (at multiple priority levels) in general, is defined by each individual module.

1. RewriteRules execute first-to-last when the section containing them is processed.  Stopping is subject to flags and things (mod_rewrite is a powerful beast): the [L] flag and any that imply it end processing of rewrite rules in that section; for example, rules in <VirtualHost> cannot stop rules in <Directory> from applying.
2. ProxyPass and ProxyPassMatch rules execute first-to-last, stopping when any match is found.  Thus the "longest match first" rule given in mod_proxy's documentation.
3. Alias, AliasMatch, Redirect, and RedirectMatch rules execute first-to-last, stopping when any match is found.  Likewise, this produces a "longest match first" rule that is given in the mod_alias documentation.
4. Whether a URL is tried with other modules (like mod_proxy or mod_alias) after RewriteRules have taken effect depends on how the RewriteRule is written and where it is placed.  I am not sure I understand the finer points of this, but the pass-through flag [PT] exists to force mod_rewrite to treat it as a URL and let other modules have a chance to handle it.

The main takeaway here is to remember what level you're working at, all the time.  If you place two ProxyPassMatch directives each in a separate <LocationMatch> block, then their order of application is defined by the rules for <LocationMatch>.  It is only when ProxyPass/Match directives are sharing the same section (say, both within <VirtualHost>) that they use the longest-match-first rule from mod_proxy.

One other quirk of laying out mod_proxy directives is that they're not actually valid within <Directory> or <Files> sections.  mod_proxy itself deals only with the URL space, and if another module has decided on a file-system path, then Apache is implicitly serving the content directly, as origin server.  It is then too late for pure URL-to-URL manipulations, which is the level that mod_proxy works at.

An AWS VPN: Part I, IPSec

I recently set up an IPSec/L2TP VPN.  This post is about how I did it, and how I debugged each individual part.  In honor of how long it took me, this will be a three-part series, one for each day of work I spent on it.  Part II will cover L2TP/PPP, and part III will get into routing and DNS.

First things first: IP addresses have been changed.  Some details will apply to Linux, OpenSWAN, xl2tpd, pppd, iproute2, dnsmasq, or road-warrior configurations in particular, but the theory is applicable cross-platform.  I want to cover both, because the theory is invaluable for debugging the practice.

This also covers a few details about putting the VPN server on an AWS instance in EC2 Classic with an elastic IP.  But first, let's take a look at the general structure we have to work with:

A client connecting from a network we don't control, over networks that are possibly evil (hence the need for a VPN), to a server that provides access to vague "other resources" in non-globally-routable space behind that server.  We don't know where the client is going to be, and/or the corporate net is sufficiently large, that we can't use dirty NAT tricks to move each network "into" the other.  (But, spoiler, we will take some inspiration from there.)

Without further ado, here's part I: IPSec.  Part II and III are forthcoming.

More FastCGI: Apache 2.4, PHP-FPM, PSGI, and hot deployment

Driven by mod_fcgid's failings at graceful restart and preforking, I've been looking hard for alternatives.

tl;dr

External FCGI is best done via mod_fastcgi on Apache prior to 2.4, or mod_proxy_fcgi on 2.4.  mod_proxy in general is super awesome on 2.4.

php-fpm ships with an init script and reloads gracefully via SIGUSR2, so that's all you really need there.

For Perl, gracefully restarting an FCGI app is difficult, so the better approach for PSGI apps is to run them in an HTTP server with Server::Starter support (e.g. Starman or others).

TLS: all those DH modes and PFS

Advice is easy to come by, explanation less so.  What are all these acronyms, and what makes them secure, or not?

Theming Drupal 7: Block vs Node vs Field et al

What is the difference among pages, regions, nodes, blocks, and fields?

There's one page that may contain multiple regions.  Regions are containers for nodes and/or blocks.  Multiple nodes can appear within a region if the URL being accessed is a listing view, in which case the nodes will be rendered in teaser form with "Read More" links leading to the full view.  Multiple blocks can also appear in a region.

Blocks seem to correspond to 'widgets' in other systems: chunks of content that can be dropped into the sidebar and remain fairly static on many pages.  For instance, Drupal's search box is a block, and it lands in the sidebar_first region of both default node types.  Blocks do not have fields.

Nodes, on the other hand, correspond to the content that actually gets added to the CMS.  Blog posts, static pages, whatever.  Nodes get URLs assigned to them, either through the URL aliasing features, or the default node/4 style path.  (Or through any SEO friendly URL generation modules you may have installed.)  Nodes have a type and node types have fields.  Fields receive values per-node that are displayed on the node type.

Everything related to a node is rendered inside a single container, corresponding to the page content variable.  Overall, the URL, node with its type and fields, and page content variable are all one interrelated thing when viewing a single node.  Pages and regions are related to the theme as a whole.  Blocks are strongly related to a theme, but also customizable based on node types through the block's Configure link.

Blocks, nodes, and fields are fairly customizable and appear in the admin interface under the "Structure" menu item.  Controlling blocks is done with the Blocks item in the menu (straightforward enough); controlling nodes is done through Content Types.

I should probably note that the paths through the admin interface given are for the default setup where Bartik is the main theme and Seven is the administration overlay theme.  Those paths are not guaranteed for other themes, since themes are PHP and can do nearly anything they want.

If you want "a content area" with several "pieces" to it, the path of least resistance is to construct node.tpl.php with the contents of that content area inside, using fields to display each individual "piece" desired.  Then in your admin interface, establish the fields so they show up when editing the page.

To make that clearer, let's say MyCorp wants a video on their front page with a blurb to the side, a graphical separator, and another couple paragraphs below.  I could make a mycorp_video content type, and add two fields (field_video_embed and field_video_blurb), then create a node--mycorp-video.tpl.php file with the container divs, central bar, and calls to <?php render($content['field_video_embed']); ?> inside their respective containers.  Then I could leave the couple of paragraphs as "body" content and print that below the separator.  Once the template is ready, the node type can be created in the admin interface, the node actually added, and finally set to be the front page of the site.

Controlling something outside of the content area based on the content (node) type is not possible by default, but can be done with an override in the template.php file for the theme:

function themeName_preprocess_page(&$vars, $hook) {
  if (isset($vars['node'])) {
    /* If the node type is "blog_madness" the template suggestion will be "page--blog-madness.tpl.php". */
    $vars['theme_hook_suggestions'][] = 'page__'. $vars['node']->type;
  }
}

The above code was posted by JamieR at drupal.org/node/1089656.  By default, drupal's page renderer only knows about specific nodes (like page--node--4.tpl.php) and not content types (aka node types) in general, which is what this override adds.

A second approach is to use the CCK Blocks module to convert fields into blocks.  This allows them to appear on the block layout and be placed in regions in spite of being node-specific.  The blocks are then made visible in region templates with a  cck_blocks_field prefix, for instance cck_blocks_field_video_embed for a video_embed field.

The latter approach is actually the one I ended up taking.  I needed to handle several optional areas in various combinations.  Instead of a big list of node types and duplication of the markup for any fields shared between types, I have two basic node types and regions handle sharing the markup and displaying the available fields (or nothing, when no fields are set.)

mod_fcgid and graceful restarts

I see plenty of this in my logs when the server needs reloaded to pick up fresh Perl:

(43)Identifier removed: mod_fcgid: can't lock process table in pid 3218

tl;dr: this appears to be harmless in practice.

The leading portion corresponds to EIDRM (see errno(3)) which comes back out of pthread_mutex_lock and cheerfully gets logged as the failure code of proctable_lock_internal.  The proctable is in turn locked during request handling.

My best guess for the order of events is that the Apache parent receives a graceful restart, unloads and reloads mod_fcgid, which destroys the mutex as a side effect.  After old-generation children tie up their requests, they try to notify their parent that they're available again, only to discover that the mutex is gone.  The child then exits, but it doesn't hurt any clients because they've already been served at this point.

This problem is not fixable in Apache 2.2 because there aren't any hooks for graceful-restart.  It just unloads DSOs without warning, and their first clue anything happened is that they start receiving config events.  By then, the mutex and process table are gone, so the newly-loaded master can't communicate with old-generation children.  Someone did make an attempt to fix this for 2.4 (along with modifying mod_cgid to test their infrastructure) but AFAICT nobody has made this available in mod_fcgid for 2.4 yet.

Perl non-CGI: The Missing Overview

After living in mod_php and Perl CGI for far too long, it was time to look at reworking our application to fit something else.  Although we had mod_perl installed and doing little more than wasting resources, I didn’t want to bind the app tightly to mod_perl in the way it was already tightly bound to CGI.  That meant surveying the landscape and trying to understand what modern Perl web development actually looks like.

But first, some history!

Autoflush vs. Scope (CGI::Session)

CGI::Session writes its session data to disk when DESTROY is called, possibly during global destruction, but the order of global destruction is non-deterministic.  It will generally work when CGI::Session is writing to files, since it doesn't depend on anything else to do that, but using other storage like memcached or a database, the connection to storage may have been cleaned up before CGI::Session can use it.  Then your session data is mysteriously lost, because it was never saved to begin with.

Another possible interaction between object lifetime occurs when there are multiple CGI::Session objects: unless both have identical data, whichever one gets destroyed last, wins.  At one point, I added an END {} block to a file which had my $session declared.  All of a sudden, that END block kept $session alive until global destruction and the other CGI::Session instance, into which I recorded that a user was in fact logged in, now flushed first.  Because the logged-in state was then overwritten by the session visible from the END block (even though the block itself never used it), nobody could log in!

Yet another problem happened when I pulled $session out of that code and stored it in a package.  The END block had finished its purpose and been deleted, so moving $session to a package once again extended its life to global destruction: a package variable stays around forever, because the package itself is a global resource.  However, since the login path had flush() calls carefully placed on it, what broke this time was logout.  The delete() call couldn't take effect because the storage was gone by the time the session was cleaned up.

Why nobody uses SRP?

Aside from it being patented in part, the security goal of SRP doesn't quite fit the way we use the Internet these days: it uses a procedure similar to Diffie-Hellman to establish a secure channel based on the username presented.  Meanwhile, we have a standard for anonymous secure channels (TLS) over which we can exchange credentials without further crypto*, and using HTML forms means not being beholden to browser UI, such as HTTP Authorization's ugly modal dialogs with no logout feature.


* Although it would be nice to be able do <input type="password" hashmode="pbkdf2;some-salt" ...> to enable the server to store something other than a cleartext password, without all the dangers of trying to do crypto in javascript.

Bonus chatter: Someone once asked why I would use Digest auth even over TLS.  "In case TLS is broken" didn't appease him, but since then, we've seen high-profile failures like DigiNotar and Comodo, and attacks like BEAST.

DynamoDB Performance

Things I learned this past week:

1. AWS Signature V4 no longer requires temporary credentials.  If you aren’t caching your tokens, this can give you a nice speedup because it cuts IAM/STS out of the connection sequence.
2. AWS service endpoints are SSL.  If you make a lot of fresh connections, you may pay a lot of overhead per connection.
3. Net::Amazon::DynamoDB and CGI are terrible things to mix.

Read on for details.

Private Streaming with CloudFront: A Guide

Update, 1 Oct 2012: This post is largely obsolete, as Amazon recently added private streaming support to the CloudFront section of the AWS Console.  The original post follows.

I'll just assume you're aware of the IaaS offering known as Amazon Web Services, AWS.  CloudFront is a CDN in the AWS micropayments-as-you-go style, which offers the ability to serve non-public content stored in S3.  This is a compendium of the things I learned setting up a private streaming distribution for use with PHP.

This is going to be fairly low-level, since I like to drink deeply of the systems I'm working with.  I don't think AWS works smoothly enough yet that you can put the API on the "it's magic" side of the line.

Character Sets: Get PHP, Perl, MySQL, and Unicode to Play Together

This post is a companion to Perl and Unicode in Brief, an attempt to cover the same ground more concisely.

This is an extended remix of my recent post on the subject, only less of a rambling story and more focused.  Again, I'll start with some background definitions.

I'll also assume that you're going to make everything UTF-8, because as a US-centric American who has the luxury of using English, that's what makes the most sense for my systems.  However, if you understand everything I wrote, it should not be difficult to make everything UTF-16 or any other encoding you desire.

Character Sets, Encodings, MySQL, and your data

This post is a companion to Perl and Unicode in Brief, an attempt to cover similar ground more concisely.  And this post is a revised version of the one you're currently reading.

I'm currently moving data from a (relatively old now) MySQL 5.0 server into Amazon RDS.  I've been here before, when I was moving data from MySQL 4.x into 5.0 and mangling character sets.  This time, I want to make 100% sure everything comes across with maximum fidelity, and also get the character encoding as stored to be labeled correctly in MySQL.

First, a quick definition or two:

• Character Set: a specific table to translate between characters and numbers.  Example: ASCII defines characters for numbers 0-127; "A" is 65.  This can also be described as "a set of characters, and their corresponding representation inside the computer."
• Character Encoding: a means of "packing" numbers from the character set into a container.  Example: UTF-8.  The Unicode character 0x2013 becomes 0xE2,80,99. The "E" signifies "Part 1 of 3", and part of the remaining bytes simply indicate "Continued"; the 0x2013 is then divided up to fit in the parts of the bytes that aren't indicating their "Part 1" or "Continued" status.  In the specific case of UTF-8, the encoding is designed so that the ASCII range 0-127 (0x00-7F) is encoded without change: a leading 0-7 means "Part 1 of 1".
• 8-bit character encoding: In older, simpler days, character sets defined only as many characters as could fit in 8 bits, and defined the encoding as simply the numbers.  Character number 181 would encode as a byte (8 bits) with value 181.
• A character encoding implies the associated character set, because the encoding defines how numbers in its character set become individual bytes.  How characters in other sets would be encoded is left undefined and basically impossible.

This last point is why MySQL lets you set "character sets" to UTF-8, though the latter is an encoding.

sapphirepaw's Introduction to Pointers, version 2

I programmed in assembly for some time, using pointers without understanding what they were, or that they were called pointers.  When I finally got to learning C, the pointer syntax was downright inscrutable, but when I got it, suddenly all of C and all of assembler laid clear before me, all at once.  It was a beautiful thing.

I was reminded about this while reading this post from HN.  It inspired me to try explaining pointers from the opposite direction.  Instead of trying to teach pointers via C syntax, let me try to start with pointers outside of programming, then discuss them in relation to C and PHP.

Variable scope, require, and use

I ran into some interesting problems in Perl, which invoked more learning around the require/use mechanisms and how constants are interpreted.  In this post, I'll lay out some general terms about variable scoping, such as lexical scope, dynamic scope, the differences between them, and how they all interact in Perl.  And then I'll cover require and use with that foundation in place.

If you've been wondering about lexicals or closures, this is your post.  I've tried to lay things out more or less from basic principles, despite the verbosity of the approach, because this has taken me years to understand.  I started programming with Perl in 2000 and still learned a bit more about it today.  Yes, it's 2011 now.  Hopefully, you can read this and get it in less time.