Pointless metrics: Uptime

Back in the day when I was on Slashdot, a popular pasttime was making fun of Windows. The 9x line had a time counter that rolled over after 30-45 days (I forget and can’t dredge the real number up off the Internet now), which would crash the system. Linux could stay up, like, forever dude, because Unix is just that stable.

So I spent a while thinking that ‘high uptime’ was a good thing. I was annoyed, once upon a time, at a VPS provider that regularly rebooted my system for security upgrades approximately monthly, because they were using Virtuozzo and needed to really, seriously, and truly block privilege escalations.

About monthly. As in 30-45 days…

I thought that was bad, but nowadays, the public-facing servers my employer runs live for less than a week. Maybe a whole week if they get abnormally old before Auto Scaling gets around to culling them. And I’m cool with this!

I try to rebuild an image monthly even if “nothing” has happened, and definitely whenever upstream releases a new base image, and sometimes just because I know “major” updates were made to our repos (e.g. I just did composer update everywhere) and it’ll save some time in the self-update scripts when the image relaunches.

It turns out that the ‘uptime’ that the Slashdot crowd were so proud of was basically counterproductive. I do not want to trade security, agility, or anything else just to make that number larger. There is no benefit from it! Nothing improves solely because the kernel has been running longer, and if it does, then the kernel needs fixed to provide that improvement instantly.

And if the business is structured around one physical system that Must Stay Running, then the business on that server is crucial enough to have redundancy, backups, failovers… and regular testing of the failover by switching onto fresh hardware with fresh uptimes.

Constant-time String Comparison

I mentioned hash_equals last post.  One of the things the documentation notes is that, should the string lengths not match, hash_equals will quickly return false and effectively reveal the length difference.

It seems to be a fairly common perception that this is a problem.  Take this StackOverflow answer:

This function still has a minor problem here:

if(strlen($a) !== strlen($b)) { 
    return false;
}

It lets you use timing attacks to figure out the correct length of the password, which lets you not bother guessing any shorter or longer passwords.

I believe an implementation that doesn’t fail fast on different lengths still leaks information, though.  Most of them (i.e. every one I’ve seen, including ones I’ve written before having this insight) compare all characters through the shorter of the two strings.  If an attacker can time comparisons and control the length of one string, then when the ‘constant time’ algorithm quits taking longer for longer strings, the attacker knows their supplied string is the longer one.

Therefore, I don’t believe “fail fast on different string lengths” is something to be concerned with.  If the threat model is concerned with a timing attack, then simply moving it around the function doesn’t actually form a defense.

When Did PHP Get That?

5.6.3

• 64-bit pack codes

5.6.0

• Variadic functions with "..." (and related packing/unpacking with the ... operator)
• ** operator for exponentiation
• Constant scalar expressions (const FOO = 2*60*60)
• use const and use function make constants/functions importable from other namespaces
• htmlspecialchars() and friends use the INI default_charset setting as default (update to 5.4 behavior)
• php://input can be reused and $HTTP_RAW_POST_DATA can be disabled (and will be, in the future)
• File uploads can be >2GB
• Constant-time string comparison (aka timing-safe comparison) added as hash_equals
  • I am still curious: why hash_equals? What's the relation to hashing?

5.5.0

• Generators
• finally keyword for exception handling with try/catch
• Password hashing API: password_hash and related functions
• list() in foreach: foreach($nest as list($x, $y))
• PBKDF2 support in hash and openssl (hash_pbkdf2, openssl_pbkdf2)
• empty() accepts arbitrary expressions, not just lvalues
• array and string literal dereferencing: "php"[2] or [1,2,10][2]
• But is that useful?
• ::class syntax: namespace A\B { class C { }; echo C::class; /* A\B\C */ }
• OpCache extension added
• MySQL-nd supports sha256-based authentication, which has been available since MySQL 5.6.6
• curl share handles via curl_share_init and friends [added to this post on 2015-02-17]
• DEPRECATED: preg_replace's /e modifier, in favor of preg_replace_callback

5.4.0
Removed:

• safe_mode
• register_globals
• magic_quotes
• call-time pass-by-reference
• using TZ when guessing the default timezone
• Salsa10 and Salsa20 hash algorithms

Added:

• Traits
• Class::{expr} syntax
• Short array syntax: [2,3] for array(2,3)
• Binary literal syntax: 0b1101
• Closures can use $this in the body: function some_method () { return function () { return $this->protected_thing(); } }
• Using new objects without assigning them: (new Foo)->bar()
• CLI server
• session_status function
• header_register_callback function
• mysqli_result now implements Traversable: $r=$dbh->query(...); foreach ($r as $res) { ... }
• htmlspecialchars() and friends default to UTF-8 instead of ISO-8859-1 for character set.
• This would be changed to use the default_charset INI setting in 5.6.
• 5.4's documentation pointed out that default_charset could be used by sending an empty string, "", as the character set to htmlspecialchars—and said you didn't really want it.

5.3.0
Added:

• Namespaces
• Late static binding and associated 'static::method()' call syntax
• "Limited" goto support
• Closures aka anonymous functions, although they cannot make use of $this until 5.4
• __callStatic() and __invoke() magic methods, to support $obj($arg) call syntax
• 'nowdoc' and double-quoted 'heredoc' syntax
• $x=<<<'FOO' consumes text until FOO and does not do any interpolation / variable substitution on it
• $x=<<<"FOO" acts just like traditional $x=<<<FOO, both consuming text until FOO and doing interpolation / variable substitution as if it were a double-quoted string.
• Constants definable outside of classes with the const keyword, not just the define function
• Short ternary operator: a ?: b equivalent to a ? a : b
• $cls::$foo syntax for accessing static members of a class whose name is stored in a variable ($cls in this example)
• Chained exceptions: the constructor sprouted an extra parameter for a previous exception (generally should mean "the cause of this exception being constructed")
• DEPRECATED: many features that would be removed in 5.4.0, including register_globals, magic_quotes, and safe_mode.

Perl Futures

I’ve been doing a project with IO::Async and Future, because Amazon::DynamoDB depends on them.

One of the weirdest bits of the API that I’ve run into is that the underlying framework only weakly references Future objects, so if the client doesn’t keep its own reference, then the callbacks will never be invoked. I may not care about “the future object” itself, but I do care about the callbacks I set on it.

I ended up writing a function:

sub hold_future {
    my $f = shift;
    $f->on_ready(sub { undef $f });
    return ();
}

The sub being set does nothing but close over $f in order to create a circular reference. That keeps the Future alive until the callbacks are run. To keep the future from persisting forever, though, it destroys that reference once the future resolves.

One may safely release the future while the callbacks are running, since the resolution machinery is holding onto a strong reference ($self) at that point. Therefore, the future won’t be destroyed until all the callbacks finish.

Now, the rest of my code has clearer purpose:

sub md_delete {
    my ($cb, $id) = @_;
    hold_future(
        dynamo_delete($id)
        ->on_done(sub { $cb->(1) })
        ->on_fail(sub { $cb->(0); log_event("delete $id: $_[0]") })
    );
}

Where dynamo_delete invokes a lot of retry/exponential backoff machinery, and md_delete is a command handler for Memcached::Server.

Having worked with jQuery Deferreds, I find it odd that Future doesn’t provide a separate Future::Promise object. It just hopes that the receiver of the future doesn’t try to call creator-related functions like done.

I also find it odd that the future doesn’t just stick around itself, automatically persisting until the callback phase unless actively canceled by the receiver. Having to manually manage them feels surprisingly alien, even though I’m used to managing file handles and lexical scopes.

DynamoDB's Greatest Missing Feature

I wrote a while ago about DynamoDB in the trenches, but that was in 2012 and now 2015 is nigh.

The real pain point of DynamoDB is that everything is a hash key.  It's not sorted, at all.  Ever.

A local secondary index isn't the answer; it's a generalization of range keys, and as such, is still subordinate to the hash key of the table.  An LSI does not cover items with different hash keys.

A global secondary index isn't helpful in this regard, either.  It is essentially building and maintaining a second table for you, where its hash key is the attribute the index is on, and then it stores the key of the target element (and optionally, any other attributes from that element) from the 'parent' table.  But as a hash key, it doesn't support ordering...  The only thing that can be done with the index is a Scan operation.

DynamoDB still offers nothing else.

Some ideas that would seem, on their surface, to be perfect for DynamoDB actually don't work out well, because there's no efficient query for ordered metadata.  Most frequently, this bites me for data that expires, like sessions.  Sessions!  The use case that's so blindingly obvious, the PHP SDK includes a DynamoDB session handler.

Update (2023-01-28): DynamoDB has support for an 'expiration time' attribute these days.  We are using it for sessions now.  Due to encoding issues, our sessions are stored as binary instead of string type, but otherwise, it's compatible with the PHP SDK's data format.  Our programming languages actually use memcached but the data is stored in DynamoDB. End of update.

We store SES message data in there to correlate bounces with the responsible party.  We have a nightly job that expires the old junk, and just live with it sucking up a ton of read capacity when it has to do that.  On the bright side, unlike sessions, the website won't grind to a halt (or log people out erroneously) if the table gets wedged.

I'd like to put some rarely-changing configuration data in there (it's in SimpleDB and cached in memcache because SimpleDB had erratic query performance), but I wouldn't be able to efficiently search over it when I wanted to look up an entry for editing in the admin interface.  And really, I still want to put that in LDAP or something.

Honestly, if all I get with DynamoDB is a blob of stuff and a hash key, why not just store it as a file in S3?  How is DynamoDB even a database if it doesn't actually do anything with the data, and doesn't let you have any metadata?  Am I the crazy one, or is that the NoSQL crowd?

Related: Tim Gross on Falling In And Out Of Love with DynamoDB.

Common Sense vs. Fatal Deprecations

In Perl, use common::sense 3.73 or newer.  Older versions, including Ubuntu Trusty’s libcommon-sense-perl package at 3.72, enable fatal deprecations and will unexpectedly kill your code if you transitively depend on a module using common::sense that throws a deprecation warning.

Say, one whose maintainer received a patch and is apparently letting it rot in their issue tracker forever.

Versioning

Dan Tao asks, “In all seriousness, what’s the point of the MINOR version in semver? If the goal is dependency stability it should just be MAJOR and PATCH.”

I think I finally remember the answer. It’s set up to flag no compatibility, source compatibility, and binary compatibility. The “thing” that dictates bumping the minor shouldn’t be “features” so much as binary-incompatible changes. For example, GTK 2: if code was written against 2.4.0 using the shiny new file chooser, it would still compile against 2.12. But, once it had been compiled against 2.12, the resulting binary wouldn’t run if 2.4 was in the library path. A binary compiled against 2.4.2 would run on 2.4.0, though, because every 2.4.x release was strictly ABI compatible.

IIRC, they had a policy of forward compatibility for their ABI, so that a binary compiled against 2.4 would run on 2.12, but I don’t remember if that’s actually necessary for SemVer. Another way to look at this is, “If we bump the minor, you’ll need to recompile your own software.” Where that could be programs using a library, or plugins for a program.

I believe that’s the motivation for SemVer to include minor, but upon reflection, it doesn’t really make sense in a non-binary world. If there is no binary with baked-in assumptions about its environment, then that layer isn’t meaningful.

Also upon reflection, most of my currently-tracked projects don’t use SemVer. The AWS SDK for PHP is (in 1.x/2.x) organized as “paradigm.major.minor”, where 2.7 indicates a breaking change vs. 2.6 (DynamoDB got a new data model) but e.g. 2.6.2 added support for loading credentials from ~/.aws/credentials. PHP itself has done things like add charset to the PDO MySQL DSN in 5.3.6. When PHP added the DateTime class, it wasn’t a compatible change, but it didn’t kick the version to 6.0.0. (They were going to add it as Date, but many, many people had classes named that in the wild. They changed to DateTime so there would be less, but not zero, breakage.)

So I’ve actually come to like the AWS 2.x line, where the paradigm represents a major update to fundamental dependencies (like straight cURL and calling new on global classes to Guzzle 3, namespaces, and factory methods) and the major/minor conveys actual, useful levels of information. It makes me a bit disappointed to know they’re switching to SemVer for 3.x, now that I’ve come to understand their existing versioning scheme. If they follow on exactly as before, we’ll have SDK 4 before we know it, and the patch level is probably going to be useless.

I think for systems level code, SemVer is a useful goal to strive for. But the meta point is that a project’s version should always be useful; if minor doesn’t make sense in a language where the engine parses all the source every time it starts, then maybe that level should be dropped.

At the same time, the people that SemVer might be most helpful for don’t really use it. It doesn’t matter that libcool 1.3.18 is binary compatible with libcool 1.3.12 that shipped with the distro, because the average distro (as defined by popular usage) won’t ship the newer libcool; they’ll backport security patches that affect their active platform/configuration. Even if that means they have effectively published 1.3.18, it’ll still be named something like 1.3.12-4ubuntu3.3 in the package manager. Even a high-impact bug fix like “makes pressure sensitivity work again in all KDE/Qt apps” won’t get backported.

Distros don’t roll new updates or releases based on versions, they snapshot the ecosystem as a whole and then smash the bugs out of whatever they got. They don’t seem to use versions to fast-track “minor” updates, nor to schedule in major merges.

One last bit of versioning awkwardness, and then I’m done: versions tend to be kind of fuzzy as it is. Although Net::Amazon::DynamoDB’s git repo has some heavy updates (notably, blob type support) since its last CPAN release, the repo and the CPAN release have the same version number stored in them. When considering development packages in an open-source world, “a version” becomes a whole list of possible builds, all carrying that version number, and each potentially subject to local changes.

Given that, there’s little hope for a One True Versioning scheme, even if everyone would follow it when it was done. I suspect there’s some popularity around SemVer simply because it’s there, and it’s not so obviously broken/inappropriate that developers reject it at first glance. It probably helps that three-part versions are quite common, from the Linux kernel all the way up to packages for interpreted languages (gems or equivalents).