Debian/Ubuntu init breakpoints (How to run zerofree and power off correctly afterwards)

Editor's Note [2020-11-21]: If you just want to run zerofree, see my updated post: Using zerofree to reduce Ubuntu/Debian OVA file size.

Most of the Debian information mentions break=init and then says “see the /init source for more options.”  Fortunately, I can use break=init and then dig through /init myself.

Here’s the possible list for Ubuntu 14.04 Trusty Tahr:

• top: after parsing commandline and setting up most variables, prior to the /scripts/init-top script
• modules: before loading modules and possibly setting up netconsole
• premount (default for break without a parameter): prior to the /scripts/init-premount script
• mount: prior to /scripts/${BOOT} (which is set to "local" in my VM but comments indicate it can be otherwise under casper)
• mountroot: prior to actually mounting the root filesystem
• bottom: prior to running /scripts/init-bottom and mounting virtual filesystems within the root
• init: prior to unsetting all initramfs-related variables and invoking the real init

break=init drops to a busybox shell with the root device filesystem mounted at $rootmnt which is /root on my system.  The virtual filesystems like /proc are also mounted under there, in the device’s root directory, not the initramfs’ root.  This is actually the state I want to be in, and digging out that list of break options was entirely irrelevant.  My apologies for the unintended shaggy-dog story.

From here, to run zerofree /dev/sda1 (your device name may vary) and shut down the system correctly afterwards:

1. Boot with break=init
2. chroot $rootmnt /bin/sh
3. zerofree /dev/sda1
4. exit
5. sync
6. poweroff

I just need to chroot into the disk first, so that zerofree can read the mount table from /proc/mounts (it doesn’t run on a writable filesystem.)  Then to clean up, I return to the initramfs where I can use its working poweroff command.

I used to use init=/bin/sh to get a shell inside the root mount, but then I didn't have a way to shut down the system cleanly.  In the image, shutdown wants to talk to init via dbus, neither of which are actually running, and it simply delivers a cryptic message like “shutdown: Unable to shutdown system”.  Recovery or single-user modes didn't work because the filesystem was already mounted read-write and enough services were up to prevent remounting it as read-only.

CVE-2014-0224

Apparently in spite of automatically updating OpenSSL to the newest version, my server was just sitting around vulnerable for a while. unattended-upgrades, you had one job, to get security updates applied without me pushing buttons manually.

I guess I've got to write a cron job to reload all my network-facing services daily, in case some dependencies are ever updated again. Because for all its strong points, the package system doesn't care if your packages are actually in use.

On the Windows XP EOL

I discovered that some problems people had connecting to our shiny new SHA-256 certificate in the wake of Heartbleed were not caused by “Windows XP” per se, but by the lack of Service Pack 3 on those systems.

SP3 itself was released in 2008, meaning that SP2 had a two-year “wind down” until it stopped receiving support in 2010. That means everyone who had problems with our certificate were:

1. Using an OS that has been obsoleted by three further OS versions if you include Windows 8.
2. Using an OS that had reached its actual end-of-life after ample warning and extensions from Microsoft.
3. Using a version of that OS which had been unpatched for nearly four years.

Combine the latter two, and you have people running an OS who never installed the SP3 update during its entire six-year support lifetime, which is longer than Windows 7 had been available.

In light of this, I can see why businesses haven’t been too worried about the end-of-life for Windows XP. It’s clear that those affected are not running SP3 on those systems, meaning they were already four years into their own unpatched period.

And if they “just happen” to get viruses and need cleanup, that just seems to be part of “having computers in the business.” Even if the machines were up-to-date, there would still be a few 0-days and plenty of user-initiated malware afflicting them. There’s little observable benefit to upgrading in that case… so little, in fact, that the business has opted not to take any steps toward it in half a decade.

GCC options on EC2

AWS second-generation instances (such as my favorite, m3.medium) claim to launch with Xeon E5-2670 v2 processors, but sometimes the v1 variant.  These chips are Ivy Bridge and Sandy Bridge, respectively.  AFAICT, this means that AMIs with code compiled for Sandy Bridge should run on any second-generation instance, but may not necessarily run on previous-generation instances.  What little I can find online about previous-generation instances shows a mix of 90nm Opteron parts (having only up to SSE3) and Intel parts from Penryn-based Harpertown up through Westmere.

GCC changed their -march options in the recently-released version 4.9.  GCC 4.8, which notably shipped in Ubuntu 14.04 (Trusty Tahr), used some inscrutable aliases; the equivalent to 4.9's sandybridge is corei7-avx for instance.  Here's a table of the subset that's most interesting to me:

GCC 4.8	GCC 4.9
corei7	nehalem
–	westmere
corei7-avx	sandybridge
corei7-avx-i	ivybridge
corei7-avx2	haswell

Broadwell and Westmere are not explicitly supported in the older release.  Based on its definition in the gcc-4.9 sources, I believe the equivalent set of flags for Westmere in gcc 4.8 would be -march=corei7 -maes -mpclmul.  And naturally, the corei7-avx2 option for Haswell would be the best for targeting Broadwell.

Fake Speed with Bitmaps

It seems that Apple has been working very hard on “hiding latency” by showing a picture of an app’s last state while the app reloads. Apps may have to opt-in to this behavior, but it seems fairly prevalent on iOS and now OS X. Specifically, I’ve noticed it on iOS in some password-locked apps, in Music Studio, and on OS X, in the App Store.

After a firmware update, logging in again “re-opens” the App Store in precisely the state it was left after reboot. Even the update being installed is still shown as available, with its “RESTART” button grayed out. That perfectly represents how the app looked… before rebooting. But it rather quickly reverts to “Cannot contact the App Store; an Internet connection is required” since the wireless isn’t up yet.

Why it needs to talk to the App Store when it could have cached the data less than five minutes ago is beyond me, but never mind that.

The point is, while OS X is busy displaying a stale screenshot, any UI interaction will be lost. Because it’s not a UI at all, it’s a highly accurate view of what it could look like. OS X remains awfully confident about the number of updates it has, even though it can’t connect to the Internet and isn’t even loaded.

This is especially noticeable on iOS where the screen doesn’t visibly change state between the picture being shown and the real UI replacing it. I have a password-protected app that always looks like an “Enter your password” screen, complete with buttons in pixel-perfect position, but if I interact with it immediately after switching to it, my touches are dropped. Instead of 1234, it only registers 234 (or even 34). Then I wait for it to check the PIN and do the unlock animation, then realize that it won’t, and finally delete each digit by mashing Delete before entering the PIN for a second time.

1234 is not my real PIN, of course; that’s the PIN an idiot would have on his luggage. Or, you know, every Bluetooth device ever with a baked-in PIN code. (sigh)

It’s also really annoying when an app takes a while to (re)load because it’s legitimately large, like Music Studio and its collection of instruments for the active tracks. The UI is completely unresponsive for several seconds, until suddenly everything happens at once, badly. The Keyboard screen often gets key stuck down, making me tap it again to get it to release. Or knowing that, I avoid touching it for a while and have to guess at when it will be responsive, not knowing how much time I’ve wasted in over-shooting that point.

In the end, using static pictures of an app for latency hiding seems like a poor user interface—because the end of the latency period is also hidden, it encourages users to try interacting early, when the result is guaranteed not to work. But instead of showing the user that, it silently fails. I’d much prefer the older “rough outlines” splash screens than the literal screenshots of late; the “ready” transition is obvious with those, when the real UI shows up.

It actually surprises me that Apple would even release UI like this, because it’s kind of frustrating to clearly have an app on-screen that’s not reacting to me. Then again, with the train wreck that was QuickTime 4.0 Player, perhaps I shouldn’t be too surprised. (Yes, that was 15 years ago or something. No, the internet will never forget.)

Musing on Iterator Design

PHP’s standard library (not the standard set of extensions, but the thing known as the “SPL”) includes a set of iterators with a kind of curious structure.

At the root is Traversable, which is technically an empty interface, except that there’s some C-level magic to enable C-level things to declare themselves Traversable. This enables them to be used in foreach loops even though they don’t support any Iterator method.

Iterator exists as the basic interface for regular PHP code to implement iteration, internally. Notably, while Traversable is a once-only iterator, Iterator is not. rewind() is part of its interface.

For classes that wrap a collection, there’s an IteratorAggregate interface, which lets them return any Traversable to be used in their place. And who could forget ArrayObject, since a plain array (in spite of being iterable in the engine) is not actually a valid return from getIterator()?

There are some interesting compositions, like AppendIterator, which can take multiple Iterators and return each of their results in sequence. AppendIterator is a concrete class that implements OuterIterator, meaning that it not only provides regular iteration but the world can ask it for its inner iterator.

There’s the curiously named IteratorIterator, which I never appreciated until today, when I tried to pass a Traversable to AppendIterator. For no apparent reason, AppendIterator can only append Iterators, not Traversables. But IteratorIterator implements Iterator and can be constructed on Traversable, thus “upgrading” them.

Long ago discovered and just now remembered, PHP provides DirectoryIterator and RecursiveDirectoryIterator—but to make the latter actually recurse if used in a foreach loop, it needs to be wrapped in RecursiveIteratorIterator, because the caller of a plain RecursiveIterator must decide whether to recurse, or not, at an element with children.

The designer in me thinks this is all a bit weird. Because PHP defined the basic Iterator as rewind-able, a bunch of stuff that isn’t rewind-able (including IteratorAggregate) can’t be used in places that expect a base Iterator. Like AppendIterator. That in turn exposes a curious lack of lazy iterators. When I wrap an IteratorAggregate in an IteratorIterator, getIterator() is called right then, before the result can even be added to an AppendIterator.

So if I want to fake a SQL union operation by executing multiple database queries in sequence, and I don’t want to call the database before the AppendIterator reaches that point, there’s no simple way to do it with SPL’s built-in tools.

I can build a full Iterator that doesn’t query until current() is actually called, by implementing a handful of functions, each of which must carefully check the current iteration state before returning a value. These would satisfy the native AppendIterator, but be lazy themselves.

Or, I can build a full LazyAppendIterator that accepts Traversable and doesn’t try to resolve anything until the outer iteration actually reaches them. Then I can add IteratorAggregates that perform the query in getIterator(), since I won’t be calling that right away.

Those two choices amount to the exact same amount of work. In each case, I need to build a lazy iterator that tracks the “current iteration state” and launches the expensive call only when starting a new iterable/iteration sequence.

In other languages I’ve used, the basic one-pass/forward-only iterator is the type accepted by all the compositing types, and everyone must live with the idea that the iterators in use may not be seekable/replayable. In PHP, I have to live with the idea that the not all iterable things may work with all Iterators, which seems supremely odd.

P.S. They didn't pack any RegexFilterIterator in the box, either?

Hidden Interfaces

I fly around my world with lots of keyboard shortcuts.  The hot-corner UI stuff in Windows 8 doesn't really bother me, except for that part where it's hard to hit in a windowed virtual machine... but they provided some new shortcuts like Win+C to get there, or I can run it full screen.


But there's something interesting about this: my parents have a much worse time using Windows 8.x than I do, and the gap is greater than it was in the XP/7 days.  And it occurs to me that I use a lot of 'secret' interfaces that are invisible to them.