Using zerofree to reduce Ubuntu 26.04 (resolute) / Debian 14 (forky) OVA size

This is an update of an older post, because the instructions there stopped working.

Ubuntu 26.04 is the first LTS after Canonical switched from initramfs-tools to dracut in 25.10.  The break=init parameter doesn’t work anymore on 26.04; the system simply performs a normal boot in that case.  I believe that Debian plans to do the same in their next release, 14 (forky).

Dracut builds an initrd that responds to a different parameter, with a different set of break points.  Full information about rd.break is located in the dracut.cmdline man page, but for our purposes, we’ll use rd.break=pre-mount to get a shell before the root filesystem is mounted.  If we build zerofree into the initrd temporarily, then everything we need will be available in that environment.

In full, this process looks like this:

1. From the prepared system, run: sudo dracut --force --install zerofree
2. Reboot in graphical mode, or shut down and do a graphical-mode restart, as applicable.
3. Interrupt the boot loader to add rd.break=pre-mount to the kernel command line.  (This general process is more fully described in the older post.)
4. Boot.
5. Press Enter (or enter root password, as required) to get to the initramfs shell.
6. Run zerofree -v [DEVICE] at the initramfs prompt (replace [DEVICE] with the actual device; this operation shouldn’t damage anything, but for reliability, double or triple-check the device name!)
7. Run poweroff when it’s done.

The machine is now prepared for export.

My Issues with Libvirt / Why I Kept VirtualBox

At work, we use VirtualBox to distribute and run development machines.  The primary reasons for this are:

1. It is free (gratis), at least the portions we require
2. It has import/export

However, it isn’t developed in the open, and it has a worrying tendency to print sanitizer warnings on the console when I shut down my laptop.

Can I replace it with kvm/libvirt/virt-manager?  Let’s try!

What matters? Experimenting with Ubuntu Desktop VM boot time

It seemed slow to me that the "minimal" installation of Ubuntu Desktop 20.10 in a VirtualBox VM takes 37 seconds to boot up, so I ran some experiments.

The host system is a Late 2012 iMac running macOS Catalina.  It has 24 GB of RAM (2x4 GB + 2x8 GB) installed, and the CPU is the Intel Core i5-3470 (Ivy Bridge, 4 cores, 4 threads, 3.2 GHz base and 3.6 GHz max turbo.)  In all cases, the guests are running with VirtualBox Ubuntu Linux 64-bit defaults, which is to say, one ACHI SATA drive that is dynamically sized.  The backing store is on an APFS-formatted Fusion Drive.

The basic "37 seconds" number was timed with a stopwatch, from the point where the Oracle VM screen is replaced with black, to the point where the wallpaper is displayed.  Auto-login is enabled for consistency in this test.  This number should be considered ±1 second, since I took few samples of each system.  I'm looking for huge improvements.

So what are the results?

• Baseline: 1 core, ext4, linux-generic, elevator=mq-deadline.  37 seconds.
• XFS: 1 core, xfs root, linux-generic, elevator=mq-deadline.  37 seconds.
• linux-virtual: 1 core, xfs root, linux-virtual, elevator=mq-deadline. 37 seconds.
• No-op scheduler: 1 core, xfs root, linux-virtual, elevator=noop. 37 seconds.
• Quad core: 4 cores, xfs root, linux-virtual, elevator=noop. 27 seconds!

The GUI status icons showed mostly disk access, with some heavy CPU usage for the last part of the boot, but only adding CPU resources made an appreciable difference.  I also tried the linux-kvm kernel, but it doesn't display any graphics or even any text on the console, so that was a complete failure.

(I've tried optimizing boot times in the past with systemd-analyze critical-chain and systemd-analyze blame, but it mostly tells me what random thing was starved for resources that boot, instead of consistently pointing to one thing causing a lot of delay.  Also, it has a habit of saying "boot complete in seven seconds" or whatever as soon as it has decided to start GDM, leaving a lot of time unaccounted for.  So I didn't bother on these tests.)

Correction: it was Ubuntu 20.10, not 20.04 LTS.  The post has been updated accordingly.

Using zerofree to reduce Ubuntu/Debian OVA file size

Editor's note [2020-11-21]: this is an update to a 2014 post on the same topic.  Since then, the mount command is also required.

When exporting an OVA file from VirtualBox's "Export Appliance" feature, any disk block that isn't filled with 0 has to be included in the resulting file.  Normally, an OS doesn't create those blocks as it works; when a file is deleted, the space is marked as free, but retains its original contents until it is re-allocated for another file.  If there are a lot of files deleted, a significant amount of non-empty "empty" space can accumulate.

Running the zerofree program writes data of value 0 over all space marked as free, allowing that free space to be excluded from the OVA file again.

The following applies to Ubuntu 18.04 (bionic) and 20.04 (focal) at least; I forget exactly how far back it goes, but the way to get at a command line changed at some point, and there was a period where poweroff didn't actually work in the pre-boot environment.

Regardless, here's how to run zerofree on Debian, Ubuntu, or derivatives.

vmconnect

I wrote a script called _vmconnect that either simply logs into a VirtualBox guest, or starts it headlessly first and then logs in.  With _vmconnect set as the command for an iTerm2 profile, I can open the session without caring whether the machine is running or not.

The full script became a bit unwieldy for a blog post (and I didn't want to maintain it forever within Blogger), so I published it to github.  You'll find it in my bin repository: https://github.com/sapphirecat/bin

Enjoy!

Update: _vmconnect version 1.0.3 fixes a possible data loss: if the VM was booted by _vmconnect, terminating the ssh connection it logged in with would also abort the VM.  You SHOULD pull if you are using _vmconnect 1.0.2 or prior (check your _vmconnect --version output!)

Brief notes on Packer

packer.io says they build identical images for multiple platforms such as EC2, VirtualBox, VMWare, and others (with even more on the way via plugins).  However, it's more about customizing a machine already running than it is about fabricating a new machine out of whole cloth and distributing the result to disparate services.  That is, the AMI builder starts from another AMI; the VirtualBox builder starts from an installer ISO (with commands to make the install unattended) or an OVF/OVA export and customizes that to produce an output OVF/OVA export.  This is, as I understand it, in contrast to Vagrant or Docker images, which build one image that runs on many systems.

Packer is about customizing existing machines that already exist in some form, then building the result as a new image for the same system.  If you're really interested in taking "one base image" to many environments, then Docker is probably the better choice there.  It's built on Linux Containers, where the virtualization happens inside the host kernel instead of a dedicated hypervisor, which makes it possible to run a bit-for-bit identical container both  inside EC2 (even PV) and inside VirtualBox.

In another contrast to vagrant, packer supports a shell provisioner that lets you upload a shell script and run wild.  Last I knew, which is probably woefully out of date, vagrant supported puppet by default and chef-solo as an option.  Since I happen to have an existing build system constructed around shell scripts for provisioning (with a dash of perl or php for the trickier parts), albeit specifically for EC2 instances, packer might suit my codebase and philosophy a little better.

Put another way, and beating the dead horse a bit more, Vagrant and Docker are about creating One True Image and later instantiating that single image on different underlying services (including EC2 and VirtualBox).  Packer is about coordinating the construction of native images for those services; it's about making predictable, repeatable changes to a running instance in the service and exporting the result for later use.

Unrelated to the above, packer is written in go 1.2 and distributes binaries for various platforms.  Docker is also written in go (but I'm not sure of the version target), leaving Vagrant in Ruby as the odd man out.  Having been subjected to rvm to install octopress at one point, I'm much more amenable to smaller dependency footprints... like static binaries produced by Go.  (Then again, I heard later that everyone hates rvm.)

Update 14 Feb 2014: I thought of one more useful contrast among the projects.  Docker and Packer are aimed toward building immutable, self-contained images.  Vagrant is focused on setting up (repeatable) development environments--particularly, in how it includes only the VirtualBox provider by default, and uses shared folders to persist changes as you develop.

Working from Home

Since I've radically changed my work VM configuration, I needed to change my home PC in order to connect properly to the new setup.

Before we dive in, a brief review of the old setup: the host at work, which we will call `audia`, ran a VirtualBox guest named `dev`.  dev had its own set of configs and configuration knobs so that I could write 127.0.0.1 dev.audia.pc dev into /etc/hosts, point Firefox to dev:10080, and code running on dev would be able to recognize "this is development," not require TLS, and know that http://dev:10080/ is the URI root of the app.  I configured VirtualBox's port forwarding to listen on 0.0.0.0 from audia, and when I connected to the VPN, my home computer (call it `starlet`) had its own /etc/hosts listing that used the internal IP of audia for dev.  The same URI from home would then get connected via port-forwarding to the dev VM.

This all changed when I wanted to test a SAN certificate and some redirection rules.  I didn't want certificate mismatch errors in Firefox, and I needed dev to think it was the production server so that it would use the production ruleset.  That became the underlying justification for devproxy.  After that point, dev had the production certificates, hostnames, and rules.  It had a host-only network added, which audia now used, and I took out the port forward.  (This was the problem behind getting Lost in the Complexity.)

But without the port forward, starlet can't reach dev anymore, only audia.  Of course, I came up with a solution:

(Despite what may be implied, the VPN isn't connected to starlet.  I just didn't feel like redrawing the whole tube.)

devproxy

Today, I'm pleased to announce the release of devproxy!  It's an HTTP proxy server written in Go (atop github.com/elazarl/goproxy) for QA/testing purposes.  It intercepts requests directed to your production servers and transparently connects them to a staging server of your choice—including HTTPS (CONNECT) redirection.

This lets your staging server perfectly mirror production, including the hostnames and certificates in use, without needing to elevate permissions to edit /etc/hosts every time you want to switch between production and staging.  Instead, you can switch the proxy on/off in your browser; I use FoxyProxy.

I'd like to thank Elazar not only for writing goproxy (it made my life a lot easier) but also for modifying it to support what I needed to do in devproxy.  I'd also like to thank my boss for letting me release this as open source.

Tor for VirtualBox Guests (idea only; no code)

To give a guest only one network card with host-only networking, yet still let it access the Internet, we can let it connect to an HTTP proxy running on the host.  If this proxy is polipo, we can configure it to connect to tor’s SOCKS server as its upstream:

Then what happens?  Any and all Internet traffic from the guest VM is delivered via Tor.  Since the guest doesn't have Internet access of its own, any software which doesn't cooperate with the proxy cannot communicate.  Although malware on a compromised guest could still exfiltrate data, it hides the host's true external IP address from the malware.  (Assuming, dangerously, no security bugs in polipo nor VirtualBox that would allow a compromise of the host.)

I said that first, but maybe not very clearly, on twitter.

But… given an appropriate proxy, traffic can be forwarded over any transport.  A proxy could accept data from the guest and transmit it via VPN.  On the other hand, building a VPN client into VirtualBox to offer a VPN network type would let a client connect to a VPN without necessarily allowing other host processes access to it, nor requiring the VPN to be mediated by an additional (dual-homed) guest.

Polipo doesn't have to use tor as a backend, either; it's also perfectly capable of forwarding using ssh's SOCKS proxy.  (This is known as "dynamic tunnel mode" in some clients.)  Compared to the amount of software and configuration needed to set up the average VPN, ssh is just as secure and much easier to get running.

Sending traffic via proxy is an effective way to apply further modifications to the destination stream, without needing the cooperation of software connecting to the proxy.

VirtualBox Webcam Activation

Perhaps your webcam turns on when starting a guest VM.  (thread 1, thread 2, thread 3, none with helpful info.  There's also at least one bug for it, only two months old but still silent at time of writing.)

It turns out that this is related to audio.  If I have a VM with the audio disabled—such as one provisioned through vagrant—then it doesn't turn on the webcam.  I conclude that VBox is actually attempting to use the camera's microphone as audio input.

(I'd report this on the bug, if they had OpenID or something.  But building a whole Oracle Account just for this? Meh.)