Making the most of APFS and xhyve

I'm running macOS Mojave, and using it to host virtual machines via xhyve.  There are some neat tricks one can use to conserve disk space while giving plenty of room to your VMs.

First, a side-thread:  I read about an issue someone had when APFS first came out.  I don't know for sure if it's still an issue, and haven't been able to find the article for this post.  The gist was, if the user fills up the root volume, there's no way to delete any files in recovery mode when there are no free extents in APFS.  The workaround involves creating a throw-away APFS volume to reserve some free space.  Then, if root ever fills up, boot into recovery, delete the throw-away volume to free up a few extents, then mount root and clean it up as needed.

When creating an Ubuntu VM from the install ISO, I use the technique explained at https://gist.github.com/mowings/f7e348262d61eebf7b83754d3e028f6c.  One has to extract the installer's initramfs and kernel image to pass to xhyve.  The often cited way is to copy the iso to a temporary file and zero out the first couple of sectors before mounting that to extract the files.  There is a way to do that using APFS's COW so you're not taking twice the disk space for the ISO.

I combine these two efforts by creating my throw-away APFS volume (with at least 500MB reserved), and storing the ISO to that, as I can always download it again if I accidentally fill up root.  Then, to make a COW copy of the ISO, duplicate it using Finder (or cp -c at bash prompt, which uses the clonefile() call rather than read/write of the file contents).  You'll see that duplicating this large file on the small volume does not increase the amount of space used on that volume!  Then just overwrite the first couple of blocks with this command:

dd if=/dev/zero of=/Volumes/DeleteIfRootFull/tmp.iso bs=2048 count=1 conv=notrunc

Now you you can keep that temporary ISO around for the next OS install, or even script the mountable ISO creation and boot right from the files on the mounted filesystem.  Note that the conv=notrunc argument is important, as that is what keeps the remaining file intact while we overwrite the first 2KiB with zeroes.

You can use a similar trick with dd to create your VM disk as a sparse file.  Even if you have less than 16GB free, you can create a 16GB or larger drive to install your OS into by seeking within the output file to just the final block.  For example, to calculate your seek size for a 16GiB disk, run this command:

echo '16 1048576*1-p' | dc

I got 16777215, which I use in the following command:

dd if=/dev/zero of=hdd1.img bs=1024 count=1 seek=16777215

You'll see the resulting file is exactly 2^34 bytes (16GiB), but if you go to the folder in Finder and view the file's info, you'll see its size as "17,179,869,184 bytes (4 KB on disk)".  Note the difference in listed size and size "on disk" indicates that this is a sparse file (only the extents that have been written to are actually allocated "on disk").  It's 4k instead of 1K because flash writes a minimum of 4K at once, I believe.

Be warned that, as you write to this file, you will be increasing the space used on your APFS volume, and your xhyve VM could try to write up to the file's given size, even if you don't have that much free space available.  So, if you don't monitor your free space, you could end up needing to delete that extra APFS volume sooner than you had expected...