I rm -rf'd my own home directory

The incident

It was a quiet evening. I was poking at FreeBSD's Linux compatibility layer, which is exactly the kind of thing that should come with a supervising adult. Under /compat/linux/home sat a symlink called twisla, pointing at my real home directory. I wanted it gone.

So I typed rm -rf at it. The symlink went. So did the contents of my home directory. Every dotfile, every project, every download folder I'd been curating since roughly the Pleistocene.

Snapshots: none. Backups: none. Excuses: an entire calendar's worth, none of them applicable, because the user responsible was me and I can't even blame the network.

ZFS for the users at the back

Now, the thing about ZFS, which I normally explain slowly and loudly to people who've just emptied a shared drive, is that it's copy-on-write. When you delete a file, ZFS doesn't shred anything. It just stops pointing at the blocks, like a manager who stops returning your emails.

It also keeps a ring of 32 uberblocks: root pointers to the pool as it was at its most recent transaction groups (txgs). Point the pool at an uberblock from before the disaster, and the disaster politely never happened.

The catch, because there's always a catch: every write creates a new txg and shoves the oldest uberblock off the end of the ring. Keep using the machine, and your escape route quietly deletes itself. Rather like the user.

Step away from the keyboard

Rule one of data recovery: stop writing. So I pressed the power button, with the confidence of a man who has told a thousand users to do exactly that.

The machine, being a well-brought-up FreeBSD box, responded with a lovely, thorough, clean shutdown. It flushed. It synced. It said goodbye to every daemon individually. It wrote to the pool with the enthusiasm of an intern who's just discovered Reply All.

ZFS survives a hard power-off perfectly well. Hold the button until it dies. Don't let it say goodbye. It doesn't deserve one, and you'll see below exactly what its manners cost me.

Getting past my own security

Recovery happens from a live system, so nothing imports the pool and nothing writes to it. I flashed a FreeBSD 15.1 memstick and plugged it into the NUC, which then booted so fast it went straight past the firmware and landed on my system's GELI passphrase prompt. Twice. Congratulations to Intel on a very quick machine.

The trick is tapping F10 right after power-on for the boot menu. But even booting from the stick, the GELI prompt came back, because the memstick's own loader probes every disk and helpfully asks for the passphrase of my encrypted system pool. An empty passphrase and Enter tells it to mind its own business, and the stick boots on. Pick Live System, log in as root, and resist the urge to type anything clever.

Eight minutes of history

My home directory lives on its own pool, home, on a whole SATA disk. zdb reads its uberblocks straight off the disk without importing anything, which is the kind of non-committal behaviour I admire:

zdb -l  /dev/ada0     # label: name 'home', good start
zdb -ul /dev/ada0     # all 32 rewind points, with timestamps

The oldest rewind point: 21:31:03. The newest: 21:39:13, the moment the machine finally stopped being polite. That's the whole window. Eight minutes, and my rm had better be somewhere inside them.

Nearly half the ring went on the shutdown's farewell tour. Another minute of normal use and the rewind point I needed would have been gone, and this would be a very different, much shorter blog post consisting mainly of swearing.

Look, don't touch

The beauty of the next step is that it commits to nothing, which makes it the most management-friendly command in ZFS. A read-only import at an older txg writes precisely zero bytes. You can peek at any rewind point as often as your nerves allow.

mkdir -p /tmp/r
zpool import -d /dev/ada0 -N -f -o readonly=on -R /tmp/r -T 26943642 home
zfs mount -a
ls -la /tmp/r/<mountpoint>/twisla

And there it was. All of it. The oldest uberblock in the ring predated the rm, and my home directory sat there looking faintly smug, like a file that knows it was never really deleted, merely misplaced.

Copy first, commit second

Any sysadmin who makes an irreversible change without a copy deserves what happens next, and I'd used up my quota of deserving things for one evening. So everything went onto a freshly formatted USB disk first:

gpart destroy -F da1
gpart create -s gpt da1
gpart add -t freebsd-ufs -a 1m da1
newfs -U /dev/da1p1
mount /dev/da1p1 /tmp/dst
cp -Rpv /tmp/r/<mountpoint>/twisla /tmp/dst/
sync && umount /tmp/dst

Then the point of no return: importing the pool read-write at that same txg, which bins every transaction that came after it. Deletion included. First, zpool export announced the pool was busy, because my own shell was sitting inside it. Decades of Unix, and cd / remains the fix for a truly embarrassing number of problems.

cd /                     # the user was standing in the way
zfs unmount -a
zpool export home
zpool import -d /dev/ada0 -N -f -T 26943642 home
zpool scrub home
zpool status -v home     # no errors, no witnesses
zpool export home

Reboot. Log in. Home directory present and correct, as if the evening never happened. Which, as far as anyone else is concerned, it didn't.

Rules for users, including the one in the mirror

1. Remove symlinks with plain rm or unlink. No -r, no -f, no trailing slash, no confidence.
2. Stop writing immediately. The ring is 32 entries deep. Every second you keep going, your way back gets shorter.
3. Pull the plug. ZFS handles a hard power-off fine. A clean shutdown cost me 14 rewind points in one second.
4. Peek read-only before committing. -o readonly=on with -T costs nothing and commits nobody.
5. Copy before the irreversible bit. Always. Even when you're sure. Especially when you're sure.
6. Take snapshots. With sanoid or sysutils/zfstools on a schedule, this entire post collapses into one zfs rollback and a smug expression.

Adjusting the records

Technically, a byte-for-byte restore means nothing was lost. It was temporarily unavailable, pending operator intervention. The incident log has been amended accordingly, and the sign has been updated to reflect the official history of this department.