Everything Always Returns 0: Adventures in Adding “Volkswagen Mode” to FreeBSD

Years ago, I was working with Poudriere to set up an ARM build system on FreeBSD on x86-64. In doing so I came across the binmiscctl(8) utility. This, paired with the imgact_binmisc kernel module, were very clever and would allow you to register different interpreters or wrappers based on fields in the ELF header of a binary being executed; most commonly the architecture and instruction set. Using binmiscctl, you could register a QEMU interpreter for ARM FreeBSD binaries, and when a binary matched the conditions, it would be executed inside of QEMU with whatever options you had set, more-or-less transparently. This was particularly useful for doing cross-platform builds, but would also let the system consume binaries from different architectures if necessary.

After playing with this for a bit, I wondered; could I register an interpreter for native binaries for the host platform (FreeBSD/x86-64) itself? My first several (many) attempts failed, and I ended up doing some ugly hacking in the kernel module, but eventually it seemed to work. I then set out to write a simple wrapper. It took quite some time to make work (mostly because of my lack of C skills) but eventually it did what it was supposed to. When my wrapper was called and passed a command, it would simply spawn the process, wait for it to return, discard the return code, and return 0 instead; in the UNIX world indicating that there were no errors. I believe I had to do some other tricks in the kernel module itself or the wrapper to avoid recursion, because after all, my wrapper had the same ELF header as everything else on the system. During this process, I broke my FreeBSD installation more times than I could count, but fortunately I was working in a VM and I could revert to snapshots as needed. Eventually all of the pieces worked as intended, and I tied everything together.

So what happens when every command on a system executes successfully? Turns out, it explodes spectacularly. And not in a significantly different manner than the myriad other times I’d broken the system throughout this adventure. I really expected nothing different, but I was still elated to see the system explode for the right reasons. These days, it’s pretty rare for me to find an opportunity to do proper hacking; poking and probing systems just to see what will happen out of pure curiosity, but it’s still the most fulfilling use of time for me, regardless of whether anything explodes.

Never stop being curious. Happy hacking!