It’s been over a year now since I promised to write about the 9P file system mounts for bHyve virtualization in FreeBSD. Finally, here are my notes on how to set the 9P2000 protocol as a simple and high-performance mounting mechanism for virtual machines.
Plan 9 from Bell Labs
Das Plan 9 Maskottchen Glenda. (MIT-Lizenz)
Once upon a time (in the late 1980s and early 1990s), in the hallowed halls of Bell Labs, a group of visionary programmers embarked on a quest to create an operating system so advanced, it would make even Unix look like a mere apprentice. Thus was born Plan 9. The name was chosen wisely as a tongue-in-cheek reference to the 1959 cult film “Plan 9 from Outer Space”, which is often cited as one of the worst films ever made.
Despite incorporating some visionary ideas, Plan 9 did not achieve significant success in terms of widespread adoption or commercial impact. It failed to replace Unix or become a mainstream operating system, largely remaining within academic and research circles.
However, some ideas maintained their appeal into this millennium. In 2005, the IBM Austin Research Lab and the Los Alamos National Labs published their paper “Grave Robbers from Outer Space” at the annual Usenix conference, presenting an implementation of the 9P2000 client/server protocol for Linux.
This protocol implementation comes surprisingly handy nowadays for exposing parts of a host filesystem to a virtualized guest without wasting resources on emulated hardware or network protocols.
My use case / history
On my FreeBSD server, I use a bHyve VM to run Roon, my prefered Music Server. There’s a large collection of lossless music albums stored as tenth of thousands of files on a local RAIDZ storage, ripped from my own CD collection as well as purchased HiRes tracks. Roon will not only scan your music directories for files and their metadata but also conduct a full scan of each music file to determine its dynamic range value (DR).
Initially, I used NFSv4 to mount the music directory into the VM. This setup worked. However, the performance was extraordinarily poor and the CPU load high. The exact reason for this remained unclear.
As an experiment, I switched to CIFS, which resulted in drastically improved performance. I used this configuration for about a year. Nonetheless, using a “Windows” network file system (the artist formerly known as SMB) to connect two Unix machines felt wrong on many levels ;-).
Eventually, I discovered the 9P protocol, which offered nice performance (I should take some benchmarks someday to compare) and was much simpler to configure, as the files' UIDs, GIDs and permissions are simply passed through without any mapping or limitation.
Usage
Interestingly, the bHyve implementation of the 9P protocol doesn’t offer many options for customization and will export the entire directory with full write permissions. In my scenario, where the VMs are used to create a separation of concerns between different tasks, this isn’t a problem. However, other use cases might need limitations that 9P cannot provide in this context.
First, we need to prepare the bHyve host. As I mentioned earlier, I use the vm-bhyve-Package for managing my bHyve VMs.
Host config
I still haven’t found any comprehensive documentation on this for bHyve and wonder where I learned about it in the first place.
By using sudo vm config roon, we can edit our VM’s configuration. We need to add three lines (adjust the disk number to your needs):
disk2_type="virtio-9p"
disk2_name="music=/storage/media/audio/music"
disk2_dev="custom"
bhyve(8) tells us that one may postfix the disk name with ,ro to disallow write access to the mounted volume, at least. Other options are not supported (or at least not documented, yet).
Now, the host config is complete. Since vm-bhyve struggles with changing the config of an online VM, this config is not yet live. However, we will reboot the VM later anyway, so this can wait.
Guest config
The configuration for the Linux guest in the VM is relatively straightforward. We just need to add a line to the /etc/fstab to enable automated mounting on boot:
music /home/music 9p trans=virtio,cache=mmap,msize=512000 0 0
At least on the Linux side, the use of the 9P protocol seems to be documented properly.
Restart
su vm roon restart
Well, that’s it. I can’t believe, how easy that was. The journey from NFSv4 and CIFS protocols to the 9P protocol not only improved performance but drastically reduced complexity in my use case.
However, please keep in mind that there is no fine granular control of permissions, it’s just read or read/write for everything. This may or may not be what you need.