There is an unexpected quick, easy and satisfying way to create new virtual machines for the bhyve supervisor in FreeBSD with the vm-bhyve utility and special cloud images supplied by many distributors.
After some years running some headless VirtualBox instances on my FreeBSD server, accompanied by a ton of headaches like mismatching kernel modules, awkward network problems or mysterious deadlocks, I finally got happy with the hypervisor that modern BSD systems already bring to the party: bhyve. All the years I regarded it as being too complicated and fine granular, not realizing it was just designed as a backend to be managed with separate tools like vm-bhyve.
Since a year or two, I have all my VMs running extremely stable and with excellent performance with bhyve. Especially helpful is the good support for 9p file systems for the mounts (this will be a separate article soon, hopefully).
Managing the VMs with vm-bhyve is ridiculously easy:
$ sudo vm list NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE dock default uefi 4 4G - Yes  Running (1998) gitlab default uefi 4 6G - Yes  Running (2247) roon default uefi 4 4G - Yes  Running (1799) $ sudo vm restart gitlab Setting guest restart flag Sending ACPI shutdown to gitlab
The whole management is integrated neatly into the BSD environment. VM devices can be managed automatically in ZFS’s ZVols, enabling all the related goodness like (automated) snapshotting, cloning, transfers, …. Automatic starting is configured in a BSDish style per settings in
vm_enable="YES" vm_dir="zfs:bigred/vm" vm_list="roon dock gitlab" vm_delay="1"
Even configuration of the specific VMs can be done with a text editor, which I do prefer over even the nicest GUIs:
$ sudo vm config gitlab loader="uefi" cpu=4 memory=6G network0_type="virtio-net" network0_switch="public" disk0_type="virtio-blk" disk0_name="zvol" disk0_dev="sparse-zvol"
Well, yes, it’s really that simple :-)
Using Cloud Init for VM creation
For creating new VMs, there is a great, yet rather undocumented1, vm-bhyve option
-n that facilitates the cloud-init standard for automated kicking off a virtual machine from a special image by passing in the required setup:
First, we have to download a cloud image of the desired guest OS. Here we’ll use an Ubuntu 22.04 (Jammy Jellyfish) Server CloudImg. To allow the vm-bhyve the passing of the VM config, we have to install a tool that allows automatically creating the second ISO image containing the cloud image parameters:
sudo pkg install cdrkit-genisoimage qemu-tools
Now we can simply create and start a fully installed, runnable virtual machine with one vm create statement within a minute or two. Not only will it be set up with the desired network configuration, it will also have our public SSH key installed to the root (in this case ‘ubuntu’) user:
$ sudo vm create -s 20G -i jammy-server-cloudimg-amd64.img -C -n "ip=192.168.1.160/24;gateway=192.168.1.1;nameservers=192.168.1.5" -k ~/.ssh/id_rsa.pub showcase1 $ sudo vm start showcase1 Starting showcase1 * found guest in /vm/showcase1 * booting... $ ssh email@example.com Welcome to Ubuntu 22.04.1 LTS (GNU/Linux 5.15.0-50-generic x86_64) ... $ uname -a Linux showcase1 5.15.0-50-generic #56-Ubuntu SMP Tue Sep 20 13:23:26 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux $ quit $ sudo vm list NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE dock default uefi 4 4G - Yes  Running (1984) gitlab default uefi 4 6G - Yes  Running (2238) roon default uefi 4 4G - Yes  Running (1787) showcase1 default uefi 1 512M - No Running (8948)
While experimenting, sometimes I had some strange issues creating new VMs that were destroyed shortly before: The ZVols reported being busy, as soon as vm-bhyve tried to create the images. On a quick view, this looks like a race condition in vm-bhyve. However, using a different VM name or a reboot helped as a workaround sometimes. Sometimes I had to repeat multiple times. Once I can clearly reproduce this, I will open an issue with vm-bhyve.
I’m so sorry, I lost the reference where I found out about this one (except of my personal notes). If you have a pointer for me, please let me know! ↩︎