Create VMs in minutes with cloud config for vm-bhyve

There is an unexpected quick, easy and satisfying way to create new virtual machines for the bhyve supervisor in FreeBSD with the vm-bhyve utility and special cloud images supplied by many distributors.

After some years running some headless VirtualBox instances on my FreeBSD server, accompanied by a ton of headaches like mismatching kernel modules, awkward network problems or mysterious deadlocks, I finally got happy with the hypervisor that modern BSD systems already bring to the party: bhyve. All the years I regarded it as being too complicated and fine granular, not realizing it was just designed as a backend to be managed with separate tools like vm-bhyve.

Since a year or two, I have all my VMs running extremely stable and with excellent performance with bhyve. Especially helpful is the good support for 9p file systems for the mounts (this will be a separate article soon, hopefully).

Managing the VMs with vm-bhyve is ridiculously easy:

$ sudo vm list
NAME    DATASTORE  LOADER  CPU  MEMORY  VNC  AUTO     STATE
dock    default    uefi    4    4G      -    Yes [2]  Running (1998)
gitlab  default    uefi    4    6G      -    Yes [3]  Running (2247)
roon    default    uefi    4    4G      -    Yes [1]  Running (1799)

$ sudo vm restart gitlab
Setting guest restart flag
Sending ACPI shutdown to gitlab

The whole management is integrated neatly into the BSD environment. VM devices can be managed automatically in ZFS’s ZVols, enabling all the related goodness like (automated) snapshotting, cloning, transfers, …. Automatic starting is configured in a BSDish style per settings in /etc/rc.conf:

vm_enable="YES"
vm_dir="zfs:bigred/vm"
vm_list="roon dock gitlab"
vm_delay="1"

Even configuration of the specific VMs can be done with a text editor, which I do prefer over even the nicest GUIs:

$ sudo vm config gitlab
loader="uefi"
cpu=4
memory=6G
network0_type="virtio-net"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="zvol"
disk0_dev="sparse-zvol"

Well, yes, it’s really that simple :-)

Using Cloud Init for VM creation

For creating new VMs, there is a great, yet rather undocumented¹, vm-bhyve option -n that facilitates the cloud-init standard for automated kicking off a virtual machine from a special image by passing in the required setup:

First, we have to download a cloud image of the desired guest OS. Here we’ll use an Ubuntu 22.04 (Jammy Jellyfish) Server CloudImg. To allow the vm-bhyve the passing of the VM config, we have to install a tool that allows automatically creating the second ISO image containing the cloud image parameters:

sudo pkg install cdrkit-genisoimage qemu-tools

Now we can simply create and start a fully installed, runnable virtual machine with one vm create statement within a minute or two. Not only will it be set up with the desired network configuration, it will also have our public SSH key installed to the root (in this case ‘ubuntu’) user:

$ sudo vm create -s 20G -i jammy-server-cloudimg-amd64.img -C -n "ip=192.168.1.160/24;gateway=192.168.1.1;nameservers=192.168.1.5" -k ~/.ssh/id_rsa.pub showcase1
$ sudo vm start showcase1
Starting showcase1
  * found guest in /vm/showcase1
  * booting...
$ ssh ubuntu@192.168.1.160
Welcome to Ubuntu 22.04.1 LTS (GNU/Linux 5.15.0-50-generic x86_64)
...
$ uname -a
Linux showcase1 5.15.0-50-generic #56-Ubuntu SMP Tue Sep 20 13:23:26 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
$ quit
$ sudo vm list
NAME       DATASTORE  LOADER  CPU  MEMORY  VNC  AUTO     STATE
dock       default    uefi    4    4G      -    Yes [2]  Running (1984)
gitlab     default    uefi    4    6G      -    Yes [3]  Running (2238)
roon       default    uefi    4    4G      -    Yes [1]  Running (1787)
showcase1  default    uefi    1    512M    -    No       Running (8948)

Caveat

While experimenting, sometimes I had some strange issues creating new VMs that were destroyed shortly before: The ZVols reported being busy, as soon as vm-bhyve tried to create the images. On a quick view, this looks like a race condition in vm-bhyve. However, using a different VM name or a reboot helped as a workaround sometimes. Sometimes I had to repeat multiple times. Once I can clearly reproduce this, I will open an issue with vm-bhyve.