Installing urgent system upgrades can be a scary and challenging task, if you have a complex remote production system with only limited console access.
To the rescue, a feature called Boot Environments has been ported from Solaris to FreeBSD. Based on snapshots and writable clones from ZFS, it allows system and package upgrades to be performed in a safe environment, without overriding the working system config (even without going down for the installation process). When problems occur, an old working configuration could be rolled back instantly, or even chosen directly from the FreeBSD boot menu.
FreeBSD has a built-in tool (bectl) for this, but it has some issues with snapshotting nested ZFS file systems, so I usually use the beadm tool from the packages/ports for managing my boot environments. The created boot environments are compatible between both.
# XXXX for the update you want to install. The boot environment
# will be named like this for convenience, too. (Example: "13.0-RELEASE-p2")
UPDATE="XXXX"
sudo beadm create $UPDATE
sudo beadm mount $UPDATE /tmp/safe
sudo chroot /tmp/safe
Now you’re left to a shell inside a writable clone of the currently running OS. You can perform all your installation and configuration needs without interfering to the – currently running – main system.
Here is my typical choreography for performing a minor FreeBSD system update and update the installed packages:
# For a minor (patch-) udate:
# Inside the chroot enviroment
mount -t devfs devfs /dev
rm -rf /var/db/freebsd-update
mkdir /var/db/freebsd-update
freebsd-update fetch
freebsd-update install
pkg upgrade
If the system update requires installation of a new kernel with a new Kernel ABI, normally the installation would require a reboot between the required two calls of freebsd-update install. In the separated boot environment, we can do this in one shot:
# For a release upgrade:
# Inside the chroot enviroment
mount -t devfs devfs /dev
rm -rf /var/db/freebsd-update
mkdir /var/db/freebsd-update
# XXXXXXX is the desired FreeBSD version to upgrade to, like '13.0-RELEASE'
freebsd-update upgrade -r XXXXXXX
freebsd-update install
pkg upgrade
freebsd-update install
After performing the installation in one of the above ways, just leave the chroot environment with exit or By pressing ^D. No unmount everything and mark the environment as active for booting:
sudo umount /tmp/safe/dev
sudo beadm unmount $UPDATE
sudo beadm activate $UPDATE
sudo reboot
After the reboot, we will find ourselves in the newly created environment (“13.0-RELEASE-p7” in this case):
$ sudo beadm list
BE Active Mountpoint Space Created
13.0-RELEASE-p4 - - 950.5M 2021-09-28 19:42
13.0-RELEASE-p5 - - 172.9M 2021-12-26 11:41
13.0-RELEASE-p6 - - 122.2M 2022-01-12 18:10
13.0-RELEASE-p7 NR / 7.5G 2022-02-05 15:37
Bonus: One-time boot into environment
Sometimes, especially with hard remote problems, it can be helpful to do a one-time boot into the new environment. Sadly, beadm does not yet implement this feature. However, FreeBSD’s own bectl can do it. The system takes the BE for one boot, but the current BE stays the default for further boots.
# XXXX is the name of the environment activated for one boot
bectl activate -t XXXX
Caveat:
Once, I did a new installation into a new boot environment, but forgot to boot into the new install immediately. When I did so, it took me a while to realize that all changes since the installation were gone (well, they were not lost, but I had to transfer them manually from the old boot environment).
This can be avoided by carefully crafting your file hierarchy in a way that all data that should persist OS changes is out of the way for boot environments. In my case, this means having separate mount points that are not part of the boot environment for /var/db and /var/log.
Pointers:
Some helpful pointers leading me here: