FreeBSD: virtualize Debian 12 with bhyve

Published on: 2023-12-31

Virtualization with bhyve (pronounced "beehive") was amazing adventure. I've learnt a lot thanks to bhyve.

Bhyve can be used on systems based on Illumos includes SmartOS, OpenIndiana and OmniOS. There is also port for MacOS called xhyve

Table of Contents:

Bhyve virtualization guides

Kick off

The start was confusing for me because there are a lot terms at the beginning. Official FreeBSD handbook describes usage of bhyve, but most other guides on the internet use vm. There is a list of terms i've met.

  1. bhyve
  2. bhyvectl
  3. vm-bhyve
  4. vm-bhyve-devel
  5. bhyve libvirt/virt-manager
  6. iohyve
  7. BVCP: Bhyve Virtual Machine Control Panel

All of these seem same. All use bhyve to create and control virtual machines. I am still not able to bring out the proper differences between them. But I supposed that the low level is bhyve and bhyvectl. The rest of the list are layer above the bhyve itself they are Management system for FreeBSD bhyve virtual machines. Anyway do not trust me I am not sure about that.

bhyve - is a hypervisor that runs guest operating systems inside a virtual machine.

bhyvectl - is a control utility for active bhyve(8) virtual machine instances.

vm-bhyve - is a management system for FreeBSD bhyve virtual machines. This utility should simplified the creation, control and configuration of virtual machines. Utilitu is run by command $ vm. This utility is most of the time recommended for work with bhyve in guides on the internet.

bhyve libvirt/virt-manager - there is a libvirt/virt-manager a bhyve is as its backend, I supposed.

Other terms used in relation to bhyve

  1. bhyveload
  2. grub2-bhyve

Environment configuration

Necessary packages to run bhyve with Linux guest


$ pkg install -y \
          vm-bhyve \
          grub2-bhyve \
          bhyve-firmware \
          edk2-bhyve \
          uefi-edk2-bhyve-csm \
          tigervnc-viewer

Bhyve templates - configuration options

My template for debian 12 


# cat /vm/.templates/debian-desktop-zvol.conf

loader="uefi"
cpu=2
memory=8G
network0_type="virtio-net"
network0_switch="public"
disk0_type="nvme"
disk0_name="disk0.img"
disk0_dev="sparse-zvol"
graphics="yes"
graphics_res="1300x700"
graphics_wait="yes"
graphics_port="5999"
graphics_listen="0.0.0.0"
  1. loader - type of boot loader
  2. cpu - number of used cpu cores
  3. memory - size of used memory. If I understand well it should be max memory which will be used but virtual machine should use only as much memory as is needed at particular moment.
  4. loader - type of boot loader
  5. loader - type of boot loader

Mouse problem

I suppose all guides on the internet about bhyve recommend add to config file option xhci_mouse="YES" to enable middle button of your mouse in vnc viewer. When I put this option to config my mouse have not worked at all. I've read that can be caused by vnc viewer, but try with tightvnc did not help. Then I've found this forum answer, which advice to omit this option because it can cause trouble with mouse for linux. It helped. If you have a problem with mouse try to play with xhci_mouse.

Prepare ZFS volume (zvol) for virtual machine


$ zfs create -o mountpoint=/vm zroot/vm

Prepare virtual machine & debian guest


$ sudo vm create -t debian-desktop-zvol debian12

This command will create a new virtual machine by specified template with option -t.

The usage is: vm [-t template] name.

The name is name of virtual machine. More information can be found by man vm. 


$ sudo vm install debian12 debian-12.4.0-amd64-DVD-1.iso

It attached the ISO to virtual machine. Thanks to this command bhyve can boot debian from ISO. If you have problem with booting before the installation was done rerun this command to reattach the ISO to virtual machine. Start a guest installation for the named virtual machine, using the specified ISO. The iso argument should be the filename of an ISO or image file downloaded into the $vm_dir/.iso

Fix problem with boot

I had problem with booting of debian12 guest when I restarted FreeBSD or the virtual machine itself. It was really strange to me, because when I created the new virtual machine with debian everything worked, even though I've restarted debian after installation the booting process was fine. Boot problem occured when I powered off or rebooted my FreeBSD or restarted the whole virtual machine. Then I've got this boot error BdsDxe: failed to load Boot0001 "UEFI bhyve-NVMe" from .... The error means that UEFI firmware can not find EFI bootloader. The boot problem is described here with Kali Linux. Computer then try the network boot >>Start PXE over IPv4. If none of available boot options is succeed, then UEFI started UEFI Interactive Shell.

Full content of error message is:


BdsDxe: failed to load Boot0001 "UEFI bhyve-NVMe NVME-4-0 from PciRoot(0x0)/Pci(0x4,0x0)/NVMe(0x1,01-00-.....): Not Found"

>>Start PXE over IPv4.
  PXE-E18: Server response timeout.
BdsDxe: failed to load Boot0002 "UEFI PXEv4 (MAC:.....)" from ...: Not Found

>>Start PXE over IPv6.

Two reasons can caused the boot problem, it depends when it occurs.

  1. ISO file is not attached or mount as CD or DVD. It can happened if you run vm install debian12 debian-12.4.0-amd64-DVD-1.iso and then you restart virtual machine or the FreeBSD host, then the ISO is detached from virtual machine and your storage device NVMe is empty (even most likely without a filesystem) because no installation has taken a place. So ISO missing and NVMe storage is empty, so EFI bootloader file can not exist anywhere. The solution is to re-attach the ISO to virtual machine by re-run the install command. Then bhyve can locate the EFI bootloader file from ISO again.

  2. The second one is that the Debian installation put EFI bootloader file to path boot/efi/EFI/debian/grubx64.efi by default. Unfortunately bhyve is looking for EFI bootloader in different location boot/efi/EFI/BOOT/bootx64.efi. So it is obvious that bhyve can not find the EFI bootloader.

    The solution is to copy the debian/grubx64.efi to BOOT/bootx64.efi. There are two options how do it. You can copy EFI bootloader file at the end of debian installation or with UEFI Interactive Shell. Both cases are described in next chapters. Below are links which describes the solutions.

Copy EFI bootloader at the end of debian installation

The whole idea is to execute shell at the end of installation and copy the debian EFI bootloader to path, where it will be found by bhyve. Steps with pictures is described in article on cyberciti

Copy EFI bootloader with UEFI Interactive Shell

If you forget to copy at the end of installation you can copy EFI bootloader file with UEFI Interactive Shell. UEFI Shell is little different than the shell from UNIX. For example slashes in path are backslashes. The command ls works only if are in some disk, where you are not by default. You have to enter to some first. I've found this article about UEFI Shell, where is fine guide to UEFI Shell.

You can get to UEFI Interactive Shell through the BIOS or just wait until all boot options failed and UEFI executes the Shell automatically.

Start point to use UEFI Interactive Shell is help command. Help will list you all command which can be use with the Shell. Unfortunately you can not scroll in console and less is not available. Fortunately command help has option -b which is enabling pager. This option is really useful and different from common shell in UNIX for me. You can also use help with any command to get usage desciption of particular command for example help map

Another step is to enter to particular storage device to copy EFI bootloader file to the right location in perspective of bhyve. To list all available storace devices use command map. If you add other storage device during working in UEFI INteractive Shell you have to refresh the list with option -r, so to see a new added storage device run $ map -r then. or you can use verbose variant as $ map -v

You should get something as below, where BLK0 can something different FS0 for example. 


Mapping table
      BLK0: Alias(s):
            PciRoot(0x0)/Pci(0x4,0x0)/NVMe(0x1,)

To find particular storage device should be easy. There should be only one disk. If there are more storages the right should be the NVMe with number in parentheses which is same as the number of NVMe disk mentioned in Boot0001 error. The NVMe(0x1,01-00-......).

Enter to particular storage to write its name with semicolon. The prompt should change from Shell> to name of particular storage device. 


Shel> BLK0:
BLK0:\>

Supervise the virtual machine


$ tail -f /vms/fbsd13/vm-bhyve.log
      

$ vm list
      

$ vm info debian12
      

$ vm image list

bhyve-webadmin utility

bhyve-webadmin does have a Graphical user Interface via webinterface and also a CLI and an API. The software provides webGUI to let you manage Virtual Machines remotely.

The graphical and secure webcontrol panel for FreeBSD Bhyve's Virtual Machines is BVCP Application (Bhyve Virtual-Machine Control Panel).