churchers / vm-bhyve

Shell based, minimal dependency bhyve manager

vm-bhyve

Management system for FreeBSD bhyve virtual machines

Some of the main features include:

• Windows/UEFI support
• Simple commands to create/start/stop bhyve instances
• Simple configuration file format
• Virtual switches supporting vlans & automatic device creation
• ZFS support
• FreeBSD/MidnightBSD/NetBSD/OpenBSD/Linux guest support
• Automatic assignment of console devices to access guest console
• Integration with rc.d startup/shutdown
• Guest reboot handling
• Designed with multiple compute nodes + shared storage in mind (NFS/iSCSI/etc)
• Multiple datastores
• VNC graphics & tmux support (1.1+ only. See wiki for instructions)
• Dependency free**

** Some additional packages may be required in certain circumstances -

• The port has a dependancy on ca_root_nss added by the ports maintainers to help avoid any SSL errors when downloading FreeBSD ISO files using the vm iso command.
• sysutils/grub2-bhyve is required to run Linux or any other guests that need a Grub bootloader.
• sysutils/bhyve-firmware is required to run UEFI guests
• sysutils/tmux is needed to use tmux console access instead of cu/nmdm

See the GitHub wiki for more information and examples.

For most users, I recommend using the version in ports (1.1+). Main development happens in the master branch on GitHub and it may contain broken or incomplete features.

Quick-Start

A simple overview of the commands needed to install vm-bhyve and start a freebsd guest. See the sections below for more in-depth details.

1. pkg install vm-bhyve
2. zfs create pool/vm
3. sysrc vm_enable="YES"
4. sysrc vm_dir="zfs:pool/vm"
5. vm init
6. cp /usr/local/share/examples/vm-bhyve/* /mountpoint/for/pool/vm/.templates/
7. vm switch create public
8. vm switch add public em0
9. vm iso https://download.freebsd.org/ftp/releases/ISO-IMAGES/11.2/FreeBSD-11.2-RELEASE-amd64-bootonly.iso
10. vm create myguest
11. vm install [-f] myguest FreeBSD-11.2-RELEASE-amd64-bootonly.iso
12. vm console myguest

• Line 1 Install vm-bhvye

• Line 2 Create a dataset for your virtual machines. If you're not using ZFS, just create a normal directory.

• Lines 3-4 Enable vm-bhyve in /etc/rc.conf and set the dataset to use. If not using ZFS, just set $vm_dir="/my/vm/folder".

• Line 5 Run the vm init command to create the required directories under $vm_dir and load kernel modules.

• Line 6 Install the sample templates that come with vm-bhyve.

• Lines 7-8 Create a virtual switch called 'public' and attach your network interface to it. Replace em0 with whatever interface connects your machine to the network.

• Line 9 Download a copy of FreeBSD from the ftp site.

• Lines 10-12 Create a new guest using the default.conf template, run the installer and then connect to its console. At this point proceed through the installation as normal. By specifying the -f option before the install command, the guest will run directly on your terminal so the console command is not required. (Bear in mind that you won't get back to your terminal until the guest is fully shutdown)

Install

Download the latest release from GitHub, or install sysutils/vm-bhyve

To install, just run the following command inside the vm-bhyve source directory

# make install

If you want to run guests other than FreeBSD, you will need the grub2-bhyve package;

# pkg install grub2-bhyve

Initial configuration

First of all, you will need a directory to store all your virtual machines and vm-bhyve configuration. If you are not using ZFS, just create a normal directory:

# mkdir /somefolder/vm

If you are using ZFS, create a dataset to hold vm-bhyve data

# zfs create pool/vm

Now update /etc/rc.conf to enable vm-bhyve, and tell it where your directory is

vm_enable="YES"
vm_dir="/somefolder/vm"

Or with ZFS:

vm_enable="YES"
vm_dir="zfs:pool/vm"

This directory will be referred to as $vm_dir in the rest of this readme.

Now run the following command to create the directories used to store vm-bhvye configuration and load any necessary kernel modules. This needs to be run once after each host reboot, which is normally handled by the rc.d script

# vm init

Virtual machine templates

When creating a virtual machine, you use a template which defines how much memory to give the guest, how many cpu cores, and networking/disk configuration. The templates are all stored inside $vm_dir/.templates. To install the sample templates, run the following command:

# cp /usr/local/share/examples/vm-bhyve/* /my/vm/path/.templates/

If you look inside the template files with a text editor, you will see they are very simple. You can create as many templates as you like. For example you could have web-server.conf, containing the setting for your web servers, or freebsd-large.conf for large FreeBSD guests, and so on. This is the contents of the default template:

guest="freebsd"
loader="bhyveload"
cpu=1
memory=256M
disk0_type="virtio-blk"
disk0_name="disk0.img"
network0_type="virtio-net"
network0_switch="public"

You will notice that each template is set to create one network interface. You can easily add more network interfaces by duplicating the two network configuration options and incrementing the number. In general you will not want to change the type from 'virtio-net', but you will notice the first interface is set to connect to a switch called 'public'. See the next section for details on how to configure virtual switches.

I recommend reading the man page or sample-templates/config.sample for a full list of supported template options and a description of their purpose. Almost all bhyve functionality is supported and a large variety of network/storage configurations can be achieved.

Virtual Switches

When a guest is started, each network interface is automatically connected to the virtual switch specified in the configuration file. By default all the sample templates connect to a switch called 'public', although you can use any name. The following section shows how to create a switch called 'public', and configure various settings:

# vm switch create public

If you just want to bridge guests to your physical network, add the appropriate real interface to the switch. Obviously you will need to replace em0 here with the correct interface name on your system:

# vm switch add public em0

If you want guest traffic to be on a specific VLAN when leaving the host, specify a vlan number. To turn off vlans, just set the vlan number to 0:

# vm switch vlan public 10
# vm switch vlan public 0

You can view current switch configuration using the list command:

# vm switch list

Creating virtual machines

Use one of the following command to create a new virtual machine:

# vm create testvm
# vm create -t templatename -s 50G testvm

The first example uses the default.conf template, and will create a 20GB disk image. The second example specifies the templatename.conf template, and tells vm-bhyve to create a 50GB disk.

You will need an ISO to install the guest with, so download one using the iso command:

# vm iso https://download.freebsd.org/ftp/releases/ISO-IMAGES/11.2/FreeBSD-11.2-RELEASE-amd64-disc1.iso

To start a guest install, run the following command. vm-bhyve will run the machine in the background, so use the console command to connect to it and finish installation.

# vm install testvm FreeBSD-11.2-RELEASE-amd64-disc1.iso
# vm console testvm

You can also specify the foreground option to run the guest directly on your terminal:

# vm install -f testvm FreeBSD-11.2-RELEASE-amd64-disc1.iso

Once installation has finished, you can reboot the guest from inside the console and it will boot up into the new OS (assuming installation was successful). Further reboots will work as expected and the guest can be shutdown in the normal way. As the console uses the cu command, type ~+Ctrl-D to exit back to your host.

The following commands start and stop virtual machines:

# vm start testvm
# vm stop testvm

The basic configuration of each machine and state can be viewed using the list command:

# vm list
NAME            GUEST      LOADER      CPU    MEMORY    AUTOSTART    STATE
alpine          linux      default     1      512M      No           Stopped
c7              linux      default     1      512M      Yes [2]      Stopped
centos          linux      default     1      512M      No           Stopped
debian          linux      default     1      512M      No           Stopped
fbsd            freebsd    default     1      256M      No           Stopped
netbsd          generic    grub        1      256M      No           Stopped
openbsd         generic    grub        1      256M      No           Stopped
pf              freebsd    default     1      256M      Yes [1]      Stopped
ubuntu          linux      default     1      512M      No           Stopped
wintest         windows    default     2      2G        No           Running (2796)

All running machines can be stopped using the stopall command

# vm stopall

On host boot, vm-bhyve will use the 'vm startall' command to start all machines. You can control which guests start automatically using the following variables in /etc/rc.conf:

vm_list="vm1 vm2"
vm_delay="5"

The first defines the list of machines to start on boot, and the order to start them. The second is the number of seconds to wait between starting each one. 5 seconds is the recommended setting, although a longer delay is useful if you have disk intensive guests and don't want them all booting at the same time.

There's also a command which opens a guest's configuration file in your default text editor, allowing you to easily make changes to the configuration. Please note that changes only take effect after a full shutdown and restart of the guest

# vm configure testvm

See the man page for a full description of all available commands.

# man vm

Using cloud images

You can use cloud images to create virtual machines. The vm img command will download the image to datastore and uncompress it if needed (.xz, .tar.gz, and .gz files are supported). The image should be in RAW or QCOW2 format. To use this feature you'll need install qemu-tools package:

# pkg install qemu-tools

To launch FreeBSD using official cloud image:

# vm img https://download.freebsd.org/ftp/releases/VM-IMAGES/11.2-RELEASE/amd64/Latest/FreeBSD-11.2-RELEASE-amd64.raw.xz
# vm create -t freebsd-zvol -i FreeBSD-11.2-RELEASE-amd64.raw freebsd-cloud
# vm start freebsd-cloud

To list downloaded images:

# vm img
DATASTORE           FILENAME
default             CentOS-7-x86_64-GenericCloud-20180930_02.raw
default             debian-9-openstack-amd64.qcow2
default             Fedora-AtomicHost-28-1.1.x86_64.raw
default             FreeBSD-11.2-RELEASE-amd64.raw
default             xenial-server-cloudimg-amd64-uefi1.img

Using cloud init

vm-bhyve has basic support for providing cloud-init configuration to the guest. You can enable it with -C option to vm create command. You can also pass public SSH key to be injected into the guest with option -k <file>.

Example:

# vm create -t linux -i xenial-server-cloudimg-amd64-uefi1.img -C -k ~/.ssh/id_rsa.pub cloud-init-ubuntu
# vm start cloud-init-ubuntu
Starting cloud-init-ubuntu
* found guest in /zroot/vm/cloud-init-ubuntu
* booting...
# ssh [email protected]
The authenticity of host '192.168.0.91 (192.168.0.91)' can't be established.
ECDSA key fingerprint is SHA256:6s9uReyhsIXRv0dVRcBCKMHtY0kDYRV7zbM7ot6u604.
No matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.0.91' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 16.04.5 LTS (GNU/Linux 4.4.0-141-generic x86_64)

Adding custom disks

Scenario: If you have a vm on one zpool and would like to add a new virtual disk to it that resides on a different zpool.

Manually create a sparse-zvol (in this case 50G in size).

# zfs create -sV 50G -o volmode=dev "zpool2/vm/yourvm/disk1"

Add it to your vm config file. Please note, for Windows guests the type will need to be ahci-hd, as it does not have virtio-blk drivers.

# vm configure yourvm

disk1_name="/dev/zvol/zpool2/vm/yourvm/disk1"
disk1_type="virtio-blk"
disk1_dev="custom"

Restart your vm.

Windows Support

Please see the Windows section in the Wiki

Autocomplete

If you are using the default csh/tcsh shell built into FreeBSD, running the following command should allow autocomplete to work for all the currently supported functions. This is especially useful for viewing and completing guest & ISO file names. Please note that there's three occurrences of '/path/to/vm' which need to be changed to the directory containing your virtual machines.

To make the autocomplete features available permanently, add the following to your $HOME/.cshrc file. Then either logout/login, or run source ~/.cshrc to cause the .cshrc file to be reloaded.

complete vm \
 'p@1@(list create install start stop console configure reset poweroff destroy clone snapshot rollback add switch iso)@' \
 'n@create@n@' \
 'n@list@n@' \
 'n@iso@n@' \
 'n@switch@(list create add remove destroy vlan nat)@' \
 'N@switch@`sysrc -inqf /path/to/vm/.config/switch switch_list`@' \
 'N@install@`ls -1 /path/to/vm/.iso`@' \
 'N@nat@(off on)@' \
 'p@2@`ls -1 /path/to/vm | grep -v "^\." | grep -v "^images"`@'

vm-bhyve's Issues

Additional disks

vm-bhyve happily allows multiple guest disks. For UEFI guests this is currently limited to 3 although bhyveload|grub-bhyve guests can have as many as bhyve will support.

However, while the first disk is created during vm create, additional disks must be created by hand. I have considered moving the disk provisioning into _run, so that disks are created at runtime if they don't already exist. If a new disk is added to the guest configuration, it will be automatically created next time the guest runs. The problem with this is that the configuration doesn't currently contain the disk size.

There are two possible solutions to this that I can see:

• Add the disk size to the configuration.
  When creating a new guest I will need to take the size given by the user and add that to the configuration. It may also make sense to update the sample templates to include a default size. The problem with this is if the user changes the size in the template. Either we end up out of sync with the real image, or we try and resize (dangerous). Also the user resizing the disk manually will make it out of sync with the configuration. All in all it seems messy to include the disk sizes in configuration.

• Provide a vm add command
  Provide a command which can be used to add disks (and possibly other devices) to the configuration. This would be most useful for disks as it can take care of creating the image file or zvol in the correct place, as well as updating the configuration.

  vm add [-d device] [-t type] [-s size] guest

  vm add -d disk -t zvol -s 30G guestname

While I aim to keep the vm command as simple as possible, I think it may be worth adding the second option. In most cases users can edit the configuration file (vm configure guest) to make changes, but the extra complication of having to create the image file or zvol will make this command useful.

SmartOS Support

As of commit 4292fee 24/10, it's possible to run a SmartOS guest. The vm-bhyve configuration is the same as windows.conf, with the following changes:

guest="generic"
uefi="csm"
hostbridge="none"
comports="com1 com2"

Seeing as it's a UEFI guest and requires no specific loader, there's no point me adding another guest support function for it when generic can be used.

Note that you'll need a copy of the UEFI CSM firmware which can be obtained from the same place as the normal UEFI firmware. The other important difference, that is only possible since the commit mentioned, is hostbridge="none". At the moment the hostbridge causes the virtio-net network adapter to be presented to SmartOS in a way that causes it to panic. I believe this is down to a VirtIO bug in Illumos which has already been fixed in development versions.

The other issue with SmartOS is that it is designed to never boot from the hard disk. It is designed to run from a CD, or more commonly a network location. The hard disk is only used as persistent storage. Because of this, the guest always needs to be started in install mode. The first time SmartOS will enter a configuration menu to set up the system and store configuration on the hard disk. The next time it boots off the ISO it will boot straight to a login prompt.

vm install smartosguest smartos-latest.iso

If you use the normal vm start command, it will fail to boot, as there's not actually any boot code or kernel installed to the hard disk. Unfortunately this does mean that you can't use the autostart (start on boot) option with SmartOS, as that uses the start command.

If there are any requests for this to be supported, comment here. If people actually want it supported properly, I can add a bootiso="isofilename.iso" configuration option, which can be used to specify an ISO file which should be used for every boot of a guest. That way you can vm start smartos, and it will still boot off the ISO. It's not worth adding this if there's no requests for it though as SmartOS is a bit of a special case.

Feature request - detailed guest/switch information

So at this point I have a number of VM's managed by this tool and given the diversity in the networking setup it's getting rather difficult to easily figure out what is doing what on the networking side.

It would be great if functionality like:

vm list <name> / <all>

printed out the vm details / all vm details.

And/or if there was some built-in method of listing out vm nic(s)/switch(s)

Much of the interface here reminds of the Xen/xm days and in that vein I'm asking for:

xm network-list

About UEFI support version

Since SVN r295124, FreeBSD 10 stable support booting Windows via UEFI.
the UEFI version checking should change from

1100000

to

1002509(FreeBSD 10.3 Beta1)

Linux stuck in grub-bhyve on boot

It seems that Linux guests (Centos at least) get stuck in grub-bhyve on boot. Unfortunately it also seems to use 100% CPU while it's stuck.

Feature Request: this of vm name and a cd command.

I think it is nice to have "this" keyword as a vm name when the current directory is in the vm directory.
and also have a cd command.
such as

root@bhyve:~ # vm list
NAME            GUEST           CPU    MEMORY    AUTOSTART    STATE               
vm01            windows         2      4G        No           Stopped             
root@bhyve:~ # vm cd vm01
root@bhyve:/zones/vm/vm01 #
root@bhyve:/zones/vm/vm01 # vm start this
root@bhyve:/zones/vm/vm01 # 

it will be easier to type command, especially when vm name is very long and hard to remember.

Feature request: launch virtual machines from cloud images

As an alternative to installing from ISO images.

Examples:

# vm image import /path/to/FreeBSD-10.2-RELEASE-amd64.raw
Image of importXXXXXXXX created with UUID ed0badc7-e84a-4ca2-b3a2-c38b3c8d6156
Successfully imported FreeBSD-10.2-RELEASE-amd64.raw
# vm image provision ed0badc7-e84a-4ca2-b3a2-c38b3c8d6156 cloudvictim0

Then after some other process takes care of creating an OpenStack config drive ISO:

# vm configure
...edit disk1_name variable to point to ISO...
# vm start cloudvictim0

I don't know yet if an additional top-level vm command would be welcome to choose the ISO; if it is then I can open another issue.

make install failed when /usr/local/etc/rc.d and /usr/local/man/man8 does not exist.

here is the log.

root@bhyve:~/vm-bhyve # make install
/bin/mkdir -p /usr/local/sbin
/bin/mkdir -p /usr/local/lib/vm-bhyve
/bin/mkdir -p /usr/local/share/examples/vm-bhyve
install -m 0500 vm /usr/local/sbin/
install lib/* /usr/local/lib/vm-bhyve/
install sample-templates/* /usr/local/share/examples/vm-bhyve/
install rc.d/* /usr/local/etc/rc.d/
install: /usr/local/etc/rc.d/: No such file or directory
*** Error code 71

Stop.
make: stopped in /root/vm-bhyve
root@bhyve:~/vm-bhyve # mkdir -p /usr/local/etc/rc.d
root@bhyve:~/vm-bhyve # make install
/bin/mkdir -p /usr/local/sbin
/bin/mkdir -p /usr/local/lib/vm-bhyve
/bin/mkdir -p /usr/local/share/examples/vm-bhyve
install -m 0500 vm /usr/local/sbin/
install lib/* /usr/local/lib/vm-bhyve/
install sample-templates/* /usr/local/share/examples/vm-bhyve/
install rc.d/* /usr/local/etc/rc.d/
rm -f vm.8.gz
gzip -k vm.8
install vm.8.gz /usr/local/man/man8/
install: /usr/local/man/man8/: No such file or directory
*** Error code 71

Stop.
make: stopped in /root/vm-bhyve
root@bhyve:~/vm-bhyve # mkdir -p /usr/local/man/man8
root@bhyve:~/vm-bhyve # make install
/bin/mkdir -p /usr/local/sbin
/bin/mkdir -p /usr/local/lib/vm-bhyve
/bin/mkdir -p /usr/local/share/examples/vm-bhyve
install -m 0500 vm /usr/local/sbin/
install lib/* /usr/local/lib/vm-bhyve/
install sample-templates/* /usr/local/share/examples/vm-bhyve/
install rc.d/* /usr/local/etc/rc.d/
rm -f vm.8.gz
gzip -k vm.8
install vm.8.gz /usr/local/man/man8/

clone after vm renamed

Here is the story...

root@bhyve:~ # vm snapshot vm01@test
root@bhyve:~ # vm rename vm01 newvm01
root@bhyve:~ # vm clone newvm01@test newvm02
mv: rename /zones/vm/newvm02/newvm01.conf to /zones/vm/newvm02/newvm02.conf: No such file or directory
tail: /zones/vm/newvm02/newvm02.conf: No such file or directory

Maybe for vm, the config name just a fixed name such as vm.config ?

cannot create vm which name contains dot

root@bhyve:/zones/vm/.templates # vm
vm: Bhyve virtual machine managament v0.9.16
...
root@bhyve:/zones/vm/.templates # vm create -t debian 170.vm02
/usr/local/sbin/vm: ERROR: invalid virtual machine name '170.vm02'

I think dot is a valid character for vm name.

Feature Request: ability to add tap device as a span member of bridge

when add tap device to vm switch, currently only act as a normal member. by
ifconifg bridge0 addm tap0 command. while tap device can be a span member
by using ifconifg bridge0 span tap0

we can specify this to vm config file. such as

network1_type="virtio-net"
network1_switch="private"
network1_switch_member_type="span"

and then when execute vm start command, vm bhyve will add this tap device to bridge as a span member.

vm clone will create a snapshot everytime.

if I use

vm clone guest guest1
vm clone guest guest2

vm-bhyve will create snapshot of guest first, then
clone that snapshot to guest1/guest2

I think it would be nice to have a feature such as

vm clone guest@snapshot guest1

for this command, clone will not create snapshot first, using the snapshot from first argc.
It will void creating useless snapshot every clone.

[Question] Firewall tips

I have a problem with pf firewall. After reboot, my virtual hosts have no access to the world. I have to manually run pfctl -f /etc/pf.conf

I think this is because bridges and tap devices are not created at the time pf initializes itself. Any tips to work around the issue?

Multiple nic's tap dev not added to bridge

relevant ifconfig:

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vm-public
        ether 02:b9:60:11:33:00
        nd6 options=1<PERFORMNUD>
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 14 priority 128 path cost 2000000
        member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 13 priority 128 path cost 2000000
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 12 priority 128 path cost 2000000
        member: bxe2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 2000
bridge2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: vm-iscsi
        ether 02:b9:60:11:33:02
        nd6 options=1<PERFORMNUD>
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: ix0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000
tap3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vmnet-demo-1-iscsi
        options=80000<LINKSTATE>
        ether 00:bd:28:85:26:03
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        Opened by PID 2538

# vm switch list
NAME            IDENT           VLAN   NAT       PORTS
public          bridge0         -      -         bxe2
internal        bridge1         4092   -         bxe2
iscsi           bridge2         -      -         ix0

relevant portion of conf file:

network0_type="virtio-net"
network0_switch="public"
network1_type="virtio-net"
network1_switch="iscsi"

relevant portion of debug file:

Oct 30 23:14:34: created network device tap2
Oct 30 23:14:34: created network device tap3
Oct 30 23:14:34: booting
Oct 30 23:14:34: bhyveload -c /dev/nmdm3A -m 1024M -d /usr/local/bhyve/vm/demo/demo demo
Oct 30 23:14:44:  [bhyve options: -c 2 -m 1024M -AHP -U 17be9253-7e88-11e5-83f9-b82a72d9c6db]
Oct 30 23:14:44:  [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,virtio-blk,/usr/local/bhyve/vm/demo/demo -s 5:0,virtio-net,tap2 -s 5:1,virtio-net,tap3]
Oct 30 23:14:44:  [bhyve console: -l com1,/dev/nmdm3A]
Oct 30 23:14:44: starting bhyve

Feature request: rename the vm name

I know I can do it by myself. just wondering if this feature will be add in the future?

like following

vm rename vm_old_name vm_new_name

Feature Request: add cpu resource limit over RCTL

bhyve may limit the guest OS in the number of CPU cores and RAM. But sometimes it is necessary to limit the guest not only by the number of cores. You can use the limitation of resources through RCTL: https://wiki.freebsd.org/SummerOfCode2012/CPULimits

Also restrictions will be added soon to the disk subsystem: https://wiki.freebsd.org/RudolfTomori/IOLimits

Can you add a resource limit function? This will require a new settings in the config file and transfer bhyve's PID as a parameter for the RCTL.

vm init can't find nmdm if it's compiled into the kernel

Same as in issue #53 : kldstat -m instead of kldstat -n in vm-common fixes it.
Also, could you pretty please update the port?
Thanks.

leftover vm lockfiles prevent vm autostart on host reboot

VM lockfiles (run.lock) are left behind on host reboot, preventing vm autostart from occuring on host reboot. This happens whether or not host is booted gracefully.

Manually removing lock file allows VM startup.

Suggested fix: Lock should be removed or ignored if VM isn't running on start (perhaps using /dev/vmm check?).

dollar make in man pages needs delete.

In man page. BASIC SETUP.
dollar mark is not needed when syntax add to rc.conf
this will not work when somebody copy & paste that to rc.conf.

BASIC SETUP
     Once vm is installed, create the directory which will store your virtual
     machine configuration and data.  This directory will be referred to as
     $vm_dir throughput this man page.

     Add the following into /etc/rc.conf

           $vm_enable="YES"
           $vm_dir="/your/vm/path"
           $vm_list=""
           $vm_delay="5"

Feature request: vm snapshot and rollback

I know for other fs than zfs, snapshot is not easy for vm-bhyve to do.
but when using zfs as backend, it is possible.
I know when I use

root@qnas:~/vm-bhyve # vm clone win2012 win2013

vm-bhyve will snapshot the source vm, and clone to dest vm.

root@qnas:~/vm-bhyve # zfs list
NAME                    USED  AVAIL  REFER  MOUNTPOINT
tank/vm                10.7G  2.62T  3.93G  /vm
tank/vm/win2012        6.76G  2.62T   104K  /vm/win2012
tank/vm/win2012/disk0  6.76G  2.62T  6.76G  -
tank/vm/win2013          76K  2.62T   100K  /vm/win2013
tank/vm/win2013/disk0     8K  2.62T  6.76G  -
root@qnas:~/vm-bhyve # zfs list -t snapshot
NAME                             USED  AVAIL  REFER  MOUNTPOINT
tank/vm/win2012@342a3d90            0      -   104K  -
tank/vm/win2012/disk0@342a3d90      0      -  6.76G  -

Base on this, I think we can have snapshot and rollback command.

tap0 device sometimes left lying around

It appears that in some circumstances, grub-bhyve will open /dev/tap0, which creates this device if it doesn't already exist. I'm not sure of the exact logic, but this seems to only happen during an install.

Because grub-bhyve is opening this itself, vm-bhyve doesn't know about it and does not remove this tap device when finished. The end result is something like this -

1. Guest with N network interfaces started in install mode
2. grub-bhyve opens tap0, creating it if it doesn't exist
3. vm-bhyve creates N tap devices for the guest
4. We now have N+1 tap devices on the host
5. vm-bhyve finishes and removes N tap devices (all that it directly created)
6. tap0 now exists and it still hanging around

need an option when create zvol backend disk.

when create a vm from template or add disk to an existing vm. if disk type is zvol/zvol-sparse, there is no way to set the volblocksize when creating. while this parameter is a little importance, affect the guest's disk performance.

here is the compare result.
using file backend.

using zvol with 8k volblocksize

using zvol with 128k volblocksize

dnsmasq fails to start up due to bridge0 absence on boot

I'm trying to use a configuration with dnsmasq. When I reboot the machine dnsmasq fails to start up because the vm service has not created the interfaces yet. Any idea how I could make this work?

Jan 11 01:27:33 aukaost dnsmasq[594]: unknown interface bridge0
Jan 11 01:27:33 aukaost dnsmasq[594]: FAILED to start up
Jan 11 01:27:33 aukaost root: /etc/rc: WARNING: failed to start dnsmasq

Unable to stop vm in Bootloader state

# vm stop ns1
/usr/local/sbin/vm: ERROR: unable to locate process id for this virtual machine
root@bhyve1:/usr/local/bhyve # vm list
NAME            GUEST           CPU    MEMORY    AUTOSTART    STATE               
ns1             freebsd         2      512M      No           Bootloader (11691)  
ns2             freebsd         1      256M      No           Stopped             
ns_master       freebsd         1      512M      Yes [1]      Running (10604)     
owncloud_demo   freebsd         2      1024M     Yes [3]      Running (6013)      
pg              freebsd         4      4096M     Yes [2]      Running (2012)      
root@bhyve1:/usr/local/bhyve # ps aux |grep 11691
root  11691  100.0  0.0  553440   3884  3  R     1:13AM     0:41.21 bhyveload -c /dev/nmdm3A -m 512M -d /usr/local/bhyve/vm/ns1/ns1 ns1

Network performance tuning tips in windows 2012?

Here is the test result using iperf.

iperf server in FreeBSD host, and from Debian 8.0/Windows 2012 iperf guest, both guests are in same bridge.
testing in TCP mode, first result comes from Debian. follows by Windows 2012.

root@gateway:~ # iperf -s -i 1 -w 256k
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  256 KByte
------------------------------------------------------------
[  4] local 10.40.0.254 port 5001 connected with 10.40.0.88 port 39817
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0- 1.0 sec   125 MBytes  1.05 Gbits/sec
[  4]  1.0- 2.0 sec   119 MBytes   998 Mbits/sec
[  4]  2.0- 3.0 sec   123 MBytes  1.03 Gbits/sec
[  4]  3.0- 4.0 sec   120 MBytes  1.00 Gbits/sec
[  4]  4.0- 5.0 sec   122 MBytes  1.02 Gbits/sec
[  4]  5.0- 6.0 sec   120 MBytes  1.01 Gbits/sec
[  4]  6.0- 7.0 sec   116 MBytes   976 Mbits/sec
[  4]  7.0- 8.0 sec   121 MBytes  1.01 Gbits/sec
[  4]  8.0- 9.0 sec   120 MBytes  1.01 Gbits/sec
[  4]  9.0-10.0 sec   118 MBytes   989 Mbits/sec
[  4]  0.0-10.0 sec  1.18 GBytes  1.01 Gbits/sec
[  5] local 10.40.0.254 port 5001 connected with 10.40.0.168 port 49831
[  5]  0.0- 1.0 sec  80.9 MBytes   679 Mbits/sec
[  5]  1.0- 2.0 sec  52.1 MBytes   437 Mbits/sec
[  5]  2.0- 3.0 sec  81.4 MBytes   683 Mbits/sec
[  5]  3.0- 4.0 sec  88.2 MBytes   739 Mbits/sec
[  5]  4.0- 5.0 sec  86.2 MBytes   723 Mbits/sec
[  5]  5.0- 6.0 sec  82.8 MBytes   694 Mbits/sec
[  5]  6.0- 7.0 sec  78.5 MBytes   658 Mbits/sec
[  5]  7.0- 8.0 sec  69.6 MBytes   584 Mbits/sec
[  5]  8.0- 9.0 sec  95.8 MBytes   803 Mbits/sec
[  5]  0.0-10.0 sec   805 MBytes   676 Mbits/sec

in UDP mode

root@gateway:~ # iperf -u -s -i 1
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 41.1 KByte (default)
------------------------------------------------------------
[  3] local 10.40.0.254 port 5001 connected with 10.40.0.88 port 44112
[ ID] Interval       Transfer     Bandwidth        Jitter   Lost/Total Datagrams
[  3]  0.0- 1.0 sec  93.3 MBytes   782 Mbits/sec   0.014 ms 1805/68344 (2.6%)
[  3]  1.0- 2.0 sec  93.2 MBytes   782 Mbits/sec   0.010 ms 1739/68218 (2.5%)
[  3]  2.0- 3.0 sec  92.6 MBytes   777 Mbits/sec   0.011 ms 2104/68135 (3.1%)
[  3]  3.0- 4.0 sec  93.3 MBytes   783 Mbits/sec   0.010 ms 1675/68221 (2.5%)
[  3]  4.0- 5.0 sec  92.3 MBytes   774 Mbits/sec   0.011 ms 2389/68217 (3.5%)
[  3]  5.0- 6.0 sec  93.0 MBytes   780 Mbits/sec   0.008 ms 1838/68141 (2.7%)
[  3]  6.0- 7.0 sec  93.0 MBytes   780 Mbits/sec   0.011 ms 1815/68156 (2.7%)
[  3]  7.0- 8.0 sec  93.9 MBytes   788 Mbits/sec   0.013 ms 1251/68252 (1.8%)
[  3]  8.0- 9.0 sec  92.5 MBytes   776 Mbits/sec   0.007 ms 2102/68052 (3.1%)
[  3]  0.0-10.0 sec   929 MBytes   780 Mbits/sec   0.012 ms 18411/681408 (2.7%)
[  3]  0.0-10.0 sec  1 datagrams received out-of-order
[  4] local 10.40.0.254 port 5001 connected with 10.40.0.168 port 54637
[  4]  0.0- 1.0 sec  20.2 MBytes   169 Mbits/sec   0.049 ms    0/14386 (0%)
[  4]  1.0- 2.0 sec  25.1 MBytes   210 Mbits/sec   0.490 ms    0/17897 (0%)
[  4]  2.0- 3.0 sec  21.7 MBytes   182 Mbits/sec   0.277 ms    0/15459 (0%)
[  4]  3.0- 4.0 sec  21.7 MBytes   182 Mbits/sec   0.097 ms    0/15500 (0%)
[  4]  4.0- 5.0 sec  23.8 MBytes   199 Mbits/sec   0.056 ms    0/16945 (0%)
[  4]  5.0- 6.0 sec  21.8 MBytes   183 Mbits/sec   0.060 ms    0/15544 (0%)
[  4]  6.0- 7.0 sec  22.1 MBytes   185 Mbits/sec   0.057 ms    0/15755 (0%)
[  4]  7.0- 8.0 sec  22.8 MBytes   191 Mbits/sec   0.050 ms    0/16249 (0%)
[  4]  8.0- 9.0 sec  22.9 MBytes   192 Mbits/sec   0.049 ms    0/16337 (0%)
[  4]  0.0-10.0 sec   223 MBytes   187 Mbits/sec   1.065 ms    0/159175 (0%)

basically, windows 2012's result is very bad.
search for some tuning tips, but all does not working.
windows driver also update to virtio-win-0.1.110.

I don't know this is relate to Windows or FreeBSD or bhyve or virtio-net driver.
any performance tuning tips?

Feature Request: custom tap ip address

need a new options to set the tap address so guest can communicate with hosts.
when a vm start, a new tap device will add automatically.
(when tap device is not specified by config)

currently, there is no way to set the ip address of that tap device.
it would be nice to have such feature.
such as by adding the following to vm config

network0_tap_ifconfig="inet 10.0.0.1 netmask 255.255.255.0"

vm-bhyve will set the newly created tap device properly.

Best way to contact author with questions?

Doing my first Bhyve install and trying to use vm-bhyve.

Don't see a mailing list. What is the best/appropriate place to ask a question?

clean init errors

On 10.2 RELEASE:

# vm init
cat: /usr/local/bhyve/vm/.config/switch: No such file or directory

Automatic MAC address assignment

Up until now vm-bhyve has worked like most other bhyve managers, and lets bhyve assign mac addresses to virtual interfaces (unless manually specified). According to documentation, bhyve uses the slot & function number, and device name to create a 'unique' mac address. There are a few problems with this:

• Guests on multiple hosts, using the same tap device number could end up getting the same mac address
• The mac address is prone to changing if a guest is restarted and gets given a different tap interface to the last time it ran. I expect this might cause problems for guests that consider an interface to be 'new' if the MAC changes (Windows?)

As such I believe the best solution is to assign a fixed mac address to each interface. Once set, the guest will see the exact same MAC permanently, regardless of device/slot/host/etc.

failure to boot Ubuntu 15.04 VM copied from VirtualBox

I've got vm-bhyve 0.9.13 (??? there's no tag for this in the repo?) installed from packages on 10.2-RELEASE. I copied this VM out of VirtualBox into a flat file. It's mostly working, but I can't get it to boot automatically in vm-bhyve. The VM is currently running Ubuntu 15.04, but it's been around a while and it's upgraded from whatever the original install was, something between 12.04 and 13.10 probably.

The three issues I'm having:

• With guest="ubuntu", grub-bhyve doesn't seem to pick up /grub/grub.cfg and boot automatically. I think this is because it's looking for /boot/grub.cfg instead, but I'm not sure.
• When I try and use grub_commands, grub_bhyve doesn't seem to pick up the configfile /grub/grub.cfg I have in there. It then hangs spinning when its redirected stdio hits EOF.
• This is all hard to debug with normal operation since vm start redirects to null and doesn't have a debug choice to redirect to a log file. So I have to use vm _run by hand (not the worst thing, but I had to figure this all out).

So currently I have to do vm start linux-apps, then vm console linux-apps and then manually type configfile /grub/grub.cfg, and everything takes off from there.

Diffs between the install vm-bhyve and github master don't seem to show anything relevant. Anything else to try before I start seriously hacking on this?

vm should not overwrite /usr/local/etc/dnsmasq.conf

Currently, vm's NAT feature assumes that vm is the only user of dnsmasq on the host. This is not necessarily true - I currently use dnsmasq to provide local DNS and DHCP services. The current behavior of vm would overwrite my dnsmasq configuration if I were to inadvertently enable NAT functionality.

Rather than assuming that the current dnsmasq.conf may be overwritten vm should instead write a sample vm-specific dnsmasq configuration file that might be included by the administrator.

vm-bhyve 'file not found' error for .config/switch

I experienced a problem when running 'vm init', the file '.config/switch' doesn't exist and vm-bhyve does not create it. I manually created the file with 'touch .config/switch' and 'vm init' completed succesfully.

thanks

Feature Request: PCI passthru

Would also be extra handy to support pci pass thru (including -S parameter for bhyveload/bhyve for wiring memory)

Windows guests ignoring stop commands

It seems that Windows can be fairly intermittent about responding to vm stop or vm stopall. Most critically, this may cause problems shutting down if a Windows guest is running; I'm not sure whether FreeBSD will wait forever or eventually kill the process.

As with most other hypervisors, I would recommend that guests are safely stopped before trying to shutdown the host, although obviously the ideal would be to have vm stopall work correctly. For Windows the best solution is to initiate a shutdown from inside the guest.

I don't believe this is an error with vm-bhyve, as we merely run a kill BHYVE-PID command. If for any reason we can't find the correct PID, we output an error, so no output from vm-bhyve suggests it is finding the process ID and sending the kill command.

I will need to see if any other bhyve users are experiencing this issue.

Support for zvols?

Hi, thank you for you your scripts - exactly what I needed.

Q: Is anyone working on adding support for zvols?

It seems like a small amount (hehe) of rework around the $_disk bits for zvols ...

Is it possble to share wireless internet on the host?

Is it possible to use NAT to allow guests to share the host wireless internet? I followed the instructions to enable the NAT with 'vm switch nat public on' and also started dnsmasq service, but DHCP did not resolve inside the VMs I tested. I did not add the physical wlan0 to the bridge, as I've read this will not work. Thanks for your time.

VM unable to access internet when using dnsmasq and a NAT switch

Hello,

I am running vm: Bhyve virtual machine managament v0.9.13
I have a switch which I have set to NAT mode:

# vm switch list
NAME            TYPE       IDENT       VLAN      NAT          PORTS
public          auto       bridge0     -         -            igb0 tap0 tap1 tap2 tap3
natswitch       auto       bridge2     -         yes          -

The VM I am using is a Debian guest and it is set to use the natswitch switch. It gets a private IP address fine but I am not able to ping out onto the internet.
I have multiple IPs on my physical interface, so I am not sure how pf's NAT will handle the NAT instruction in pf.conf. I tried changing the line

nat on igb0 from {172.16.2.0/24} to any -> (igb0)

to

nat on igb0 from {172.16.2.0/24} to any -> $EXT_IF

then loaded the ruleset into pf, but I am still not able to access the internet.
In tcpdump I see the traffic on the bridge2 interface but I do not see any traffic leaving igb0 to the ping command's destination.

Any tips would be appreciated.

Thanks,
Manas

define root partition in vm config

Hello.

I have issues with CentOS installation with hostname set in installer. It that case, root partition is named as /dev/mapper/centos_$HOSTNAME-root, resulting unbootable system. As boot.cmd is regenerated every time, this can be a problem :)

Suggestions

1. Pass root partition name from vm config
2. Do not regenerate boot.cmd, allowing users to edit it once created.

"msdos1" is hard coded as boot-device

Any Linux Virtual-Host will be started by the following command where the partition "msdos1" is hard coded.

grub-bhyve -r hd0,msdos1 -m "${vm_dir}/${_name}/device.map" -M "${_memory}" "${_name}

this should not be hard coded to msdos1, you cant expect msdos1 to be the boot-partition. There should be a config option for this.

New guest provisioned from old guest's image has identical UUID

freebsdguest was installed from a FreeBSD-10.2 ISO as per the instructions in the vm man page.

[root@hesseth /vm/vm-bhyve]# vm image create -d freebsdimage freebsdguest
Creating a compressed image, this may take some time...
Image of freebsdguest created with UUID c6feaf0d-ba26-11e5-9985-000c29330a8b
[root@hesseth /vm/vm-bhyve]# vm image provision c6feaf0d-ba26-11e5-9985-000c29330a8b
[root@hesseth /vm/vm-bhyve]# echo freebsd*/freebsd*.conf
freebsdclone/freebsdclone.conf freebsdguest/freebsdguest.conf
[root@hesseth /vm/vm-bhyve]# grep -h uuid freebsd*/freebsd*.conf
uuid="987971a8-ba23-11e5-9985-000c29330a8b"
uuid="987971a8-ba23-11e5-9985-000c29330a8b"

Should the UUID of the newly-created guest really be identical to that of the guest from which it was created? I realize that this probably doesn't have a lot of material effects on the operation of each individual guest, but the concept of uniqueness of UUIDs seems a bit abused here.

Template for Windows?

Hi!

I am planning of building a server with FreeBSD, ZFS and bhyve.

Aside of a debian VM, I want to create a Windows (Server 2012 R2) VM as well, according to the latest news this is now officially supported in bhyve!

Do you have any plans of creating a Windows template?

Doesn't support alternative block devs

Other block dev should be able to be used eg /dev/zvol/zroot/demo

disk0_name="/dev/zvol/zroot/owncloud-demo"

fails to boot because it appends that string to the vm_dir.
changing to:

disk0_name="owncloud-demo"

and creating a soft link to /dev/zvol/zroot/owncloud-demo works.

Probably there should be some sort of mechanism to pass extra params to bhyve too. Then some vm's needing some obtuse configuration can still be used in the framework.

Feature Request: add tmux/screens console support

Would anyone else like to use tmux and/or screens for console in addition to nmdm?

Capture the error message of bhyve

When using vm _run guest, the bhyve error message can be displayed in console like the following

root@bhyve:/zones/vm/debian # vm _run debian
Unable to setup memory (17)

while in vm start guest, this error message cannot be captured in vm-bhyve.log
It will be nice to display that kind of log in vm-bhyve.log.

Oct 22 05:35:19: create file /zones/vm/debian/device.map
Oct 22 05:35:19:  -> (hd0) /dev/zvol/zones/vm/debian/disk0
Oct 22 05:35:19: grub-bhyve -c /dev/nmdm1A -r hd0,msdos1 -m /zones/vm/debian/device.map -M 512M debian
Oct 22 05:35:25:  [bhyve options: -c 1 -m 512M -AHP -U 2c208d8f-77f0-11e5-b01a-0021cc60a290 -S]
Oct 22 05:35:25:  [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,virtio-blk,/dev/zvol/zones/vm/debian/disk0 -s 5:0,virtio-net,tap1 -s 6:0,passthru,0/22/0]
Oct 22 05:35:25:  [bhyve console: -l com1,/dev/nmdm1A]
Oct 22 05:35:25: starting bhyve
Oct 22 05:35:25: bhyve exited with status 1
Oct 22 05:35:25: destroying network device tap1
Oct 22 05:35:25: stopped

Cannot get DHCP to work

Hi,

First of all, thanks for vm-bhyve! It makes bhyve a breeze.
I would like to get DHCP/DNS up and running for my VMs, but I am having some difficulties. (No DHCP)
dnsmasq is running

Config: http://pastebin.com/yZDwRdcL

proxy.conf:

# guest (required)
# The type of operating system this virtual machine will use.
# Different systems often require specific steps to load the
# virtual machine, or specific bhyve options. The generic option
# does no specific loading and can be used for standard uefi guests.
#
# Valid Options: freebsd,netbsd,openbsd,centos,centos-grub,alpine
#                ubuntu,debian,windows,generic
guest="debian"

# uefi
# Tells bhyve that it should load the UEFI firmware. To use UEFI
# this can be any value other than [empty]/off/false/no/0. If
# it contains the string 'csm', the UEFI BIOS compatability (CSM)
# firmware will be used
#
# Default: no
#
uefi=""

# cpu (required)
# specify the number of cpu cores to give to the guest
#
cpu="1"

# memory (required)
# specify the amount of ram to give to the guest. This can be
# followed by M or G.
#
memory="512M"

# uuid
# This is set automatically by vm-bhyve when creating a new guest. Normally
# bhyve assigns a UUID at runtime based on host and guest name. This 
# option allows you to specify a fixed UUID that will always be used. Remove
# this or leave blank to return to the normal bhyve behaviour.
#
uuid="529f0770-a973-11e5-b0eb-10bf487fe811"

# disk0_type (required)
# This specifies the emulation type for disk0. Please note that each disk requires
# at least a type and name.
#
# Valid Options: virtio-blk,ahci-hd
#
disk0_type="virtio-blk"

# disk0_dev
# The type of device used as the backing store for this disk. The default is 'file',
# which means a sparse file is used. This file is stored in the guest's directory.
# For the zvol options, the zvol must be directly under the guest dataset.
# There is also a 'custom' option, in which case the disk name should be the full path
# to the file or device you want to use.
#
# Default: file
# Valid Options: file,zvol,sparse-zvol,custom
#
disk0_dev="sparse-zvol"

# disk0_name (required)
# The name of the file or zvol for this disk. If the device type is 'custom', it should
# be the full path to whichever device or file you want to use
#
# This value is translated to a path as follows, based on disk0_dev
#
# DEVICE TYPE        DISK NAME      BHYVE PATH USED
# file              'disk0.img' -> '$vm_dir/$name/disk0.img'
# zvol|sparse-zvol  'disk0'     -> '/dev/zvol/pool/dataset/path/guest/disk0'
# custom            '/dev/da10' -> '/dev/da10'
#
disk0_name="zvol-proxy"

# network0_type
# This specifies the emulation type to use for the first network interface.
# Networking is not required, although this field is mandatory if you do want
# to add an interface
#
# Valid Options: virtio-net
#
network0_type="virtio-net"
network1_type="virtio-net"

# network0_switch
# The name of the virtual switch to connect this interface to. When starting the
# guest, if this switch cannot be found, or no switch is specified, the interface
# is still created but will not be connected to anything.
#
# All default templates use a switch called 'public', although it's perfectly
# reasonable to use other switch names that make sense in your environment
#
network0_switch="public"
network1_switch="external"

# network0_device
# If you do not want vm-bhyve to create a new interface, but use an existing
# one, enter the interface name here. This allows you to preconfigure the network
# device in a custom configuration, then instruct vm-bhyve to use that rather
# than create all interfaces dynamically at run time.
#
network0_device=""
network1+device=""

# network0_mac
# This allows you to specify a fixed mac address for this interface inside the guest.
# Without this option, bhyve will automatically assign a mac address to the
# interface.
#
network0_mac="58:9c:fc:00:7a:25"
network1_mac="58:9c:fc:00:e8:92"

centos template not valid for Centos 6

centos os type tries to boot centos 6 with something like:

linux /vmlinuz-2.6.32-573.el6.x86_64 root=/dev/mapper/centos-root

However in standard linux fashion CentOS has decided to arbitrarily rename foundational elements for no particular reason.

For CentOS 6 default, something like

root=/dev/mapper/VolGroup-lv_root

is needed instead.

I don't know how you wish to handle this as custom grub is already supported, but it was a bit of a surprise when CentOS 6 didn't work.

Add support for manually configured bridges to vm switch

In my configuration my VMs share the host LAN interface (igb0). I have a vm switch configured (lan) which creates the bridge0 interface and binds to igb0:

switch_list="lan wan"
ports_lan="igb0"
ports_wan="igb3"

My rc.conf initially looked like this:

ifconfig_igb0="inet xx.xx.xx.5 netmask 255.255.255.0"
ifconfig_igb0_ipv6="inet6 xxxx::5 accept_rtadv"

After creating a test fbsd10 VM bound to 'lan', on boot that VM is unable to communicate with the host on IPv6 unless the host first sends a packet to the VM.

After some troubleshooting I was able to narrow this down to the bridge configuration. Per the FreeBSD handbook, the bridge0 interface must be the addressed interface. So, this is my 2nd configuration:

# new config moving IP to the bridge
cloned_interfaces="bridge0"
ifconfig_bridge0="addm igb0 up"
ifconfig_bridge0_alias0="inet xx.xx.xx.5 netmask 255.255.255.0"
ifconfig_bridge0_ipv6="inet6 xxxx::5 accept_rtadv"
ifconfig_igb0="up"
ifconfig_igb3="up"

The system comes up properly, but the vm script would not properly bind the tap interfaces to the bridges because they didn't have the vm script bridge identifiers. So, I added the bridge identifiers and manually specified both bridges:

# new config moving IP to the bridge
cloned_interfaces="bridge0 bridge1"
ifconfig_bridge0="addm igb0 up description vm-lan"
ifconfig_bridge1="addm igb3 up description vm-wan"
ifconfig_bridge0_alias0="inet xx.xx.xx.5 netmask 255.255.255.0"
ifconfig_bridge0_ipv6="inet6 xxxx::5 accept_rtadv auto_linklocal"
ifconfig_igb0="up"
ifconfig_igb3="up"

The vm script now binds tap interfaces to the bridges properly, and vm's work as expected. (I added auto_linklocal as because link-local addresses are not automatically added to bridge interfaces even with sysctl -w net.inet6.ip6.auto_linklocal=1)

However, because the vm switch configuration automatically configures bridge interfaces there is potential for breakage if switch configuration is modified using 'vm switch' commands.

It would be ideal if in the switch configuration supported manually specified bridge interfaces as well as bridge autoconfiguration. The bridge interface description could be configured by 'vm switch' to preserve existing behavior (in effect, vm would 'adopt' a manually configured bridge). In such case 'vm switch destroy' should not be able to destroy a bridge that was manually configured.

The resulting switch configuration for this feature might look something like this:

switch_list="lan wan"
bridge_lan="bridge0"
bridge_wan="bridge1"

HOW-TO: Install RHEL 6.5

I think Linux distribution is a mess.... so many distributions.
and since bhyve does't support vga output yet, it is better to have some HOW-TOs to particular Linux Distributions.
and here is some tips on How to install Redhat Enterprise Linux 6.5 under vm-bhyve.

Feature request: vm_dir per VM ?

It would be great if there was a way to create VMs in a different data store than the default vm_dir. Perhaps this could be a command line switch that is then added to the global config ? Or, all your data stores are in rc.conf and only those can be selected on the cmd line ?

Scenario: I have an ssd as the main boot drive and some VMs are living there. But, I'd like to create file storage and caching VMs on a spiny disk.

"ERROR: ZFS support requested but ZFS not available" when zfs is compiled into the kernel

Hi,

I have an FreeBSD 10.2-STABLE #0 r292683 amd64 with zfs compiled into the kernel.
On "vm init" I get "ERROR: ZFS support requested but ZFS not available".
Looking at vm-zfs, I noticed that the the script only checks, if the kernel module is loaded:
# check zfs running
kldstat -n zfs >/dev/null 2>&1
[ $? -ne 0 ] && __err "ZFS support requested but ZFS not available"

Adding "-m zfs" to the line seems to work, and should also find a loaded kernel module, but I can't test it.

Should I also (or instead) open an issue for the port on bugzilla?

Patch attached
vm-zfs.patch.txt