tmux and bhyve

http://cft.lv/28

#virtualization

July 6th, 2020

Version 1.0

© Michael Dexter

tmux can provide console logging and command injection to bhyve virtual machines

The bench testing of virtual machines for up.bsd.lv benefits from a workbench test harness that captures console output from the loader onward and enables command injection at every stage of the virtual machine's life cycle. While OpenSSH is generally the go-to remote management tool of physical and virtual machines, it is of little use for a machine with failed or absent networking. Fortunately, the familiar tmux(1) terminal multiplexer provides exactly the desired functionality.

Console Management

tmux(1) is used by several bhyve management tools to allow an operator to attach to and detach from a virtual machine's standard I/O console. A simple example using the in-base vmrun.sh is:

Console Logging

This example can be expanded to include a log file freebsd.img.log using the pipe-pane option feature in tmux(1):

Note that the exact target window and pane in the form of vm:0.0 is needed for logging to play nicely with other tmux sessions. Thank you Tim Chase for pointing this out. Also note that > will overwrite the log while >> will append to it. The logging is quite remarkable in that it works with curses-based utilities and sometimes provides cleaner output than direct attachment.

Command Injection

The ability to inject commands is what makes the tmux(1) strategy most useful as an external test harness. The tmux-send command allows for command injection and the following will type root <ENTER> on a downloaded FreeBSD VM-IMAGE to log in:

Other example commands are:

tmux send -t vm 'sysctl hw.ncpu' Enter
tmux send -t vm 'host cft.lv | head -1' Enter
tmux send -t vm 'pkg info' Enter

To send a CTRL-t to a running command:

Putting it all together

The following script takes in the name of a disk image, boots a virtual machine using it, and begins reading the console log with tail -f. Attach to the VM in a second terminal with tmux attach -t vm and the two should be in sync. You can embellish the script by replacing the VM name vm with a variable to allow multiple virtual machines to be accessed simultaneously. If the tmux session does not auto-exit upon VM shutdown, run tmux kill-session -t vm. Note that eval is needed to properly expand the variables in the boot string.

Prepare for VM creation if needed, substituting your network interface for em0:

kldload vmm if_tap
ifconfig tap0 create
sysctl net.link.tap.up_on_open=1
ifconfig bridge0 create
ifconfig bridge0 addm em0 addm tap0
ifconfig bridge0 up

The Script

#!/bin/sh

[ $1 ] || { echo "Usage: tmux-boot.sh <vm image>" ; exit 1 ; }

echo ; echo "To connect to the VM console: tmux attach -t vm" ; echo
echo "Example commands from outside the VM:" ; echo
echo "tmux send -t vm 'root' Enter"
echo "tmux send -t vm 'sysctl hw.ncpu' Enter"
echo "tmux send -t vm 'uname -r' Enter"
echo ; echo "Also may be useful: tmux kill-session -t vm"

# Note that cat >> is used to append the log each time, rather than overwrite
bootstr="tmux new -d -s vm '/bin/sh /usr/share/examples/bhyve/vmrun.sh -m 4G -t tap0 -d $1 vm' \; pipe-pane -t vm:0.0 -o 'cat >> ${1}.log'"

echo ; echo About to run: ; echo ; echo $bootstr
echo ; echo Followed by tail -f ${1}.log ; echo
echo ; echo Starting the VM in 3 seconds... ; echo

sleep 3
eval $bootstr
tail -f ${1}.log

Copyright © 2020 Michael Dexter unless specified otherwise. Feedback and corrections welcome.