Is it possible to script starting up a FreeBSD VM, running a program in it, and fetching the result?

Viewed 216

I have a library that I'd like to test on FreeBSD. My CI setup doesn't have any FreeBSD systems, and adding them would be difficult, but I can spin up a VM inside my CI script. (In fact, I already do this to test on more exotic Linux kernel versions.)

For Linux, this is pretty easy: grab a pre-built machine image from some distro site, and use cloud-init to inject a first-run script, done.

Is it possible to do the same thing with FreeBSD? I'm looking for an automated way to take a standard FreeBSD machine image (e.g. downloaded from https://freebsd.org), boot it, and inject a program to run. The tricky part is that it should be entirely automated – I don't want to have to manually click through an installer ever time FreeBSD makes a new release.

3 Answers

Out of the box, there is no option like cloud-init but you could create your own image and use firstboot for example, this script is used to bootstrap a VM with saltstack in AWS:

#!/bin/sh

# KEYWORD: firstboot
# PROVIDE: set_hostname
# REQUIRE: NETWORKING
# BEFORE:  login

. /etc/rc.subr

name="set_hostname"
rcvar=set_hostname_enable
start_cmd="set_hostname_run"
stop_cmd=":"

export AWS_ACCESS_KEY_ID=key
export AWS_SECRET_ACCESS_KEY=secret
export AWS_DEFAULT_REGION=region

TAG_NAME="Salt"
INSTANCE_ID=$(/usr/local/bin/curl -s http://169.254.169.254/latest/meta-data/instance-id)
REGION=$(/usr/local/bin/curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone | sed 's/.$//')
TAG_VALUE=$(/usr/local/bin/aws ec2 describe-tags --filters "Name=resource-id,Values=${INSTANCE_ID}" "Name=key,Values=$TAG_NAME" --region ${REGION} --output=text | cut -f5)

set_hostname_run()
{
    hostname ${INSTANCE_ID}
    sysrc hostname="${INSTANCE_ID}"
    sysrc salt_minion_enable="YES"
    echo ${INSTANCE_ID} > /usr/local/etc/salt/minion_id
    pw usermod root -c "root on ${INSTANCE_ID}"
    if [ ! -z "${TAG_VALUE}" ]; then
        echo "node_type: ${TAG_VALUE}" > /usr/local/etc/salt/grains
    fi
    service salt_minion start
}

load_rc_config $name
run_rc_command "$1"

To create your own images you could use this script as a starting point: https://github.com/fabrik-red/images/blob/master/fabrik.sh#L124, more info here: https://fabrik.red/post/creating-the-image/

You can also simply install FreeBSD in Virtualbox, configure scripts related to firstboot, test and when you are happy with the results, export it, just be careful before exporting it that /firstboot exists (touch /firstboot) since after the first boot it will be removed and it could happen that after you export it if is not present it will not call the scripts.

After you have created the image you can use it multiple times, no need to create a new "custom" VM every time, it all depends on the scripts you use to bootstrap and load the scripts on the "firstboot".

I looked into this a bit more, and it turns out that it is possible, though it's quite awkward.

There are three kinds of official FreeBSD releases: pre-installed VMs, ISO installers, and USB stick installers.

The official pre-installed VM images don't offer any way to script them from outside by default. And they use the FreeBSD UFS filesystem, which isn't modifiable from any common OS. (Linux can mount UFS read-only, and has some code for read-write support, but the read-write support is disabled by default and requires a custom kernel.) So there's no easy way to programmatically modify them unless you already have a FreeBSD install.

The USB stick installers also use UFS filesystems, so that's out. So do the pre-built live CD's I found, like mfsBSD (the CD itself is iso9660, but it's just a container for a big UFS blob that gets unpacked into memory).

So that leaves the CD installers. It turns out that these actually use iso9660 for their actual file layout. And we don't need FreeBSD to work with iso9660!

So what you have to do is:

  • Download the CD installer
  • Modify the files on it to do the install without user interaction, apply some custom configuration to the new system, and then shut down
  • Use your favorite VM runner to boot up the CD with a blank hard drive image, and let it run to install FreeBSD onto that hard drive
  • Boot up the hard drive, and it will do whatever you want.

There are a ton of fiddly details that I'm glossing over, but that's the basic idea. There's also a fully-worked example here: https://github.com/python-trio/trio/pull/1509/

To "inject" your software you generally need to be able to write to the filesystem, and the most reliable way of doing that is to run the system itself. In other words, have a FreeBSD VM to create FreeBSD VMs - you can either build them locally (man 7 release), or fetch VM images from http://download.FreeBSD.org, mount their rootfs somewhere, put your software wherever you need it, and make it execute it from mounted filesystem's /etc/rc.local.

Related