Bastille is a system of managing FreeBSD jails. It aims to make setting up jails easy. AWS EC2 is a cloud infrastructure system run by Amazon.

I wanted to try Bastille on AWS but there was a lack of AWS examples on Bastille’s website. I ended up repurposing an old laptop and putting 13.1 on it and getting a VNET jail working with it locally before attempting this on AWS.

Use Case

We have an older, FreeBSD 11.1 machine on EC2 which uses openconnect and a set of python scripts to connect to Juniper Pulse and we needed a way to deploy files to it and then connect to this VPN in order to transfer the files to the target machines.

At the time I launched the machine, 11.1 was available and I was using a -STABLE AMI. Unfortunately, you can’t upgrade these easily.

When the VPN connects it takes over the network stack, setting the DNS servers in /etc/resolv.conf and changing the default gateway. This makes it difficult to transfer files out of the machine when it’s still connected to the VPN.

Enter FreeBSD VNET jails

FreeBSD VNET jails allow you to run FreeBSD’s jails with their own network stack that is independent of the host’s. Since 13.1 came out a few weeks ago, I decided to use the FreeBSD 13.1-RELEASE AMI provided by Colin Percival.

I spun up a FreeBSD 13.1-RELEASE t3a.small instance (2 GB RAM) and installed bastille on it:

pkg install bastille

Normally, I run freebsd-update fetch install however, it’s too soon after release for any updates yet. It’s always a good idea to keep the system up to date with this method, but you must make sure you choose the FreeBSD 13.1-RELEASE AMI and don’t use any STABLE ones as they will allow you to run freebsd-update on them.

Network setup

All modern Amazon EC2 instances use the ena(4) driver which is an enhanced network adapter. There’s one caveat with it: it uses jumbo frames. The jail VNET script doesn’t exactly support this yet, which I’ll get to below. First, let’s define what the network looks like:

  • ena0: 10.x.x.x (AWS VPC IP address in a RFC1918 subnet)
  • host epair interface: 192.168.50.1
  • jail epair interface: 192.168.50.2

I chose a fairly simple 192.168.50/0/24 network for the jail. It has to be different than what ena0 has.

Setting up and Configuring the jail on Bastille

When I attempted to bring up the VNET jail as per Bastille’s instructions, the jail failed to run. When Bastille starts a VNET jail, it creates 2 epair(4), and then it renames them from epair0a -> e0a_bastille0, and epair0b -> e0b_bastille0. However, the jail creation never got that far due to errors.

Then I noticed these messages in the kernel:

kernel: ena0bridge: invalid MTU: 1500(epair0a) != 9001

I was also left with epair0a and epair0b (sometimes with epair1a and epair1b as well if I tried to start the jail multiple times). The /usr/local/bin/jib script (which gets copied from /usr/share/examples/jails) does not set the mtu of the epair(4) interfaces so they default to a mtu of 1500 which does NOT match ena0 or the ena0bridge interface which uses mtu of 9001.

So, I ended up hacking /usr/local/bin/jib and editing line 309 to be:

new=$( ifconfig epair create mtu 9001 ) || return

Now that the epair(4) network devices now use the same mtu as ena0 (9001), we can start by creating the jail (I use ‘alcatraz’ as the jail name in this example):

bastille create -V alcatraz 192.168.50.2 ena0

This should set up the jail without any errors.

Run bastille edit alcatraz and above the exec.poststop += line, add this line:

exec.prestart += "ifconfig e0a_bastille0 192.168.50.1/24";

This ensures the host side of the epair(4) set of interfaces gets the IP address 192.168.50.1 (the jail will be 192.168.50.2 as above).

For some reason, the jail doesn’t properly detect the default gateway, so we need to set it in rc.conf like this:

bastille sysrc alcatraz defaultrouter=192.168.50.1

Now you can start up the jail with bastille start alcatraz

Host pf.conf and other config

Make sure you add gateway_enable="YES" in your host’s /etc/rc.conf so that it can route packets from the jail.

Here’s a very basic pf.conf I created for the machine:

ext_if="ena0"

set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo

table <jails> persist
nat on $ext_if from 192.168.50.0/24 to any -> ($ext_if:0)
rdr-anchor "rdr/*"

block in all
pass out quick keep state
pass in from 192.168.50.0/24 to any keep state
antispoof for $ext_if inet
pass in inet proto tcp from any to any port ssh flags S/SA modulate state

Add these values to /etc/rc.conf so that pf is used:

pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""
pflog_enable="YES"
pflog_logfile="/var/log/pflog"
pflog_flags=""

Run these commands to enable the jail at boot time:

sysrc bastille_enable=YES
sysrc bastille_list="alcatraz"

If you’ve followed all these steps and there have been no errors, you should now have a fully functioning 13.1-RELEASE VNET jail on your Amazon EC2 system.