Installing FreeBSD into a live ZFS filesystem from f.ex. Linux

Posted on February 9, 2016 by peter

I have been using ZFS for a number of years, and have had my live filesystem moving between different Operating system. Hence my ZFS filesystem lives on, even if the operating systems get updated/changed or even replaced. In other words: my ZFS file-system is one day booted up by a Linux-kernel, some time later the same box could be running FreeBSD or Illumos.

And who cares what the name of Operating system is, those day are over, we just want to get our own job done, and your data is what is relevant in end. — With ZFS we can finally keep our filesystem even when we change our Operating system.

A little history

My first experience with ZFS was on FreeNAS v0.7 around 2009. One day while adding a disk to a zpool I made a grave mistake, I wanted to make a mirror, instead I added it as a concatenation. I immediately realized my mistake and removed the disk. That happened to be an even worse mistake. I now had a defunct ZFS system with 2 Terabytes of inaccessible files. At that time there was no way of fixing that problem under FreeNAS, which used an older version of ZFS. But it helped me learn a lot about the internal structure of ZFS. Basicly the way I got the ZFS filesystem back to life was to move the disks to a Solaris box, and scrolling back through the uberblocks to a time just before I made the fatal mistake.

I kept Solaris as the O.S. for some time, until ZFSonLinux became stable enough to use. Then I just installed Ubuntu onto one of the datasets, and booted Linux on the exact same ZFS-filesystem. Since then I have been using ZFS-on-linux as the main”bootloader” for my ZFS filesystem. By now my ZFS-filesystem had been:

  • created under FreeNAS which is based on FreeBSD ,
  • brought into an unusable and unrepairable state by a systems administrator error
  • brought back to life under Illumos, and used there for years until
  • ubuntu and zfs-on-linux was installed on a dataset
  • used daily for years under Ubuntu.

Without copying or making changes to the main ZFS datasets and home-dirs.

Lately I stumbled into a bug in ZFS-on-Linux github.com/zfsonlinux/zfs/issues/4122, which I am happy to say is fixed, but that made me adopt a back-up-plan – NAMELY – give myself the option to boot into FreeBSD. Which is what this post is all about.

To Day

My current system is running Ubuntu-14.04.3 after this: the system will run FreeBSD-10.2. The only thing tricky with standard FreeBSD is that they use their own boot-loader. We will avoid that and just use GRUB2, the ZFS enabled version from ubuntu in this case, but the one from PC-BSD and the version from Illumos should work too.

I would like to have the option to boot the system into either Linux or FreeBSD. To do that, we just have to add that functionality to /etc/grub.d/09-zfs which will generate the magic grub-lines that can boot FreeBSD too.

The plan for todays work is:

  • Install FreeBSD into a dataset
  • Create a /boot/zfs/zpool.cache so that FreeBSD will recognize the ZFS pool
  • Remove ZFS internal mount-points
  • Teach GRUB2 how to boot into a FreeBSD ZFS-data set
  • Use the same SSH credentials to avoid a lot of fuss

Most of the work is done on a live system system with very little down-time.

Install FreeBSD into a dataset

This little shell-script will do the majority of the work.

#!/bin/bash
#
# install FreeBSD onto a ZFS dataset
VERSION=10.2-RELEASE
FROM=ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/$VERSION
# assuming only one ZPOOL
POOL=$(zpool status | awk '/pool:/ {print $NF}')
DEST=$POOL/ROOT/FreeBSD-$VERSION
R=/$DEST
echo "# Install FreeBSD-$VERSION to $R"
zfs create $DEST
 
for file in base.txz lib32.txz kernel.txz doc.txz ports.txz src.txz; do
    wget $FROM/$file
done
for file in base.txz lib32.txz kernel.txz doc.txz ports.txz src.txz; do
    tar --unlink -xpJf $file -C $R
done
 
echo 'zfs_load="YES"' >> $R/boot/loader.conf
echo 'vfs.root.mountfrom="zfs:zroot"' >> $R/boot/loader.conf
 
echo 'zfs_enable="YES"' >> $R/etc/rc.conf
echo 'hostname="freebsd"' >> $R/etc/rc.conf
echo 'ifconfig_re0="dhcp"' >> $R/etc/rc.conf
echo 'sshd_enable="YES"' >> $R/etc/rc.conf
echo 'syslogd_flags="-ss"' >> $R/etc/rc.conf
 
touch $R/etc/fstab
sync

Create a /boot/zfs/zpool.cache so that FreeBSD will recognize the ZFS pool

Most likely the zpool.cache from Linux is not compatible with FreeBSD (linux/bsd, gcc/clang, device-naming). Regarding the device-naming, tweaking udev on Linux might be able to fix it part of the way, like we discuss in github.com/zfsonlinux/grub/issues/5 (I am storepeter).

The easiest way to fix this though is to boot FreeBSD from an USB-stick, start a shell, and create a zpool.cache manually this way

zpool import -o altroot=/tmp -o cachefile=/tmp/zpool.cache zhome
cp /tmp/zpool.cache /tmp/zhome/ROOT/FreeBSD*/boot
zpool export zhome

Remove ZFS internal mount-points

One thing I really don’t like about ZFS, is the facility to set mount-points inside ZFS. Hence if you import ZFS filesystem, have also imported the mount-points. and mounting Linux-root-fileystem on / for FreeBSD, is BAD.

To get rid of all the explicit mount-points I make the following changes to /etc/rc.d/zfs on FreeBSD (Recursively inherit mount-point for all datasets under /ROOT)

zfs_start_main()
{
#start change
        for pool in $(zpool list -H | cut -f1); do
                zfs inherit mountpoint -r $pool/ROOT
        done
#end change
        zfs mount -va
        zfs share -a
        if [ ! -r /etc/zfs/exports ]; then
                touch /etc/zfs/exports
        fi
}

Teach GRUB2 how to boot into a FreeBSD ZFS-data set

I have been using ZFS-on-Linux for a long time and have made my own grub configuration so I can boot into various Linux versions. Kind of my own home-grown boot-environments, but much simpler than the Solaris/Illumos/PCBSD beadm

GRUB2 uses a number of shell-script to generate the grub.cfg that the bootloader uses when booting. Below is /etc/grub.d/09-zfs which makes it possible to boot into all the root-filesystems available on the zpool

#! /bin/sh
# Create boot menu for zfs root systems choices are:
# if current root is Linux: menu-point for all kernel-versions on this dataset
# if current root is FreeBSD: menu-point for FreeBSD on here
# for all Linux root-filsystems: menu-point  with the newest kernel on that root filesystem,
# for all FreeBSD root-filsystems: menu-point for FreeBSD on here
# current root will be default
#
if [ -d /usr/share/grub ]; then # Linux
	prefix="/usr"
else # FreeBSD
	prefix="/usr/local"
	PATH=/usr/local/bin:/usr/local/sbin:$PATH
	export PATH
fi
exec_prefix="${prefix}"
datarootdir="${prefix}/share"

. "${datarootdir}/grub/grub-mkconfig_lib"

export TEXTDOMAIN=grub
export TEXTDOMAINDIR="${datarootdir}/locale"

CLASS="--class gnu-linux --class gnu --class s"

HOSTNAME=`hostname`
BOOT=/boot
VERBOSE=1
 vecho() { if [ x"$VERBOSE" != x ]; then echo "# $*" >/dev/stderr; fi; }
vvecho() { if [ x"$VERBOSE" = x2 ]; then echo "# $*" >/dev/stderr; fi; }
  show() { (echo -n "#";for show_index in $*;do echo -n " ";eval echo -n "$show_index=\$show_index";done;echo;)>/dev/stderr; }
 vshow() { if [ x"$VERBOSE" != x ]; then show $*; fi; }
vvshow() { if [ x"$VERBOSE" = x2 ]; then show $*; fi; }

# menues to boot */ROOT/* with "highest" kernels available on /boot
# sort the boot entries for host "z" like this
# z/ROOT/ first.
# z/ROOT/ubuntu-14.04-z
# z/ROOT/ubuntu-13.10-z
# z/ROOT/ubuntu-13.04-z
ROOT_ON_ZFS=`df -T / | awk '$2 == "zfs" && $7 == "/" {print $1}'`
get_mountpoint()
{
	if [ $1 = $ROOT_ON_ZFS ]; then
		echo /
	else
		zfs list -H $1 | awk '{print $NF}'
	fi
	#zfs get -H mountpoint $1 | awk '{print $3}'
}
# root filestem might no be mounted
get_root_filesystem_list()
{
	list_us=""
	list_not_us=""
	list=`zfs list | awk '/^[a-zA-Z0-9]*\/ROOT\// { print $1}'| grep -v @ |sort`
	for z in ${list}; do
		if [ x$z = x${ROOT_ON_ZFS} ]; then
			continue	# skip current root
			echo -n "$z "
		else
			r=$(get_mountpoint $z)
			if [ $r = / ]; then # mountpoint is / but not mounted
				zfs inherit mountpoint $z
				zfs mount $z
			fi
			if [ noauto = $(sudo zfs get -H  canmount  zhome/ROOT/10.2-RELEASE-p10-up-20151228_181943 | cut -f3) ]; then
				zfs mount $z
			fi
			# root filesystem that end in -myhostname has priority
			if [ `basename $z -$HOSTNAME` != `basename $z` ]; then
				list_us="$z $list_us"
			else
				list_not_us="$z $list_not_us"
			fi
		fi
	done
	echo $list_us $list_not_us
}
get_mount_dev()
{
	df $1| cut -f1 -d' ' | tail -1
}
get_pool()
{
	echo $1 | cut -f1 -d/
}
get_ds()
{
	echo $1 | cut -f2-99 -d/
}
get_rescue()
{
	boot_dev=$(get_mount_dev /boot)
	for d in /boot-sd[a-z][0-9] /root-sd[a-z][0-9]/boot; do
		if [ $boot_dev != $(get_mount_dev $d) ]; then
			echo -n "$d "
		fi
	done
}

# two scenarios
# /boot on ZFS, requires grub to understand the current ZFS version.
# /boot is not on ZFS, The only option if ZFS is ahead of the version in GRUB
linux_entry()
{
	z=$1
	ds=$1
	r=`get_mountpoint $ds`
	kversion=$2
	boot=$3
	vshow "linux_entry z kversion boot"
	linux=$boot/vmlinuz-${kversion}
	if [ ! -f $linux ]; then
		vecho "skipping no linux=$linux"
		return
	fi
	initrd=$boot/initrd.img-${kversion}
	if [ ! -f $initrd ]; then
		vecho "skipping no initrd=$initrd"
		return
	fi
	if [ -f $r/etc/lsb-release ]; then
		. $r/etc/lsb-release
		OS=$DISTRIB_DESCRIPTION
	else
		OS="Unknown"
	fi
	echo "ZFS ${ds} - ${OS} - ${kversion}" >&2
	title="ZFS ${ds} ${OS} ${kversion} $boot"
	rpool=`echo $ds | cut -f1 -d/`
	CMDLINE_ZFS="boot=zfs rpool=$rpool bootfs=$ds"
	linux=`make_system_path_relative_to_its_root $linux`
	initrd=`make_system_path_relative_to_its_root $initrd`
	ZFS_DEV=`grub-probe --target=device $boot`
	vecho "grub-probe $boot --target=device returned $ZFS_DEV"

	echo "menuentry '$title' --class ubuntu --class gnu-linux --class gnu --class os) {"
	echo "	insmod gzio"
	vshow ZFS_DEV
      	prepare_grub_to_access_device $ZFS_DEV | sed -e "s/^/	/"
	echo "	echo \"Loading $OS $kversion from ZFS $ds\""
	echo "	linux $linux $CMDLINE_ZFS $GRUB_CMDLINE_LINUX"
	echo "	initrd $initrd"
	echo "}"
}

freebsd_entry()
{
	z=$1
	pool=$(get_pool $1)
	ds=$(get_ds $1)
	r=$(get_mountpoint $1)
	version=$(basename $ds | sed 's/-/_/g')
	vshow freebsd_entry z ds r version
	OS="FreeBSD"
	echo "ZFS ${1} - ${OS}" >&2
	title="ZFS ${1} ${OS}"
	ZFS_DEV=`grub-probe --target=device $r`

	echo "menuentry '$title' {"
	echo "	insmod gzio"
	#echo "	set root='hd0,msdos7'"
	echo "	insmod zfs"
	echo "	search --no-floppy -s -l zhome"
	echo "	echo \"Loading $OS from ZFS $1, edit to append -s or -v to line below\""
	echo "	kfreebsd /$ds/@/boot/kernel/kernel"
	echo "	kfreebsd_loadenv /$ds/@/boot/device.hints"
	echo "	kfreebsd_module /$ds/@/boot/zfs/zpool.cache type=/boot/zfs/zpool.cache"
	echo "	set kFreeBSD.vfs.root.mountfrom=zfs:$pool/$ds"
	echo "	set kFreeBSD.module_path=\"/boot/kernel;/boot/modules\""
	echo "# unfortunately does the kernel not use the above when booting, so you have to include all depencies in /boot/loader.conf"
	VARS=""
# load modules and set variables for boot
	for i in $(cat $r/boot/loader.conf* | grep -v "^#" | grep = | sed 's/#.*$//');do
		name=$(echo $i | cut -f1 -d= | sed 's/\./_/g')
		value=$(echo $i | cut -f2 -d=)
		vvshow i name value
        	if eval [ x\$version_$name !=  x ]; then
                	vecho $version_$name already set
        	else
                	vvecho new $version_$name
                	eval $version_$name=$value
			m=$(echo $i | grep '_load="YES"' | sed 's/_load.*$//')
			#if [ x$m != x ]; then
			if [ $m ]; then
                		vecho load module $m
				if [ -f $r/boot/kernel/$m.ko ]; then
					echo "	kfreebsd_module_elf /$ds/@/boot/kernel/$m.ko"
				elif [ -f $r/boot/modules/$m.ko ]; then
					echo "	kfreebsd_module_elf /$ds/@/boot/modules/$m.ko"
        			fi
			else
                		vecho variable $i
				# not a module - set it as a variable
				VARS="$VARS $i"
			fi
		fi
	done
	echo "	set kFreeBSD.bootfile=\"kernel\""
	echo "	set kFreeBSD.kernel=\"kernel\""
	echo "	set kFreeBSD.kernel_options=\"\""
	echo "	set kFreeBSD.kernelname=\"/boot/kernel/kernel\""
	echo "	set kFreeBSD.module_path=\"/boot/kernel;/boot/modules\""
	for v in $VARS; do
		echo "	set kFreeBSD.$v"
	done
	echo "}"
}

### current root all kernels
z=$ROOT_ON_ZFS
vshow ZFS_ROOTS
vshow ROOT_ON_ZFS
r=/
if [ -d $r/lib/modules ]; then
# we want newest kernel first
	for t in `ls -t $r/lib/modules/*/*order`;do
		d=`dirname $t`
		kvers=`basename $d`
		vecho "Found $z has modules: $kvers"
		linux_entry $z $kvers /boot
		linux_entry $z $kvers $z/boot
		if [ $(get_mount_dev /boot) != $(get_mount_dev $r/boot) ]; then
			linux_entry $z $kvers $r/boot
		fi
	done
elif [ -f $r/etc/freebsd-update.conf ]; then
	freebsd_entry $z
fi

# for all roots newest kernel
for z in $(get_root_filesystem_list); do
	r=$(get_mountpoint $z)
	vecho "Creating boot entires for $z root=$r"
	if [ $r = "/" ]; then
		echo "zfs inherit mountpoint $z"
		echo "zfs mount $z"
		continue
	fi
	r=$(get_mountpoint $z)
	if [ $r = "/" ]; then
		continue
	fi
	if [ -d $r/lib/modules ]; then
# we want highest kernel first
		kvers=`ls -t $r/lib/modules| head -1`
		linux_entry $z $kvers /boot
		if [ $(get_mount_dev /boot) != $(get_mount_dev $r/boot) ]; then
			linux_entry $z $kvers $r/boot
		fi
	elif [ -f $r/etc/freebsd-update.conf -a -d $r/boot/kernel ]; then
		freebsd_entry $z
	fi
done

That was quite a number of script lines, but straight forward, and it is easy to test just run the script and see if it generates what you expect.

Install grub2 from FreeBSD

I already have GRUB2 installed on the harddrives, This one was installed from Ubuntu.
And it will happily boot both Ubuntu and FreeBSD. But if I have to update the Grub menus I have to reboot into Ubuntu and run update-grub.

I am normally running my ZFS-systems as mirrors, so it would be handy to have one disk boot with a Linux-based GRUB2, and the other with a FreeBSD-based GRUB2. So that we we will do. Assuming that we have booted the system into FreeBSD.

    • install the GRUB2 package
pkg install grub2-pcbsd
    • install the GRUB2 bootloader to the disk

grub-install –modules=”part_gpt zfs” /dev/ada0

    • add script to create menus /usr/local/etc/grub.d/09-zfs

The 09-zfs script above, will also work from within FreeBSD

  • update /boot/grub/grub.cfg
grub-mkconfig -o /boot/grub/grub.cfg

Keep ssh credentials

I copy over the ssh credential, so ssh-wise it looks like the same host.

rsync -av --exclude "ssh*_config" /zhome/ROOT/ubuntu-14.04.3/etc/ssh /etc/

That was all for now

This entry was posted in FreeBSD, Linux, ZFS. Bookmark the permalink.