I have been using ZFS for a number of years, and have had my live filesystem moving between different Operating system. Hence my ZFS filesystem lives on, even if the operating systems get updated/changed or even replaced. In other words: my ZFS file-system is one day booted up by a Linux-kernel, some time later the same box could be running FreeBSD or Illumos.
And who cares what the name of Operating system is, those day are over, we just want to get our own job done, and your data is what is relevant in end. — With ZFS we can finally keep our filesystem even when we change our Operating system.
A little history
My first experience with ZFS was on FreeNAS v0.7 around 2009. One day while adding a disk to a zpool I made a grave mistake, I wanted to make a mirror, instead I added it as a concatenation. I immediately realized my mistake and removed the disk. That happened to be an even worse mistake. I now had a defunct ZFS system with 2 Terabytes of inaccessible files. At that time there was no way of fixing that problem under FreeNAS, which used an older version of ZFS. But it helped me learn a lot about the internal structure of ZFS. Basicly the way I got the ZFS filesystem back to life was to move the disks to a Solaris box, and scrolling back through the uberblocks to a time just before I made the fatal mistake.
I kept Solaris as the O.S. for some time, until ZFSonLinux became stable enough to use. Then I just installed Ubuntu onto one of the datasets, and booted Linux on the exact same ZFS-filesystem. Since then I have been using ZFS-on-linux as the main”bootloader” for my ZFS filesystem. By now my ZFS-filesystem had been:
- created under FreeNAS which is based on FreeBSD ,
- brought into an unusable and unrepairable state by a systems administrator error
- brought back to life under Illumos, and used there for years until
- ubuntu and zfs-on-linux was installed on a dataset
- used daily for years under Ubuntu.
Without copying or making changes to the main ZFS datasets and home-dirs.
Lately I stumbled into a bug in ZFS-on-Linux github.com/zfsonlinux/zfs/issues/4122, which I am happy to say is fixed, but that made me adopt a back-up-plan – NAMELY – give myself the option to boot into FreeBSD. Which is what this post is all about.
To Day
My current system is running Ubuntu-14.04.3 after this: the system will run FreeBSD-10.2. The only thing tricky with standard FreeBSD is that they use their own boot-loader. We will avoid that and just use GRUB2, the ZFS enabled version from ubuntu in this case, but the one from PC-BSD and the version from Illumos should work too.
I would like to have the option to boot the system into either Linux or FreeBSD. To do that, we just have to add that functionality to /etc/grub.d/09-zfs which will generate the magic grub-lines that can boot FreeBSD too.
The plan for todays work is:
- Install FreeBSD into a dataset
- Create a /boot/zfs/zpool.cache so that FreeBSD will recognize the ZFS pool
- Remove ZFS internal mount-points
- Teach GRUB2 how to boot into a FreeBSD ZFS-data set
- Use the same SSH credentials to avoid a lot of fuss
Most of the work is done on a live system system with very little down-time.
Install FreeBSD into a dataset
This little shell-script will do the majority of the work.
#!/bin/bash # # install FreeBSD onto a ZFS dataset VERSION=10.2-RELEASE FROM=ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/$VERSION # assuming only one ZPOOL POOL=$(zpool status | awk '/pool:/ {print $NF}') DEST=$POOL/ROOT/FreeBSD-$VERSION R=/$DEST echo "# Install FreeBSD-$VERSION to $R" zfs create $DEST for file in base.txz lib32.txz kernel.txz doc.txz ports.txz src.txz; do wget $FROM/$file done for file in base.txz lib32.txz kernel.txz doc.txz ports.txz src.txz; do tar --unlink -xpJf $file -C $R done echo 'zfs_load="YES"' >> $R/boot/loader.conf echo 'vfs.root.mountfrom="zfs:zroot"' >> $R/boot/loader.conf echo 'zfs_enable="YES"' >> $R/etc/rc.conf echo 'hostname="freebsd"' >> $R/etc/rc.conf echo 'ifconfig_re0="dhcp"' >> $R/etc/rc.conf echo 'sshd_enable="YES"' >> $R/etc/rc.conf echo 'syslogd_flags="-ss"' >> $R/etc/rc.conf touch $R/etc/fstab sync
Create a /boot/zfs/zpool.cache so that FreeBSD will recognize the ZFS pool
Most likely the zpool.cache from Linux is not compatible with FreeBSD (linux/bsd, gcc/clang, device-naming). Regarding the device-naming, tweaking udev on Linux might be able to fix it part of the way, like we discuss in github.com/zfsonlinux/grub/issues/5 (I am storepeter).
The easiest way to fix this though is to boot FreeBSD from an USB-stick, start a shell, and create a zpool.cache manually this way
zpool import -o altroot=/tmp -o cachefile=/tmp/zpool.cache zhome cp /tmp/zpool.cache /tmp/zhome/ROOT/FreeBSD*/boot zpool export zhome
Remove ZFS internal mount-points
One thing I really don’t like about ZFS, is the facility to set mount-points inside ZFS. Hence if you import ZFS filesystem, have also imported the mount-points. and mounting Linux-root-fileystem on / for FreeBSD, is BAD.
To get rid of all the explicit mount-points I make the following changes to /etc/rc.d/zfs on FreeBSD (Recursively inherit mount-point for all datasets under /ROOT)
zfs_start_main() { #start change for pool in $(zpool list -H | cut -f1); do zfs inherit mountpoint -r $pool/ROOT done #end change zfs mount -va zfs share -a if [ ! -r /etc/zfs/exports ]; then touch /etc/zfs/exports fi }
Teach GRUB2 how to boot into a FreeBSD ZFS-data set
I have been using ZFS-on-Linux for a long time and have made my own grub configuration so I can boot into various Linux versions. Kind of my own home-grown boot-environments, but much simpler than the Solaris/Illumos/PCBSD beadm
GRUB2 uses a number of shell-script to generate the grub.cfg that the bootloader uses when booting. Below is /etc/grub.d/09-zfs which makes it possible to boot into all the root-filesystems available on the zpool
#! /bin/sh # Create boot menu for zfs root systems choices are: # if current root is Linux: menu-point for all kernel-versions on this dataset # if current root is FreeBSD: menu-point for FreeBSD on here # for all Linux root-filsystems: menu-point with the newest kernel on that root filesystem, # for all FreeBSD root-filsystems: menu-point for FreeBSD on here # current root will be default # if [ -d /usr/share/grub ]; then # Linux prefix="/usr" else # FreeBSD prefix="/usr/local" PATH=/usr/local/bin:/usr/local/sbin:$PATH export PATH fi exec_prefix="${prefix}" datarootdir="${prefix}/share" . "${datarootdir}/grub/grub-mkconfig_lib" export TEXTDOMAIN=grub export TEXTDOMAINDIR="${datarootdir}/locale" CLASS="--class gnu-linux --class gnu --class s" HOSTNAME=`hostname` BOOT=/boot VERBOSE=1 vecho() { if [ x"$VERBOSE" != x ]; then echo "# $*" >/dev/stderr; fi; } vvecho() { if [ x"$VERBOSE" = x2 ]; then echo "# $*" >/dev/stderr; fi; } show() { (echo -n "#";for show_index in $*;do echo -n " ";eval echo -n "$show_index=\$show_index";done;echo;)>/dev/stderr; } vshow() { if [ x"$VERBOSE" != x ]; then show $*; fi; } vvshow() { if [ x"$VERBOSE" = x2 ]; then show $*; fi; } # menues to boot */ROOT/* with "highest" kernels available on /boot # sort the boot entries for host "z" like this # z/ROOT/ first. # z/ROOT/ubuntu-14.04-z # z/ROOT/ubuntu-13.10-z # z/ROOT/ubuntu-13.04-z ROOT_ON_ZFS=`df -T / | awk '$2 == "zfs" && $7 == "/" {print $1}'` get_mountpoint() { if [ $1 = $ROOT_ON_ZFS ]; then echo / else zfs list -H $1 | awk '{print $NF}' fi #zfs get -H mountpoint $1 | awk '{print $3}' } # root filestem might no be mounted get_root_filesystem_list() { list_us="" list_not_us="" list=`zfs list | awk '/^[a-zA-Z0-9]*\/ROOT\// { print $1}'| grep -v @ |sort` for z in ${list}; do if [ x$z = x${ROOT_ON_ZFS} ]; then continue # skip current root echo -n "$z " else r=$(get_mountpoint $z) if [ $r = / ]; then # mountpoint is / but not mounted zfs inherit mountpoint $z zfs mount $z fi if [ noauto = $(sudo zfs get -H canmount zhome/ROOT/10.2-RELEASE-p10-up-20151228_181943 | cut -f3) ]; then zfs mount $z fi # root filesystem that end in -myhostname has priority if [ `basename $z -$HOSTNAME` != `basename $z` ]; then list_us="$z $list_us" else list_not_us="$z $list_not_us" fi fi done echo $list_us $list_not_us } get_mount_dev() { df $1| cut -f1 -d' ' | tail -1 } get_pool() { echo $1 | cut -f1 -d/ } get_ds() { echo $1 | cut -f2-99 -d/ } get_rescue() { boot_dev=$(get_mount_dev /boot) for d in /boot-sd[a-z][0-9] /root-sd[a-z][0-9]/boot; do if [ $boot_dev != $(get_mount_dev $d) ]; then echo -n "$d " fi done } # two scenarios # /boot on ZFS, requires grub to understand the current ZFS version. # /boot is not on ZFS, The only option if ZFS is ahead of the version in GRUB linux_entry() { z=$1 ds=$1 r=`get_mountpoint $ds` kversion=$2 boot=$3 vshow "linux_entry z kversion boot" linux=$boot/vmlinuz-${kversion} if [ ! -f $linux ]; then vecho "skipping no linux=$linux" return fi initrd=$boot/initrd.img-${kversion} if [ ! -f $initrd ]; then vecho "skipping no initrd=$initrd" return fi if [ -f $r/etc/lsb-release ]; then . $r/etc/lsb-release OS=$DISTRIB_DESCRIPTION else OS="Unknown" fi echo "ZFS ${ds} - ${OS} - ${kversion}" >&2 title="ZFS ${ds} ${OS} ${kversion} $boot" rpool=`echo $ds | cut -f1 -d/` CMDLINE_ZFS="boot=zfs rpool=$rpool bootfs=$ds" linux=`make_system_path_relative_to_its_root $linux` initrd=`make_system_path_relative_to_its_root $initrd` ZFS_DEV=`grub-probe --target=device $boot` vecho "grub-probe $boot --target=device returned $ZFS_DEV" echo "menuentry '$title' --class ubuntu --class gnu-linux --class gnu --class os) {" echo " insmod gzio" vshow ZFS_DEV prepare_grub_to_access_device $ZFS_DEV | sed -e "s/^/ /" echo " echo \"Loading $OS $kversion from ZFS $ds\"" echo " linux $linux $CMDLINE_ZFS $GRUB_CMDLINE_LINUX" echo " initrd $initrd" echo "}" } freebsd_entry() { z=$1 pool=$(get_pool $1) ds=$(get_ds $1) r=$(get_mountpoint $1) version=$(basename $ds | sed 's/-/_/g') vshow freebsd_entry z ds r version OS="FreeBSD" echo "ZFS ${1} - ${OS}" >&2 title="ZFS ${1} ${OS}" ZFS_DEV=`grub-probe --target=device $r` echo "menuentry '$title' {" echo " insmod gzio" #echo " set root='hd0,msdos7'" echo " insmod zfs" echo " search --no-floppy -s -l zhome" echo " echo \"Loading $OS from ZFS $1, edit to append -s or -v to line below\"" echo " kfreebsd /$ds/@/boot/kernel/kernel" echo " kfreebsd_loadenv /$ds/@/boot/device.hints" echo " kfreebsd_module /$ds/@/boot/zfs/zpool.cache type=/boot/zfs/zpool.cache" echo " set kFreeBSD.vfs.root.mountfrom=zfs:$pool/$ds" echo " set kFreeBSD.module_path=\"/boot/kernel;/boot/modules\"" echo "# unfortunately does the kernel not use the above when booting, so you have to include all depencies in /boot/loader.conf" VARS="" # load modules and set variables for boot for i in $(cat $r/boot/loader.conf* | grep -v "^#" | grep = | sed 's/#.*$//');do name=$(echo $i | cut -f1 -d= | sed 's/\./_/g') value=$(echo $i | cut -f2 -d=) vvshow i name value if eval [ x\$version_$name != x ]; then vecho $version_$name already set else vvecho new $version_$name eval $version_$name=$value m=$(echo $i | grep '_load="YES"' | sed 's/_load.*$//') #if [ x$m != x ]; then if [ $m ]; then vecho load module $m if [ -f $r/boot/kernel/$m.ko ]; then echo " kfreebsd_module_elf /$ds/@/boot/kernel/$m.ko" elif [ -f $r/boot/modules/$m.ko ]; then echo " kfreebsd_module_elf /$ds/@/boot/modules/$m.ko" fi else vecho variable $i # not a module - set it as a variable VARS="$VARS $i" fi fi done echo " set kFreeBSD.bootfile=\"kernel\"" echo " set kFreeBSD.kernel=\"kernel\"" echo " set kFreeBSD.kernel_options=\"\"" echo " set kFreeBSD.kernelname=\"/boot/kernel/kernel\"" echo " set kFreeBSD.module_path=\"/boot/kernel;/boot/modules\"" for v in $VARS; do echo " set kFreeBSD.$v" done echo "}" } ### current root all kernels z=$ROOT_ON_ZFS vshow ZFS_ROOTS vshow ROOT_ON_ZFS r=/ if [ -d $r/lib/modules ]; then # we want newest kernel first for t in `ls -t $r/lib/modules/*/*order`;do d=`dirname $t` kvers=`basename $d` vecho "Found $z has modules: $kvers" linux_entry $z $kvers /boot linux_entry $z $kvers $z/boot if [ $(get_mount_dev /boot) != $(get_mount_dev $r/boot) ]; then linux_entry $z $kvers $r/boot fi done elif [ -f $r/etc/freebsd-update.conf ]; then freebsd_entry $z fi # for all roots newest kernel for z in $(get_root_filesystem_list); do r=$(get_mountpoint $z) vecho "Creating boot entires for $z root=$r" if [ $r = "/" ]; then echo "zfs inherit mountpoint $z" echo "zfs mount $z" continue fi r=$(get_mountpoint $z) if [ $r = "/" ]; then continue fi if [ -d $r/lib/modules ]; then # we want highest kernel first kvers=`ls -t $r/lib/modules| head -1` linux_entry $z $kvers /boot if [ $(get_mount_dev /boot) != $(get_mount_dev $r/boot) ]; then linux_entry $z $kvers $r/boot fi elif [ -f $r/etc/freebsd-update.conf -a -d $r/boot/kernel ]; then freebsd_entry $z fi done
That was quite a number of script lines, but straight forward, and it is easy to test just run the script and see if it generates what you expect.
Install grub2 from FreeBSD
I already have GRUB2 installed on the harddrives, This one was installed from Ubuntu.
And it will happily boot both Ubuntu and FreeBSD. But if I have to update the Grub menus I have to reboot into Ubuntu and run update-grub.
I am normally running my ZFS-systems as mirrors, so it would be handy to have one disk boot with a Linux-based GRUB2, and the other with a FreeBSD-based GRUB2. So that we we will do. Assuming that we have booted the system into FreeBSD.
-
- install the GRUB2 package
pkg install grub2-pcbsd
-
- install the GRUB2 bootloader to the disk
grub-install –modules=”part_gpt zfs” /dev/ada0
-
- add script to create menus /usr/local/etc/grub.d/09-zfs
The 09-zfs script above, will also work from within FreeBSD
- update /boot/grub/grub.cfg
grub-mkconfig -o /boot/grub/grub.cfg
Keep ssh credentials
I copy over the ssh credential, so ssh-wise it looks like the same host.
rsync -av --exclude "ssh*_config" /zhome/ROOT/ubuntu-14.04.3/etc/ssh /etc/