My Christmas present for myself this season were 10Gbe Ethernet.  I had no idea what to get so I bought the cheapest I could find on E-bay, $11 each delivered. These were IBM branded OCe10102 cards made by Emulex.

Two of the Emulex cards came with 2 SFP+ transceivers so I have the hardware to make two independent fiber connections, fiber-patch-cable is available on Amazon for $5 new delivered.

I also bought 4 DAC-cables for $2 each.The only thing left now is to install-connect-configure-enjoy.

Unfortunately it didn’t just work, hence this BLOG post with the story so far.

The simplest way to fix a software problem is to buy some more hardware.  And I were in need of a two low-profile brackets for the 10Gbe Ethernet boards, so I had a good excuse to buy some different brand 10Gbe Ethernet cards this time I payed $15 each. These we Solarflare SFB5122F

 

 

Emulex OCe10102

• The Emulex 10Gb-Ethernet-adapters are 8 lane PCI-Express, but unfortunately they are a little too wide so they need to go into a 16 lane slot, or some surgery is needed, this is/was another good reason to try another brand.
• lshw identifies the board as: Emulex OneConnect OCe10100/OCe10102 Series 10 GbE
• According to dmesg the Emulex 10Gb-Ethernet-adapters are found as be2net but the interface do not become active when connected, output of dmesg below

dmesg | grep be2net
[    1.956424] be2net 0000:01:00.0: be2net version is 12.0.0.0
[    1.965890] be2net 0000:01:00.0: PCIe error reporting enabled
[    2.347419] be2net 0000:01:00.0: FW config: function_mode=0x20003, function_caps=0x7
[    2.555421] be2net 0000:01:00.0: Max: txqs 1, rxqs 5, rss 4, eqs 16, vfs 0
[    2.573792] be2net 0000:01:00.0: Max: uc-macs 30, mc-macs 64, vlans 15
[    2.627490] be2net 0000:01:00.0: enabled 1 MSI-x vector(s) for NIC
[    2.785022] be2net 0000:01:00.0: created 1 TX queue(s)
[    2.823423] be2net 0000:01:00.0: created 1 RX queue(s)
[    2.960433] be2net 0000:01:00.0: FW version is 4.6.142.10
[    2.967533] be2net 0000:01:00.0: HW Flow control - TX:1 RX:1
[    2.975427] be2net 0000:01:00.0: Adapter does not support HW error recovery
[    2.975574] be2net 0000:01:00.0: Emulex OneConnect: PF vNIC-2 port 0
[    2.975591] be2net 0000:01:00.1: be2net version is 12.0.0.0
[    2.975790] be2net 0000:01:00.1: PCIe error reporting enabled
[    3.243412] be2net 0000:01:00.1: FW config: function_mode=0x20003, function_caps=0x7
[    3.387416] be2net 0000:01:00.1: Max: txqs 1, rxqs 5, rss 4, eqs 16, vfs 0
[    3.387417] be2net 0000:01:00.1: Max: uc-macs 30, mc-macs 64, vlans 15
[    3.387470] be2net 0000:01:00.1: enabled 1 MSI-x vector(s) for NIC
[    3.503420] be2net 0000:01:00.1: created 1 TX queue(s)
[    3.537474] be2net 0000:01:00.1: created 1 RX queue(s)
[    3.644448] be2net 0000:01:00.1: FW version is 4.6.142.10
[    3.651427] be2net 0000:01:00.1: HW Flow control - TX:1 RX:1
[    3.659419] be2net 0000:01:00.1: Adapter does not support HW error recovery
[    3.659609] be2net 0000:01:00.1: Emulex OneConnect: PF vNIC-2 port 1
[    3.848457] be2net 0000:01:00.0 enp1s0f0: renamed from eth2
[    3.867638] be2net 0000:01:00.1 enp1s0f1: renamed from eth0
[   87.256441] be2net 0000:01:00.1 enp1s0f1: Link is Down
[   87.281087] be2net 0000:01:00.0 enp1s0f0: Link is Down

So it is time to turn to google for help

• Wikipedia: 10 Gbit Ethernet
• Jurriaan: My 10 gigabit fiber ethernet home setup on a budget OCe11102 not OCe10102
• Serverfault: Emulex OC10 CNA card be2net twinax No Link
• ServeTheHome: Beware – Emulex 10GbE Virtual Fabric Adapter II x8 PCI-E slot
• Reddit: help_me_get_my_hp_nc550sfp_10gbe_card_working_on/
• Jortxing: hp-emulex-oneconnect-10gbps-nic-be3-initialization-failed-in-debian-squeeze
• Vmware: vSphere-vNetwork-Discussions/Anyone-using-Emulex-OCe10102-with-ESXi-5-5
• Russian: The interfaces do not rise on Emulex OCe10102 (49Y4202 / 49Y4201)

According to RedHat: hardware-enablement_considerations-in-adopting-rhel-8 support for OCe10102 has been removed from be2iscsi, what implications that has for be2net I do not understand. The Firmware on my Emulex cards shows copyright 2013, but I do not understand why they suddenly are not useful anymore.  I also saw this as early as an announcement for RHEL-7.2

So now we know why these boards are available for next to nothing on E-bay.

I tried CentOS-8.3-2011 and the Emulex card was not recognized, then I tried CentOS-7.9-2009, where be2net finds the card as expected, but it still has link down.

I also tried FreeBSD-12.2 where the card was recognized by the oce driver but the link does not come up.

The newest firmware for the OCe10102 I could find, was

• IBM oc10-4.6.281.26.ufi from elx_fw_cna_ibm1212-4.6.281.26-2_linux_32-64.bin
• HP oc10-4.9.416.2.ufi from elxflashStandalone-linux-10.2.477.23-1 from hp-firmware-cna-emulex-2015.02.02-1.1.x86_64.rpm

Both flashes fine, but I have not been able to get Link Up on these (nor with the firmware the card was delivered with)

Some people have indicated that a selftest by ethtool -t  or a hbacmd setPortEnabled should do the trick, but this doesn’t work for me.  Some of the success-stories seems to use the OCe11102 while mine is a OCe10102, I also red somewhere that the older be2 boards would not be supported anymore, the newer OCe11102 uses the be3.

So what is the problem?  My speculation is that the driver which covers both the be2 and the newer be3 based boards seems to be maintained by the manufacturer and the manufacturer has stated that they will not support the be2 anymore. So unless someone takes over and maintains a driver for the be2 boards, they have no future.

So  I am just about to give-up on these boards. If you have got more energy than me, you might find inspiration in this etcfstab.com: emulex_fw_update.html, You need to find and download all the relevant files 2015 or older, install RHEL6 or Centos6 on a machine and try your luck, using the proprietary tools and drivers

If anyone know an easy way to get the OCe10102 working, I would very much appreciate to hear how.

Solarflare SFN5122F

Getting the Solarflare boards to work was as easy as could be, no extra software had to be installed on my Ubuntu-20.04.1 systems:

• I plugged one module into a Supermicro x9SCA with a e3-1220v2 @ 3.1GHz
• The other system was a Dell Optiplex 7010 with a i5-3470 @ 3.2 GHz

 And it worked right out the box, I just had to set the IP address and ran a few iperf tests

• 0.93 Gbits/sec over 1 Gbit Ethernet via an OpenWrt based gigabit-switch MTU 1500
• 5.90 Gbits/sec over 10 Gbit Ethernet via DAC-cable MTU 1500 in 4-lane PCI-E slot
• 6.41 Gbits/sec over 10 Gbit Ethernet via DAC-cable MTU 9000 in 4-lane PCI-E slot
• 9.19 Gbits/sec over 10 Gbit Ethernet via DAC-cable MTU 1500 in 8-lane PCI-E slot
• 9.83 Gbits/sec over 10 Gbit Ethernet via DAC-cable MTU 9000 in 8-lane PCI-E slot

The 4 lane PCIE slot says its upper speed-limit is 8Gbit/s. to get 8-lanes on the Dell Otiplex the Ethernet-board must be placed in the 16-lane slot normally used by the graphics card.

Funny to see how little a gain there is with jumbo-packets (MTU 9000). I still remember a discussion with some of my colleagues 25 years ago when we were dreaming of equipment like this, now it can be yours:

For less than $100 you can get into 10Gb Ethernet.

Even though I wasted a third on the 3 Ethernet-board that I could NOT get working (the $33 also gave me 4 SFP+, which is probably worth $20)  you can certainly upgrade your network backbone without breaking the bank.

And who cares what the name of Operating system is, those day are over, we just want to get our own job done, and your data is what is relevant in end. — With ZFS we can finally keep our filesystem even when we change our Operating system.

A little history

My first experience with ZFS was on FreeNAS v0.7 around 2009. One day while adding a disk to a zpool I made a grave mistake, I wanted to make a mirror, instead I added it as a concatenation. I immediately realized my mistake and removed the disk. That happened to be an even worse mistake. I now had a defunct ZFS system with 2 Terabytes of inaccessible files. At that time there was no way of fixing that problem under FreeNAS, which used an older version of ZFS. But it helped me learn a lot about the internal structure of ZFS. Basicly the way I got the ZFS filesystem back to life was to move the disks to a Solaris box, and scrolling back through the uberblocks to a time just before I made the fatal mistake.

I kept Solaris as the O.S. for some time, until ZFSonLinux became stable enough to use. Then I just installed Ubuntu onto one of the datasets, and booted Linux on the exact same ZFS-filesystem. Since then I have been using ZFS-on-linux as the main”bootloader” for my ZFS filesystem. By now my ZFS-filesystem had been:

• created under FreeNAS which is based on FreeBSD ,
• brought into an unusable and unrepairable state by a systems administrator error
• brought back to life under Illumos, and used there for years until
• ubuntu and zfs-on-linux was installed on a dataset
• used daily for years under Ubuntu.

Without copying or making changes to the main ZFS datasets and home-dirs.

Lately I stumbled into a bug in ZFS-on-Linux github.com/zfsonlinux/zfs/issues/4122, which I am happy to say is fixed, but that made me adopt a back-up-plan – NAMELY – give myself the option to boot into FreeBSD. Which is what this post is all about.

To Day

My current system is running Ubuntu-14.04.3 after this: the system will run FreeBSD-10.2. The only thing tricky with standard FreeBSD is that they use their own boot-loader. We will avoid that and just use GRUB2, the ZFS enabled version from ubuntu in this case, but the one from PC-BSD and the version from Illumos should work too.

I would like to have the option to boot the system into either Linux or FreeBSD. To do that, we just have to add that functionality to /etc/grub.d/09-zfs which will generate the magic grub-lines that can boot FreeBSD too.

The plan for todays work is:

• Install FreeBSD into a dataset
• Create a /boot/zfs/zpool.cache so that FreeBSD will recognize the ZFS pool
• Remove ZFS internal mount-points
• Teach GRUB2 how to boot into a FreeBSD ZFS-data set
• Use the same SSH credentials to avoid a lot of fuss

Most of the work is done on a live system system with very little down-time.

Install FreeBSD into a dataset

This little shell-script will do the majority of the work.

#!/bin/bash
#
# install FreeBSD onto a ZFS dataset
VERSION=10.2-RELEASE
FROM=ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/$VERSION
# assuming only one ZPOOL
POOL=$(zpool status | awk '/pool:/ {print $NF}')
DEST=$POOL/ROOT/FreeBSD-$VERSION
R=/$DEST
echo "# Install FreeBSD-$VERSION to $R"
zfs create $DEST
 
for file in base.txz lib32.txz kernel.txz doc.txz ports.txz src.txz; do
    wget $FROM/$file
done
for file in base.txz lib32.txz kernel.txz doc.txz ports.txz src.txz; do
    tar --unlink -xpJf $file -C $R
done
 
echo 'zfs_load="YES"' >> $R/boot/loader.conf
echo 'vfs.root.mountfrom="zfs:zroot"' >> $R/boot/loader.conf
 
echo 'zfs_enable="YES"' >> $R/etc/rc.conf
echo 'hostname="freebsd"' >> $R/etc/rc.conf
echo 'ifconfig_re0="dhcp"' >> $R/etc/rc.conf
echo 'sshd_enable="YES"' >> $R/etc/rc.conf
echo 'syslogd_flags="-ss"' >> $R/etc/rc.conf
 
touch $R/etc/fstab
sync

Create a /boot/zfs/zpool.cache so that FreeBSD will recognize the ZFS pool

Most likely the zpool.cache from Linux is not compatible with FreeBSD (linux/bsd, gcc/clang, device-naming). Regarding the device-naming, tweaking udev on Linux might be able to fix it part of the way, like we discuss in github.com/zfsonlinux/grub/issues/5 (I am storepeter).

The easiest way to fix this though is to boot FreeBSD from an USB-stick, start a shell, and create a zpool.cache manually this way

zpool import -o altroot=/tmp -o cachefile=/tmp/zpool.cache zhome
cp /tmp/zpool.cache /tmp/zhome/ROOT/FreeBSD*/boot
zpool export zhome

Remove ZFS internal mount-points

One thing I really don’t like about ZFS, is the facility to set mount-points inside ZFS. Hence if you import ZFS filesystem, have also imported the mount-points. and mounting Linux-root-fileystem on / for FreeBSD, is BAD.

To get rid of all the explicit mount-points I make the following changes to /etc/rc.d/zfs on FreeBSD (Recursively inherit mount-point for all datasets under /ROOT)

zfs_start_main()
{
#start change
        for pool in $(zpool list -H | cut -f1); do
                zfs inherit mountpoint -r $pool/ROOT
        done
#end change
        zfs mount -va
        zfs share -a
        if [ ! -r /etc/zfs/exports ]; then
                touch /etc/zfs/exports
        fi
}

Teach GRUB2 how to boot into a FreeBSD ZFS-data set

I have been using ZFS-on-Linux for a long time and have made my own grub configuration so I can boot into various Linux versions. Kind of my own home-grown boot-environments, but much simpler than the Solaris/Illumos/PCBSD beadm

GRUB2 uses a number of shell-script to generate the grub.cfg that the bootloader uses when booting. Below is /etc/grub.d/09-zfs which makes it possible to boot into all the root-filesystems available on the zpool

#! /bin/sh
# Create boot menu for zfs root systems choices are:
# if current root is Linux: menu-point for all kernel-versions on this dataset
# if current root is FreeBSD: menu-point for FreeBSD on here
# for all Linux root-filsystems: menu-point  with the newest kernel on that root filesystem,
# for all FreeBSD root-filsystems: menu-point for FreeBSD on here
# current root will be default
#
if [ -d /usr/share/grub ]; then # Linux
	prefix="/usr"
else # FreeBSD
	prefix="/usr/local"
	PATH=/usr/local/bin:/usr/local/sbin:$PATH
	export PATH
fi
exec_prefix="${prefix}"
datarootdir="${prefix}/share"

. "${datarootdir}/grub/grub-mkconfig_lib"

export TEXTDOMAIN=grub
export TEXTDOMAINDIR="${datarootdir}/locale"

CLASS="--class gnu-linux --class gnu --class s"

HOSTNAME=`hostname`
BOOT=/boot
VERBOSE=1
 vecho() { if [ x"$VERBOSE" != x ]; then echo "# $*" >/dev/stderr; fi; }
vvecho() { if [ x"$VERBOSE" = x2 ]; then echo "# $*" >/dev/stderr; fi; }
  show() { (echo -n "#";for show_index in $*;do echo -n " ";eval echo -n "$show_index=\$show_index";done;echo;)>/dev/stderr; }
 vshow() { if [ x"$VERBOSE" != x ]; then show $*; fi; }
vvshow() { if [ x"$VERBOSE" = x2 ]; then show $*; fi; }

# menues to boot */ROOT/* with "highest" kernels available on /boot
# sort the boot entries for host "z" like this
# z/ROOT/ first.
# z/ROOT/ubuntu-14.04-z
# z/ROOT/ubuntu-13.10-z
# z/ROOT/ubuntu-13.04-z
ROOT_ON_ZFS=`df -T / | awk '$2 == "zfs" && $7 == "/" {print $1}'`
get_mountpoint()
{
	if [ $1 = $ROOT_ON_ZFS ]; then
		echo /
	else
		zfs list -H $1 | awk '{print $NF}'
	fi
	#zfs get -H mountpoint $1 | awk '{print $3}'
}
# root filestem might no be mounted
get_root_filesystem_list()
{
	list_us=""
	list_not_us=""
	list=`zfs list | awk '/^[a-zA-Z0-9]*\/ROOT\// { print $1}'| grep -v @ |sort`
	for z in ${list}; do
		if [ x$z = x${ROOT_ON_ZFS} ]; then
			continue	# skip current root
			echo -n "$z "
		else
			r=$(get_mountpoint $z)
			if [ $r = / ]; then # mountpoint is / but not mounted
				zfs inherit mountpoint $z
				zfs mount $z
			fi
			if [ noauto = $(sudo zfs get -H  canmount  zhome/ROOT/10.2-RELEASE-p10-up-20151228_181943 | cut -f3) ]; then
				zfs mount $z
			fi
			# root filesystem that end in -myhostname has priority
			if [ `basename $z -$HOSTNAME` != `basename $z` ]; then
				list_us="$z $list_us"
			else
				list_not_us="$z $list_not_us"
			fi
		fi
	done
	echo $list_us $list_not_us
}
get_mount_dev()
{
	df $1| cut -f1 -d' ' | tail -1
}
get_pool()
{
	echo $1 | cut -f1 -d/
}
get_ds()
{
	echo $1 | cut -f2-99 -d/
}
get_rescue()
{
	boot_dev=$(get_mount_dev /boot)
	for d in /boot-sd[a-z][0-9] /root-sd[a-z][0-9]/boot; do
		if [ $boot_dev != $(get_mount_dev $d) ]; then
			echo -n "$d "
		fi
	done
}

# two scenarios
# /boot on ZFS, requires grub to understand the current ZFS version.
# /boot is not on ZFS, The only option if ZFS is ahead of the version in GRUB
linux_entry()
{
	z=$1
	ds=$1
	r=`get_mountpoint $ds`
	kversion=$2
	boot=$3
	vshow "linux_entry z kversion boot"
	linux=$boot/vmlinuz-${kversion}
	if [ ! -f $linux ]; then
		vecho "skipping no linux=$linux"
		return
	fi
	initrd=$boot/initrd.img-${kversion}
	if [ ! -f $initrd ]; then
		vecho "skipping no initrd=$initrd"
		return
	fi
	if [ -f $r/etc/lsb-release ]; then
		. $r/etc/lsb-release
		OS=$DISTRIB_DESCRIPTION
	else
		OS="Unknown"
	fi
	echo "ZFS ${ds} - ${OS} - ${kversion}" >&2
	title="ZFS ${ds} ${OS} ${kversion} $boot"
	rpool=`echo $ds | cut -f1 -d/`
	CMDLINE_ZFS="boot=zfs rpool=$rpool bootfs=$ds"
	linux=`make_system_path_relative_to_its_root $linux`
	initrd=`make_system_path_relative_to_its_root $initrd`
	ZFS_DEV=`grub-probe --target=device $boot`
	vecho "grub-probe $boot --target=device returned $ZFS_DEV"

	echo "menuentry '$title' --class ubuntu --class gnu-linux --class gnu --class os) {"
	echo "	insmod gzio"
	vshow ZFS_DEV
      	prepare_grub_to_access_device $ZFS_DEV | sed -e "s/^/	/"
	echo "	echo \"Loading $OS $kversion from ZFS $ds\""
	echo "	linux $linux $CMDLINE_ZFS $GRUB_CMDLINE_LINUX"
	echo "	initrd $initrd"
	echo "}"
}

freebsd_entry()
{
	z=$1
	pool=$(get_pool $1)
	ds=$(get_ds $1)
	r=$(get_mountpoint $1)
	version=$(basename $ds | sed 's/-/_/g')
	vshow freebsd_entry z ds r version
	OS="FreeBSD"
	echo "ZFS ${1} - ${OS}" >&2
	title="ZFS ${1} ${OS}"
	ZFS_DEV=`grub-probe --target=device $r`

	echo "menuentry '$title' {"
	echo "	insmod gzio"
	#echo "	set root='hd0,msdos7'"
	echo "	insmod zfs"
	echo "	search --no-floppy -s -l zhome"
	echo "	echo \"Loading $OS from ZFS $1, edit to append -s or -v to line below\""
	echo "	kfreebsd /$ds/@/boot/kernel/kernel"
	echo "	kfreebsd_loadenv /$ds/@/boot/device.hints"
	echo "	kfreebsd_module /$ds/@/boot/zfs/zpool.cache type=/boot/zfs/zpool.cache"
	echo "	set kFreeBSD.vfs.root.mountfrom=zfs:$pool/$ds"
	echo "	set kFreeBSD.module_path=\"/boot/kernel;/boot/modules\""
	echo "# unfortunately does the kernel not use the above when booting, so you have to include all depencies in /boot/loader.conf"
	VARS=""
# load modules and set variables for boot
	for i in $(cat $r/boot/loader.conf* | grep -v "^#" | grep = | sed 's/#.*$//');do
		name=$(echo $i | cut -f1 -d= | sed 's/\./_/g')
		value=$(echo $i | cut -f2 -d=)
		vvshow i name value
        	if eval [ x\$version_$name !=  x ]; then
                	vecho $version_$name already set
        	else
                	vvecho new $version_$name
                	eval $version_$name=$value
			m=$(echo $i | grep '_load="YES"' | sed 's/_load.*$//')
			#if [ x$m != x ]; then
			if [ $m ]; then
                		vecho load module $m
				if [ -f $r/boot/kernel/$m.ko ]; then
					echo "	kfreebsd_module_elf /$ds/@/boot/kernel/$m.ko"
				elif [ -f $r/boot/modules/$m.ko ]; then
					echo "	kfreebsd_module_elf /$ds/@/boot/modules/$m.ko"
        			fi
			else
                		vecho variable $i
				# not a module - set it as a variable
				VARS="$VARS $i"
			fi
		fi
	done
	echo "	set kFreeBSD.bootfile=\"kernel\""
	echo "	set kFreeBSD.kernel=\"kernel\""
	echo "	set kFreeBSD.kernel_options=\"\""
	echo "	set kFreeBSD.kernelname=\"/boot/kernel/kernel\""
	echo "	set kFreeBSD.module_path=\"/boot/kernel;/boot/modules\""
	for v in $VARS; do
		echo "	set kFreeBSD.$v"
	done
	echo "}"
}

### current root all kernels
z=$ROOT_ON_ZFS
vshow ZFS_ROOTS
vshow ROOT_ON_ZFS
r=/
if [ -d $r/lib/modules ]; then
# we want newest kernel first
	for t in `ls -t $r/lib/modules/*/*order`;do
		d=`dirname $t`
		kvers=`basename $d`
		vecho "Found $z has modules: $kvers"
		linux_entry $z $kvers /boot
		linux_entry $z $kvers $z/boot
		if [ $(get_mount_dev /boot) != $(get_mount_dev $r/boot) ]; then
			linux_entry $z $kvers $r/boot
		fi
	done
elif [ -f $r/etc/freebsd-update.conf ]; then
	freebsd_entry $z
fi

# for all roots newest kernel
for z in $(get_root_filesystem_list); do
	r=$(get_mountpoint $z)
	vecho "Creating boot entires for $z root=$r"
	if [ $r = "/" ]; then
		echo "zfs inherit mountpoint $z"
		echo "zfs mount $z"
		continue
	fi
	r=$(get_mountpoint $z)
	if [ $r = "/" ]; then
		continue
	fi
	if [ -d $r/lib/modules ]; then
# we want highest kernel first
		kvers=`ls -t $r/lib/modules| head -1`
		linux_entry $z $kvers /boot
		if [ $(get_mount_dev /boot) != $(get_mount_dev $r/boot) ]; then
			linux_entry $z $kvers $r/boot
		fi
	elif [ -f $r/etc/freebsd-update.conf -a -d $r/boot/kernel ]; then
		freebsd_entry $z
	fi
done

That was quite a number of script lines, but straight forward, and it is easy to test just run the script and see if it generates what you expect.

Install grub2 from FreeBSD

I already have GRUB2 installed on the harddrives, This one was installed from Ubuntu.
And it will happily boot both Ubuntu and FreeBSD. But if I have to update the Grub menus I have to reboot into Ubuntu and run update-grub.

I am normally running my ZFS-systems as mirrors, so it would be handy to have one disk boot with a Linux-based GRUB2, and the other with a FreeBSD-based GRUB2. So that we we will do. Assuming that we have booted the system into FreeBSD.

• • install the GRUB2 package

pkg install grub2-pcbsd

• • install the GRUB2 bootloader to the disk

grub-install –modules=”part_gpt zfs” /dev/ada0

• • add script to create menus /usr/local/etc/grub.d/09-zfs

The 09-zfs script above, will also work from within FreeBSD

• update /boot/grub/grub.cfg

grub-mkconfig -o /boot/grub/grub.cfg

Keep ssh credentials

I copy over the ssh credential, so ssh-wise it looks like the same host.

rsync -av --exclude "ssh*_config" /zhome/ROOT/ubuntu-14.04.3/etc/ssh /etc/

That was all for now