10 Gbit Ethernet interconnecting my servers.

Posted on January 19, 2021 by peter

My Christmas present for myself this season were 10Gbe Ethernet.  I had no idea what to get so I bought the cheapest I could find on E-bay, $11 each delivered. These were IBM branded OCe10102 cards made by Emulex.

Two of the Emulex cards came with 2 SFP+ transceivers so I have the hardware to make two independent fiber connections, fiber-patch-cable is available on Amazon for $5 new delivered.

I also bought 4 DAC-cables for $2 each.The only thing left now is to install-connect-configure-enjoy.

Unfortunately it didn’t just work, hence this BLOG post with the story so far.

The simplest way to fix a software problem is to buy some more hardware.  And I were in need of a two low-profile brackets for the 10Gbe Ethernet boards, so I had a good excuse to buy some different brand 10Gbe Ethernet cards this time I payed $15 each. These we Solarflare SFB5122F

 

 

Emulex OCe10102

  • The Emulex 10Gb-Ethernet-adapters are 8 lane PCI-Express, but unfortunately they are a little too wide so they need to go into a 16 lane slot, or some surgery is needed, this is/was another good reason to try another brand.
  • lshw identifies the board as: Emulex OneConnect OCe10100/OCe10102 Series 10 GbE
  • According to dmesg the Emulex 10Gb-Ethernet-adapters are found as be2net but the interface do not become active when connected, output of dmesg below
dmesg | grep be2net
[    1.956424] be2net 0000:01:00.0: be2net version is 12.0.0.0
[    1.965890] be2net 0000:01:00.0: PCIe error reporting enabled
[    2.347419] be2net 0000:01:00.0: FW config: function_mode=0x20003, function_caps=0x7
[    2.555421] be2net 0000:01:00.0: Max: txqs 1, rxqs 5, rss 4, eqs 16, vfs 0
[    2.573792] be2net 0000:01:00.0: Max: uc-macs 30, mc-macs 64, vlans 15
[    2.627490] be2net 0000:01:00.0: enabled 1 MSI-x vector(s) for NIC
[    2.785022] be2net 0000:01:00.0: created 1 TX queue(s)
[    2.823423] be2net 0000:01:00.0: created 1 RX queue(s)
[    2.960433] be2net 0000:01:00.0: FW version is 4.6.142.10
[    2.967533] be2net 0000:01:00.0: HW Flow control - TX:1 RX:1
[    2.975427] be2net 0000:01:00.0: Adapter does not support HW error recovery
[    2.975574] be2net 0000:01:00.0: Emulex OneConnect: PF vNIC-2 port 0
[    2.975591] be2net 0000:01:00.1: be2net version is 12.0.0.0
[    2.975790] be2net 0000:01:00.1: PCIe error reporting enabled
[    3.243412] be2net 0000:01:00.1: FW config: function_mode=0x20003, function_caps=0x7
[    3.387416] be2net 0000:01:00.1: Max: txqs 1, rxqs 5, rss 4, eqs 16, vfs 0
[    3.387417] be2net 0000:01:00.1: Max: uc-macs 30, mc-macs 64, vlans 15
[    3.387470] be2net 0000:01:00.1: enabled 1 MSI-x vector(s) for NIC
[    3.503420] be2net 0000:01:00.1: created 1 TX queue(s)
[    3.537474] be2net 0000:01:00.1: created 1 RX queue(s)
[    3.644448] be2net 0000:01:00.1: FW version is 4.6.142.10
[    3.651427] be2net 0000:01:00.1: HW Flow control - TX:1 RX:1
[    3.659419] be2net 0000:01:00.1: Adapter does not support HW error recovery
[    3.659609] be2net 0000:01:00.1: Emulex OneConnect: PF vNIC-2 port 1
[    3.848457] be2net 0000:01:00.0 enp1s0f0: renamed from eth2
[    3.867638] be2net 0000:01:00.1 enp1s0f1: renamed from eth0
[   87.256441] be2net 0000:01:00.1 enp1s0f1: Link is Down
[   87.281087] be2net 0000:01:00.0 enp1s0f0: Link is Down

So it is time to turn to google for help

According to RedHat: hardware-enablement_considerations-in-adopting-rhel-8 support for OCe10102 has been removed from be2iscsi, what implications that has for be2net I do not understand. The Firmware on my Emulex cards shows copyright 2013, but I do not understand why they suddenly are not useful anymore.  I also saw this as early as an announcement for RHEL-7.2

So now we know why these boards are available for next to nothing on E-bay.

I tried CentOS-8.3-2011 and the Emulex card was not recognized, then I tried CentOS-7.9-2009, where be2net finds the card as expected, but it still has link down.

I also tried FreeBSD-12.2 where the card was recognized by the oce driver but the link does not come up.

The newest firmware for the OCe10102 I could find, was

  • IBM oc10-4.6.281.26.ufi from elx_fw_cna_ibm1212-4.6.281.26-2_linux_32-64.bin
  • HP oc10-4.9.416.2.ufi from elxflashStandalone-linux-10.2.477.23-1 from hp-firmware-cna-emulex-2015.02.02-1.1.x86_64.rpm

Both flashes fine, but I have not been able to get Link Up on these (nor with the firmware the card was delivered with)

Some people have indicated that a selftest by ethtool -t  or a hbacmd setPortEnabled should do the trick, but this doesn’t work for me.  Some of the success-stories seems to use the OCe11102 while mine is a OCe10102, I also red somewhere that the older be2 boards would not be supported anymore, the newer OCe11102 uses the be3.

So what is the problem?  My speculation is that the driver which covers both the be2 and the newer be3 based boards seems to be maintained by the manufacturer and the manufacturer has stated that they will not support the be2 anymore. So unless someone takes over and maintains a driver for the be2 boards, they have no future.

So  I am just about to give-up on these boards. If you have got more energy than me, you might find inspiration in this etcfstab.com: emulex_fw_update.html, You need to find and download all the relevant files 2015 or older, install RHEL6 or Centos6 on a machine and try your luck, using the proprietary tools and drivers

If anyone know an easy way to get the OCe10102 working, I would very much appreciate to hear how.

Solarflare SFN5122F

Getting the Solarflare boards to work was as easy as could be, no extra software had to be installed on my Ubuntu-20.04.1 systems:

  • I plugged one module into a Supermicro x9SCA with a e3-1220v2 @ 3.1GHz
  • The other system was a Dell Optiplex 7010 with a i5-3470 @ 3.2 GHz

 And it worked right out the box, I just had to set the IP address and ran a few iperf tests

  • 0.93 Gbits/sec over 1 Gbit Ethernet via an OpenWrt based gigabit-switch MTU 1500
  • 5.90 Gbits/sec over 10 Gbit Ethernet via DAC-cable MTU 1500 in 4-lane PCI-E slot
  • 6.41 Gbits/sec over 10 Gbit Ethernet via DAC-cable MTU 9000 in 4-lane PCI-E slot
  • 9.19 Gbits/sec over 10 Gbit Ethernet via DAC-cable MTU 1500 in 8-lane PCI-E slot
  • 9.83 Gbits/sec over 10 Gbit Ethernet via DAC-cable MTU 9000 in 8-lane PCI-E slot

The 4 lane PCIE slot says its upper speed-limit is 8Gbit/s. to get 8-lanes on the Dell Otiplex the Ethernet-board must be placed in the 16-lane slot normally used by the graphics card.

Funny to see how little a gain there is with jumbo-packets (MTU 9000). I still remember a discussion with some of my colleagues 25 years ago when we were dreaming of equipment like this, now it can be yours:

For less than $100 you can get into 10Gb Ethernet.

Even though I wasted a third on the 3 Ethernet-board that I could NOT get working (the $33 also gave me 4 SFP+, which is probably worth $20)  you can certainly upgrade your network backbone without breaking the bank.

Print Friendly, PDF & Email
This entry was posted in FreeBSD, internet, Linux. Bookmark the permalink.