10GbE SFP+ problems with ixgbe

Support
1

I have some old servers (Lanner NCA-2512) with 10GbE ports, using intel X553 and SFP+.

Back in the day, these were running CentOS7, and happily connecting to each other using a variety of SFPs and fibres.

After upgrading to AlmaLinux9, I’m experiencing something very weird. As long as one of the pair runs CentOS7, I get a link up, and can transfer data at full speed. But if both run AlmaLinux, the link never comes up. I’ve been trying a variety of ip link set <device> up and similar, but never managed to get a working link. I have tried to compile with the latest version (6.3.4) from intel, and it has the same behaviour.

It’s probably a problem with autonegotiation, but even if I try to force the link up, it doesn’t come on.

I’ve recompiled the latest driver from intel (ixgbe v6.3.4), but it has the same behaviour.

Reading the README from that driver, I came upon this:

In addition, SFP+ devices based on the Intel(R) Ethernet ConnectionX552 and Intel(R) Ethernet Connection X553 do not support the following features: 
  * Speed and duplex auto-negotiation 
  * Wake on LAN 
  * 1000BASE-T SFP Modules

That sounds rather ominous. But it looks like that text has been in the driver for a long time. On that page they only go back to version 5.14, rather than the 5.0.1 which was used in our old CentOS systems, but in version 5.14 there’s the same disclaimer.

How can I move on from here? What alternative routes may I have missed?

Is there maybe an alternative driver for these chips?

2

Update

If I run a live USB version of AlmaLinux10-kittten on one of the systems — Then it works! It automagically autonegotiates the link with the system running Alma9. (I have tried 1GSFPs, 10G SFP+ and 10G DAC. All work OK!)

To summarize:

CentOS7 <->Alma9 :ok_button:
Alma9<->Alma9 :prohibited:
Alma9 ↔︎ Alma10 :ok_button:

3

Update - If I use (a recent version of) the Out-of-Tree driver from intel it does work - as long as I don’t boot with the in-tree driver first!

(I made my tests with the OOT driver by doing an rmmod ixgbe / insmod ./ixgbe.ko - and that’s not the same as booting with the OOT driver! Apparently the driver leaves the PHY (or something else) in an undefined state, where reloading the driver is not sufficient to initialize it properly.)

1 Like