mellanox | panticz.de

Mellanox: Install pre-build driver

https://www.mellanox.com/products/ethernet-drivers/linux/mlnx_en

VERSION=24.04-0.7.0.0
URL=http://www.mellanox.com/downloads/ofed/MLNX_EN-${VERSION}/mlnx-en-${VERSION}-ubuntu$(lsb_release -rs)-$(uname -p).tgz

# download
wget ${URL} -q -O /tmp/${URL##*/}
tar -C /tmp -xzf /tmp/mlnx-en-*-ubuntu*.tgz

# install prerequirements
# apt install -y apt-utils bzip2 lsof debhelper gcc make build-essential automake chrpath quilt autotools-dev perl  autoconf libltdl-dev dkms swig graphviz

# install
cd /tmp/mlnx-en-${VERSION}-ubuntu*/

# prebuild driver only
#./install --add-kernel-support-build-only

# install
#./install --force
./install -q --dpdk --without-fw-update

# show packages
ll /tmp/mlnx-en-*-generic/mlnx-en-*-ext.tgz

# install
# ./install -q --dkms --without-fw-update --force

Links
https://developer.nvidia.com/networking/ethernet-software

Mellanox: SR-IOV (Single Root IO Virtualization)

Install Mellanox Driver
http://www.panticz.de/mellanox/install-dirver

lspci | grep Mellanox
mstconfig -y -d  18:00.1 set SRIOV_EN=1 NUM_OF_VFS=16

#cat /etc/modprobe.d/mlnx.conf 
#options mlx4_core num_vfs=5 probe_vf=5

apt install -y sysfsutils

cat < /etc/sysfs.d/mlnx-sriov_numvfs.conf
class/net/ens6f0/device/sriov_numvfs = 8
class/net/ens6f1/device/sriov_numvfs = 8
class/net/ens7f0/device/sriov_numvfs = 8
class/net/ens7f1/device/sriov_numvfs = 8
EOF

# /boot/grub/grub.cf
intel_iommu=on

ll /sys/class/net/en{p,s}*
echo 8 > /sys/class/net/ens6f0/device/sriov_numvfs

Configure VLAN

ip link add link ens7f0v6 vlan1234  type vlan id 1234
dhclient -v vlan1234

Configure netplan
http://www.panticz.de/netplan

network:
  version: 2
  renderer: networkd

  ethernets:
    eno1:
      dhcp4: yes
    eno2:
      virtual-function-count: 4
    vf1:
      match:
        name: eno2v[0-3]
      dhcp4: no
      #    eno2v0:
      #dhcp4: no
       #mtu: 9000

  vlans:
    stage-mgmt:
      id: 3649
      #link: eno2v0
      link: vf1
      dhcp4: yes

OpenStack SR-IOV
https://docs.openstack.org/neutron/latest/admin/config-sriov.html

Links
https://docs.mellanox.com/pages/viewpage.action?pageId=19801751
https://www.mellanox.com/related-docs/prod_software/Mellanox_EN_for_Linux_User_Manual_v3_20.pdf
https://community.mellanox.com/s/article/howto-configure-and-probe-vfs-on-mlx5-drivers
http://bloggf.dannf.org/index.php/2019/10/22/passing-lots-of-pcie-devices-to-a-kvm-guest/

Install Mellanox MFT

https://www.mellanox.com/products/adapter-software/firmware-tools

#URL=https://www.mellanox.com/downloads/MFT/mft-4.26.1-3-x86_64-deb.tgz
URL=https://www.mellanox.com/downloads/MFT/mft-4.27.0-83-x86_64-deb.tgz

wget -q ${URL} -P /tmp
tar xzf /tmp/mft-*-x86_64-deb.tgz -C /tmp/

sudo apt install -y gcc make dkms linux-headers-$(uname -r)
/tmp/mft-*-x86_64-deb/install.sh 

mst start

Mellanox ConnectX-3 Pro UEFI iPXE boot

Device Type:      ConnectX3Pro
  Part Number:      MCX312B-XCC_Ax
  Description:      ConnectX-3 Pro EN network interface card; 10GigE; dual-port SFP+; PCIe3.0 x8 8GT/s; RoHS R6
  PSID:             MT_1200111023
  PCI Device Name:  /dev/mst/mt4103_pci_cr0
  Port1 MAC:        ec0d9a00aab1
  Port2 MAC:        ec0d9a00aab2
  Versions:         Current        Available    
     FW             2.42.5000      N/A          
     PXE            3.4.0752       N/A

Flash UEFI firmware
Request UEFI firmware from support@mellanox.com
http://www.panticz.de/mellanox/firmware-update

Install MFT
http://www.panticz.de/install-mellanox-mft

Flash firmware

flint -y -d /dev/mst/mt4103_pci_cr0 -i firmware fw-ConnectX3Pro-rel-2_42_5000-MCX312B-XCC_Ax-FlexBoot-3.4.752-UEFI-14.11.46.bin b

Device Type:      ConnectX3Pro
  Part Number:      MCX312B-XCC_Ax
  Description:      ConnectX-3 Pro EN network interface card; 10GigE; dual-port SFP+; PCIe3.0 x8 8GT/s; RoHS R6
  PSID:             MT_1200111023
  PCI Device Name:  /dev/mst/mt4103_pci_cr0
  Port1 MAC:        ec0d9a00aab1
  Port2 MAC:        ec0d9a00aab2
  Versions:         Current        Available    
     FW             2.42.5000      N/A          
     PXE            3.4.0752       N/A          
     UEFI           14.11.0046     N/A

Fix iPXE boot issue (recompile ipxe.efi) when connected to LACP swith port
http://www.panticz.de/ipxe/compile

# disable NET_PROTO_LACP
diff --git a/src/config/general.h b/src/config/general.h
index 3c14a2cd..d860f450 100644
--- a/src/config/general.h
+++ b/src/config/general.h
@@ -38,7 +38,6 @@ FILE_LICENCE ( GPL2_OR_LATER_OR_UBDL );
 #undef NET_PROTO_IPV6          /* IPv6 protocol */
 #undef NET_PROTO_FCOE          /* Fibre Channel over Ethernet protocol */
 #define        NET_PROTO_STP           /* Spanning Tree protocol */
-#define        NET_PROTO_LACP          /* Link Aggregation control protocol */

# compile
make bin-x86_64-efi/ipxe.efi

# deploy ipxe.efi
scp /tmp/ipxe/src/bin-x86_64-efi/ipxe.efi tftp.example.com:/tftpboot/

mellanox