So I recently got my hands on a mellanox sn2700 switch and a few ConnectX-6 DX cards...

I have played with creating VF's before with my CX3-Pro cards before, but I was used to using the mlx4 driver which does not have the ability to put my card into switchdev mode...

What I have been doing on this new card so far is the following....

I create a VF on the card, set it up using the ip command to give the VF a vlan and then I actually add a static ip address on the VF . I know maybe this isn't what it's meant to be used for but I liked using it in this way. I could also setup more VF's with different vlans and use them as UPLINK OVN networks for my LXD setup.

So I understand that I have been using the legacy mode of my card ....

Now I would like to switch to using switchdev (because I want to understand it better), but im running into trouble and im not sure I can even achieve what Im trying to do..

I know that when I create my VF's I then unbind them from the card, switch the card to switchdev mode , add any offloading capabilities and then rebind the VF's back to the card.

I now have a Physical Nic , a virtual function for that Nic, and then (I guess its called) a physical representation of that virtual function (i.e physicalNic: eno1 , virtualFunction: eno1v0, physical representation: eth0).

I would like to setup one of my virtual functions on my card while im in switchdev mode with a static IP and a vlan. I want to do this because I am using NVME over RDMA on one of my nodes and it seems to be the best option to use my CX6-DX card for that reason.

I am unsure sure how to go about this , ive tried following quite a few guides like this one from Intel(link) or even this one from Nvidia that talks about VF-Lag(link) but have had no success.

I have ended up with some method to be able to attach an ip address to eth0 (physical representation of the virtual function eno1v0) after I put the card in switchdev mode but I can only ping the address I statically set on it and no other addresses on that same subnet.

My OVN setup is pretty simple and I only have a default br-int interface. So far I have no ports added to the br-int interface.

How can I achieve what I want to do which is to make a useable virtual function on my host OS with a vlan attached to it using switchdev mode?

I recently made some necessary updates to our lab by upgrading some of our older servers to handle storage.

I currently have 3 poweredge R720's on my rack and I wanted to use them specifically for Ceph storage handling.

I have installed the GY1TD card which has a PEX 8734 switch internally and can handle x4x4x4x4 bifurcation. I had also replaced the sas backplane with the necessary one to allow u.2 drives to work. All these parts are Dell parts and the drives light up and looks like they connect.

The problem is the following..

If I have the drives connected at boot, the boot process gets stuck at "initializing firmware".

If I remove the drives out of the caddy but I have the backplane and pic card connected then the server boots fine. But if I put the drives back in then the drive caddy lights up green and looks like it's doing something but I can't see the drive at all on the host. fdisk, blkid, lsblk nothing shows the drives.

I do not want to boot from these drives but I do want to use them strictly for storage on ceph as the poweredge servers have all been updated to 100Gb fiber links in-between the cluster.

I have also removed the perc card that was in the servers originally.

What can I do to make this card work ? I want to create an all flash ceph cluster and im having a real hard time with it.

lspci output below

04:00.0 Ethernet controller [0200]: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] [15b3:1007] `Subsystem: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] [15b3:0007]` `Kernel driver in use: mlx4_core` `Kernel modules: mlx4_core` 05:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8734 32-lane, 8-Port PCI Express Gen 3 (8.0GT/s) Switch [10b5:8734] (rev ab) `Subsystem: Dell PEX 8734 32-lane, 8-Port PCI Express Gen 3 (8.0GT/s) Switch [1028:1f84]` `Kernel driver in use: pcieport` 06:04.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8734 32-lane, 8-Port PCI Express Gen 3 (8.0GT/s) Switch [10b5:8734] (rev ab) `Subsystem: Dell PEX 8734 32-lane, 8-Port PCI Express Gen 3 (8.0GT/s) Switch [1028:1f84]` `Kernel driver in use: pcieport` 06:05.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8734 32-lane, 8-Port PCI Express Gen 3 (8.0GT/s) Switch [10b5:8734] (rev ab) `Subsystem: Dell PEX 8734 32-lane, 8-Port PCI Express Gen 3 (8.0GT/s) Switch [1028:1f84]` `Kernel driver in use: pcieport` 06:06.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8734 32-lane, 8-Port PCI Express Gen 3 (8.0GT/s) Switch [10b5:8734] (rev ab) `Subsystem: Dell PEX 8734 32-lane, 8-Port PCI Express Gen 3 (8.0GT/s) Switch [1028:1f84]` `Kernel driver in use: pcieport` 06:07.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8734 32-lane, 8-Port PCI Express Gen 3 (8.0GT/s) Switch [10b5:8734] (rev ab) `Subsystem: Dell PEX 8734 32-lane, 8-Port PCI Express Gen 3 (8.0GT/s) Switch [1028:1f84]` `Kernel driver in use: pcieport` 0d:00.0 PCI bridge [0604]: Renesas Technology Corp. SH7757 PCIe Switch [PS] [1912:0013] `Subsystem: Renesas Technology Corp. SH7757 PCIe Switch [PS] [1912:0013]` `Kernel driver in use: pcieport` 0e:00.0 PCI bridge [0604]: Renesas Technology Corp. SH7757 PCIe Switch [PS] [1912:0013] `Subsystem: Renesas Technology Corp. SH7757 PCIe Switch [PS] [1912:0013]` `Kernel driver in use: pcieport` 0e:01.0 PCI bridge [0604]: Renesas Technology Corp. SH7757 PCIe Switch [PS] [1912:0013] `Subsystem: Renesas Technology Corp. SH7757 PCIe Switch [PS] [1912:0013]` `Kernel driver in use: pcieport` 0f:00.0 PCI bridge [0604]: Renesas Technology Corp. SH7757 PCIe-PCI Bridge [PPB] [1912:0012] `Subsystem: Renesas Technology Corp. SH7757 PCIe-PCI Bridge [PPB] [1912:0012]` 10:00.0 VGA compatible controller [0300]: Matrox Electronics Systems Ltd. G200eR2 [102b:0534] `DeviceName: Embedded Video` `Subsystem: Dell G200eR2 [1028:048c]` `Kernel driver in use: mgag200` `Kernel modules: mgag200`