Skip to content

Commit

Permalink
net/mlx5: Add devlink subfunction port documentation
Browse files Browse the repository at this point in the history 
Add documentation for subfunction management using devlink
port.

Signed-off-by: Parav Pandit <parav@nvidia.com>
Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
  • Loading branch information
Parav Pandit authored and Saeed Mahameed committed Jan 22, 2021
1 parent 6474ce7 commit 142d93d
Showing 1 changed file with 210 additions and 0 deletions.
210 changes: 210 additions & 0 deletions Documentation/networking/device_drivers/ethernet/mellanox/mlx5.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ Contents
- `Enabling the driver and kconfig options`_
- `Devlink info`_
- `Devlink parameters`_
- `mlx5 subfunction`_
- `mlx5 port function`_
- `Devlink health reporters`_
- `mlx5 tracepoints`_

Expand Down Expand Up @@ -181,6 +183,214 @@ User command examples:
values:
cmode driverinit value true

mlx5 subfunction
================
mlx5 supports subfunction management using devlink port (see :ref:`Documentation/networking/devlink/devlink-port.rst <devlink_port>`) interface.

A Subfunction has its own function capabilities and its own resources. This
means a subfunction has its own dedicated queues (txq, rxq, cq, eq). These
queues are neither shared nor stolen from the parent PCI function.

When a subfunction is RDMA capable, it has its own QP1, GID table and rdma
resources neither shared nor stolen from the parent PCI function.

A subfunction has a dedicated window in PCI BAR space that is not shared
with ther other subfunctions or the parent PCI function. This ensures that all
devices (netdev, rdma, vdpa etc.) of the subfunction accesses only assigned
PCI BAR space.

A Subfunction supports eswitch representation through which it supports tc
offloads. The user configures eswitch to send/receive packets from/to
the subfunction port.

Subfunctions share PCI level resources such as PCI MSI-X IRQs with
other subfunctions and/or with its parent PCI function.

Example mlx5 software, system and device view::

_______
| admin |
| user |----------
|_______| |
| |
____|____ __|______ _________________
| | | | | |
| devlink | | tc tool | | user |
| tool | |_________| | applications |
|_________| | |_________________|
| | | |
| | | | Userspace
+---------|-------------|-------------------|----------|--------------------+
| | +----------+ +----------+ Kernel
| | | netdev | | rdma dev |
| | +----------+ +----------+
(devlink port add/del | ^ ^
port function set) | | |
| | +---------------|
_____|___ | | _______|_______
| | | | | mlx5 class |
| devlink | +------------+ | | drivers |
| kernel | | rep netdev | | |(mlx5_core,ib) |
|_________| +------------+ | |_______________|
| | | ^
(devlink ops) | | (probe/remove)
_________|________ | | ____|________
| subfunction | | +---------------+ | subfunction |
| management driver|----- | subfunction |---| driver |
| (mlx5_core) | | auxiliary dev | | (mlx5_core) |
|__________________| +---------------+ |_____________|
| ^
(sf add/del, vhca events) |
| (device add/del)
_____|____ ____|________
| | | subfunction |
| PCI NIC |---- activate/deactive events---->| host driver |
|__________| | (mlx5_core) |
|_____________|

Subfunction is created using devlink port interface.

- Change device to switchdev mode::

$ devlink dev eswitch set pci/0000:06:00.0 mode switchdev

- Add a devlink port of subfunction flaovur::

$ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
pci/0000:06:00.0/32768: type eth netdev eth6 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
function:
hw_addr 00:00:00:00:00:00 state inactive opstate detached

- Show a devlink port of the subfunction::

$ devlink port show pci/0000:06:00.0/32768
pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcisf pfnum 0 sfnum 88
function:
hw_addr 00:00:00:00:00:00 state inactive opstate detached

- Delete a devlink port of subfunction after use::

$ devlink port del pci/0000:06:00.0/32768

mlx5 function attributes
========================
The mlx5 driver provides a mechanism to setup PCI VF/SF function attributes in
a unified way for SmartNIC and non-SmartNIC.

This is supported only when the eswitch mode is set to switchdev. Port function
configuration of the PCI VF/SF is supported through devlink eswitch port.

Port function attributes should be set before PCI VF/SF is enumerated by the
driver.

MAC address setup
-----------------
mlx5 driver provides mechanism to setup the MAC address of the PCI VF/SF.

The configured MAC address of the PCI VF/SF will be used by netdevice and rdma
device created for the PCI VF/SF.

- Get the MAC address of the VF identified by its unique devlink port index::

$ devlink port show pci/0000:06:00.0/2
pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1
function:
hw_addr 00:00:00:00:00:00

- Set the MAC address of the VF identified by its unique devlink port index::

$ devlink port function set pci/0000:06:00.0/2 hw_addr 00:11:22:33:44:55

$ devlink port show pci/0000:06:00.0/2
pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1
function:
hw_addr 00:11:22:33:44:55

- Get the MAC address of the SF identified by its unique devlink port index::

$ devlink port show pci/0000:06:00.0/32768
pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcisf pfnum 0 sfnum 88
function:
hw_addr 00:00:00:00:00:00

- Set the MAC address of the VF identified by its unique devlink port index::

$ devlink port function set pci/0000:06:00.0/32768 hw_addr 00:00:00:00:88:88

$ devlink port show pci/0000:06:00.0/32768
pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcivf pfnum 0 sfnum 88
function:
hw_addr 00:00:00:00:88:88

SF state setup
--------------
To use the SF, the user must active the SF using the SF function state
attribute.

- Get the state of the SF identified by its unique devlink port index::

$ devlink port show ens2f0npf0sf88
pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
function:
hw_addr 00:00:00:00:88:88 state inactive opstate detached

- Activate the function and verify its state is active::

$ devlink port function set ens2f0npf0sf88 state active

$ devlink port show ens2f0npf0sf88
pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
function:
hw_addr 00:00:00:00:88:88 state active opstate detached

Upon function activation, the PF driver instance gets the event from the device
that a particular SF was activated. It's the cue to put the device on bus, probe
it and instantiate the devlink instance and class specific auxiliary devices
for it.

- Show the auxiliary device and port of the subfunction::

$ devlink dev show
devlink dev show auxiliary/mlx5_core.sf.4

$ devlink port show auxiliary/mlx5_core.sf.4/1
auxiliary/mlx5_core.sf.4/1: type eth netdev p0sf88 flavour virtual port 0 splittable false

$ rdma link show mlx5_0/1
link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev p0sf88

$ rdma dev show
8: rocep6s0f1: node_type ca fw 16.29.0550 node_guid 248a:0703:00b3:d113 sys_image_guid 248a:0703:00b3:d112
13: mlx5_0: node_type ca fw 16.29.0550 node_guid 0000:00ff:fe00:8888 sys_image_guid 248a:0703:00b3:d112

- Subfunction auxiliary device and class device hierarchy::

mlx5_core.sf.4
(subfunction auxiliary device)
/\
/ \
/ \
/ \
/ \
mlx5_core.eth.4 mlx5_core.rdma.4
(sf eth aux dev) (sf rdma aux dev)
| |
| |
p0sf88 mlx5_0
(sf netdev) (sf rdma device)

Additionally, the SF port also gets the event when the driver attaches to the
auxiliary device of the subfunction. This results in changing the operational
state of the function. This provides visiblity to the user to decide when is it
safe to delete the SF port for graceful termination of the subfunction.

- Show the SF port operational state::

$ devlink port show ens2f0npf0sf88
pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
function:
hw_addr 00:00:00:00:88:88 state active opstate attached

Devlink health reporters
========================

Expand Down

0 comments on commit 142d93d

Please sign in to comment.