Documentation: mlx5.rst: Add note for eswitch MD

Add a note when using esw_port_metadata. The parameter has runtime
mode but setting it does not take effect immediately. Setting it must
happen in legacy mode, and the port metadata takes effects when the
switchdev mode is enabled.

Disable eswitch port metadata::
  $ devlink dev param set pci/0000:06:00.0 name esw_port_metadata value \
    false cmode runtime
Change eswitch mode to switchdev mode where after choosing the metadata value::
  $ devlink dev eswitch set pci/0000:06:00.0 mode switchdev

Note that other mlx5 devlink runtime parameters, esw_multiport and
flow_steering_mode, do not have this limitation.

Signed-off-by: William Tu <witu@nvidia.com>
Reviewed-by: Jiri Pirko <jiri@nvidia.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

Documentation/networking/devlink/mlx5.rst

@@ -97,6 +97,10 @@ parameters.
 
        When metadata is disabled, the above use cases will fail to initialize if
        users try to enable them.
+
+       Note: Setting this parameter does not take effect immediately. Setting
+       must happen in legacy mode and eswitch port metadata takes effect after
+       enabling switchdev mode.
    * - ``hairpin_num_queues``
      - u32
      - driverinit