Skip to main content

NVMe over RoCE Express Setup

You can use NVMe with the RDMA over Converged Ethernet (RoCE) network protocol.

  1. Verify the Linux configuration is supported
    To ensure reliable operation, you create an implementation plan and then use the Lenovo Interoperability Matrix to verify that the entire configuration is supported.
  2. Configuring IP addresses using DHCP
    In this express method for configuring communications between the management station and the storage array, you use Dynamic Host Configuration Protocol (DHCP) o provide IP addresses. Each storage array has either one controller (simplex) or two controllers (duplex), and each controller has two storage management ports. Each management port will be assigned an IP address.
  3. Download and Setup SMcli from System Manager
    For ThinkSystem Software bundle 11.60.2 and newer, the ThinkSystem Secure CLI (SMcli) is included in the ThinkSystem SAN OS and downloadable through the ThinkSystem System Manager. For more information on how to download the SMcli through the ThinkSystem System Manager, refer to the Download the command line interface (CLI) topic under the Command Line Interface (CLI Reference).
  4. Access ThinkSystem System Manager and use Setup wizard
    You use the Setup wizard in ThinkSystem System Manager to configure your storage array.
  5. Configure the switch
    You configure the switches according to the vendor’s recommendations for NVMe over RoCE. These recommendations might include both configuration directives as well as code updates.
  6. Set up NVMe over RoCE on the host side
    NVMe initiator configuration in an NVMe-RoCE environment includes installing and configuring the rdma-core and nvme-cli packages, configuring initiator IP addresses, and setting up the NVMe-oF layer on the host.
  7. Configure storage array NVMe over RoCE connections
    If your controller includes a connection for NVMe over RoCE (RDMA over Converged Ethernet), you can configure the NVMe port settings from the Hardware page or the System page in ThinkSystem System Manager
  8. Discover and connect to the storage from the host
    Before making definitions of each host in ThinkSystem System Manager, you must discover the target controller ports from the host, and then establish NVMe connections.
  9. Define a host
    Using ThinkSystem System Manager, you define the hosts that send data to the storage array. Defining a host is one of the steps required to let the storage array know which hosts are attached to it and to allow I/O access to the volumes.
  10. Assign a volume
    You must assign a volume (namespace) to a host or host cluster so it can be used for I/O operations. This assignment grants a host or host cluster access to one or more namespaces in a storage array.
  11. Display the volumes visible to the host
    The SMdevices tool, part of the nvme-cli package, allows you to view the volumes currently visible on the host. This tool is an alternative to the nvme list command.
  12. Set up failover on the host
    Multipath software provides a redundant path to the storage array in case one of the physical paths is disrupted. There are currently two methods of multipathing available for NVMe, and which you will be using is going to be dependent on which OS version you are running. For SLES 12 SP5 and later, device mapper multipath (DMMP) will be used. For RHEL 7 and SLES 15, a native NVMe multipathing solution will be used.
  13. Accessing NVMe Volumes
    You can configure the I/O directed to the device target based on your Linux version.
  14. Create filesystems
    You create a file system on the namespace or native nvme device and mount the filesystem.
  15. Verify storage access on the host
    Before using the namespace, you verify that the host can write data to the namespace and read it back.
  16. Record NVMe over RoCE-specific information for Linux
    Select the NVMe over RoCE worksheet to record your protocol-specific storage configuration information. You need this information to perform provisioning tasks.