How to Configure Windows Server 2016 RDMA Networking

Remote Direct Memory Access (RDMA) is a high-performance networking technology that enables direct memory-to-memory data transfers between networked computers without involving the host CPU or operating system. RDMA dramatically reduces latency, increases throughput, and lowers CPU utilization compared to traditional TCP/IP networking. Windows Server 2016 provides comprehensive support for RDMA across multiple transport types including InfiniBand, iWARP, and RoCE. This tutorial covers configuring RDMA networking in Windows Server 2016, including adapter setup, verification, and integration with Hyper-V networking.

RDMA Transport Types

Windows Server 2016 supports three RDMA transport technologies. InfiniBand is a specialized high-performance interconnect commonly used in HPC environments providing very low latency and high bandwidth. iWARP (RDMA over TCP/IP) uses standard Ethernet hardware at the physical layer but implements RDMA in hardware on the NIC, making it relatively easy to deploy in existing Ethernet environments. RoCE (RDMA over Converged Ethernet) encapsulates InfiniBand transport semantics over Ethernet frames and comes in two versions: RoCE v1 (layer 2 only) and RoCE v2 (layer 3 routable). RoCE requires Priority Flow Control (PFC) on lossless Ethernet fabrics for reliable operation.

Installing RDMA Network Adapters

Install the RDMA-capable network adapter in the server and install the appropriate drivers from the hardware vendor. After installation, verify the adapter is detected and RDMA is available:

Get-NetAdapter | Select-Object Name, InterfaceDescription, Status

Check RDMA capability specifically:

Get-NetAdapterRdma | Select-Object Name, InterfaceDescription, Enabled

For iWARP adapters, verify the RDMA hardware is present and enabled in the adapter properties. For RoCE adapters, additional configuration at the switch and adapter level is required.

Enabling RDMA on Network Adapters

If RDMA is not enabled by default on a capable adapter, enable it:

Enable-NetAdapterRdma -Name "RDMA_NIC1"

Verify RDMA is enabled and confirm the type of RDMA supported:

Get-NetAdapterRdma -Name "RDMA_NIC1" | Select-Object Name, Enabled, RdmaKernelModeProviderDescription

Configuring RoCE with Priority Flow Control

RoCE requires a lossless Ethernet fabric. Configure PFC on the server adapter to assign RDMA traffic to a specific priority class and prevent packet drops. First, enable QoS on the adapter:

Enable-NetAdapterQos -Name "RDMA_NIC1"

Create a QoS policy to mark SMB Direct traffic (port 445) with DSCP/priority 3:

New-NetQosPolicy -Name "SMBDirect_QoS" -NetDirectPortMatchCondition 445 -PriorityValue8021Action 3

Enable PFC for the traffic class corresponding to priority 3:

Enable-NetQosFlowControl -Priority 3

Disable PFC for all other priorities to avoid unintended flow control:

Disable-NetQosFlowControl -Priority 0,1,2,4,5,6,7

Configuring RDMA on a Hyper-V Virtual Switch

Windows Server 2016 supports RDMA on Hyper-V virtual switch host virtual NICs (vNICs) through a feature called RDMA on a converged NIC. This allows the same physical adapters to carry both Hyper-V VM traffic and host RDMA SMB traffic simultaneously. Create a Hyper-V virtual switch on the RDMA adapter:

New-VMSwitch -Name "ConvergedSwitch" -NetAdapterName "RDMA_NIC1" -EnableEmbeddedTeaming $true -AllowManagementOS $true

Add a second adapter to the switch for redundancy using Switch Embedded Teaming (SET):

Add-VMSwitchTeamMember -VMSwitchName "ConvergedSwitch" -NetAdapterName "RDMA_NIC2"

Rename the default management virtual NIC for clarity:

Rename-VMNetworkAdapter -ManagementOS -Name "ConvergedSwitch" -NewName "Management_vNIC"

Add a dedicated host vNIC for SMB Direct RDMA traffic:

Add-VMNetworkAdapter -ManagementOS -Name "SMBDirect_vNIC1" -SwitchName "ConvergedSwitch"

Enable RDMA on the SMB Direct host vNIC:

Enable-NetAdapterRdma -Name "vEthernet (SMBDirect_vNIC1)"

Verify RDMA is active on the vNIC:

Get-NetAdapterRdma | Select-Object Name, Enabled

Verifying End-to-End RDMA Connectivity

Test RDMA connectivity between two servers using the NDK (Network Direct Kernel) ping tool included with Windows Server 2016. NDK ping sends test RDMA messages between two endpoints to verify the RDMA data path is working. Run the server side on the target machine:

NdkPerfCmd.exe -S -ServerAddr 192.168.10.10 -ServerPort 9999 -TestType rPingPong

Run the client side on the source machine:

NdkPerfCmd.exe -C -ServerAddr 192.168.10.10 -ServerPort 9999 -TestType rPingPong

Monitoring RDMA Performance

Monitor RDMA performance using built-in Windows performance counters. Open Performance Monitor and add counters from the RDMA Activity category including RDMA Inbound Bytes/sec, RDMA Outbound Bytes/sec, RDMA Accepted Connections, and RDMA Active Connections. For a quick PowerShell summary of RDMA statistics:

Get-NetAdapterStatistics -Name "RDMA_NIC1" | Select-Object *Rdma*

Best Practices for RDMA Networking

Use dedicated physical network interfaces for RDMA storage traffic to isolate it from management and VM migration traffic. For RoCE deployments, ensure the Ethernet switches support and are configured for PFC and DCBX to maintain a lossless fabric. Keep RDMA adapter firmware and drivers updated to the latest versions from the hardware vendor. Use Switch Embedded Teaming (SET) with Hyper-V for redundant RDMA-capable converged NICs. Monitor RDMA connections and throughput regularly using Performance Monitor. Document your RDMA network configuration including VLAN assignments, PFC priority settings, and QoS policies for accurate troubleshooting reference.

Configuring RDMA networking on Windows Server 2016 unlocks extreme storage and inter-node communication performance for demanding workloads including Hyper-V, SQL Server, and scale-out file servers, delivering data center-grade performance at a fraction of the cost of proprietary storage networks.