How to Configure Windows Server 2019 RDMA Networking

How to Configure Windows Server 2019 RDMA Networking

Remote Direct Memory Access (RDMA) networking allows data to be transferred directly between application memory on different servers, bypassing the CPU and operating system kernel for data movement. This dramatically reduces CPU utilization, latency, and memory bandwidth consumption for high-throughput workloads. Windows Server 2019 supports RDMA through the SMB Direct protocol and provides full RDMA management through PowerShell, Network Adapter settings, and Data Center Bridging configuration. This guide covers the complete RDMA networking setup for Windows Server 2019.

RDMA Technology Overview

Three RDMA transport technologies are supported on Windows Server 2019. iWARP operates over standard TCP/IP networks and works without any special switch configuration, but has slightly higher latency than RoCE. RoCE v1 operates over Layer 2 Ethernet and requires lossless fabric (Priority Flow Control). RoCE v2 operates over UDP/IP and requires DCB-enabled switches with PFC configured. InfiniBand provides the highest performance but requires dedicated InfiniBand switch infrastructure.

Verifying RDMA Hardware

# Check all network adapters for RDMA capability
Get-NetAdapterRdma | Select-Object Name, InterfaceDescription, Enabled, OperationalState

# Get hardware details of RDMA adapters
Get-NetAdapter | ForEach-Object {
    $rdma = $_ | Get-NetAdapterRdma -ErrorAction SilentlyContinue
    if ($rdma) {
        [PSCustomObject]@{
            Name = $_.Name
            Description = $_.InterfaceDescription
            LinkSpeed = $_.LinkSpeed
            Status = $_.Status
            RDMAEnabled = $rdma.Enabled
            RDMAOperational = $rdma.OperationalState
        }
    }
} | Format-Table -AutoSize

# Check RDMA capabilities via Get-NetAdapterAdvancedProperty
Get-NetAdapterAdvancedProperty | Where-Object { $_.DisplayName -like "*RDMA*" } |
  Select-Object Name, DisplayName, DisplayValue

Enabling RDMA on Network Adapters

# Enable RDMA on specific adapters
Enable-NetAdapterRdma -Name "Slot3-Port1"
Enable-NetAdapterRdma -Name "Slot3-Port2"

# Alternatively, enable on all RDMA-capable adapters
Get-NetAdapterRdma | Where-Object { -not $_.Enabled } | Enable-NetAdapterRdma

# Configure jumbo frames on RDMA adapters (9000 bytes recommended)
Set-NetAdapterAdvancedProperty -Name "Slot3-Port1" -DisplayName "Jumbo Packet" -DisplayValue "9014 Bytes"
Set-NetAdapterAdvancedProperty -Name "Slot3-Port2" -DisplayName "Jumbo Packet" -DisplayValue "9014 Bytes"

# Verify jumbo frames are set
Get-NetAdapterAdvancedProperty -Name "Slot3-Port1" | Where-Object { $_.DisplayName -like "Jumbo*" }

Configuring a Converged NIC for RDMA

A converged NIC uses a single physical adapter for both standard Ethernet traffic and RDMA traffic using Hyper-V Virtual Switch and Switch Embedded Teaming (SET). This reduces hardware costs while maintaining RDMA performance:

# Create a Switch Embedded Teaming (SET) team with RDMA-capable adapters
New-VMSwitch -Name "ConvergedSwitch" `
  -NetAdapterName @("Slot3-Port1","Slot3-Port2") `
  -EnableEmbeddedTeaming $true `
  -AllowManagementOS $false

# Create host virtual NICs on the SET team for different traffic types
Add-VMNetworkAdapter -ManagementOS -Name "SMB-RDMA-1" -SwitchName "ConvergedSwitch"
Add-VMNetworkAdapter -ManagementOS -Name "SMB-RDMA-2" -SwitchName "ConvergedSwitch"
Add-VMNetworkAdapter -ManagementOS -Name "Management" -SwitchName "ConvergedSwitch"

# Assign IP addresses to the vNICs
New-NetIPAddress -InterfaceAlias "vEthernet (SMB-RDMA-1)" -IPAddress "10.10.20.1" -PrefixLength 24
New-NetIPAddress -InterfaceAlias "vEthernet (SMB-RDMA-2)" -IPAddress "10.10.21.1" -PrefixLength 24
New-NetIPAddress -InterfaceAlias "vEthernet (Management)" -IPAddress "192.168.1.100" -PrefixLength 24

# Enable RDMA on the host virtual NICs
Enable-NetAdapterRdma -Name "vEthernet (SMB-RDMA-1)"
Enable-NetAdapterRdma -Name "vEthernet (SMB-RDMA-2)"

# Verify RDMA is working on vNICs
Get-NetAdapterRdma | Select-Object Name, Enabled, OperationalState

Configuring Data Center Bridging for RoCE

# Install DCB feature
Install-WindowsFeature -Name "Data-Center-Bridging"

# Create QoS policy for SMB Direct traffic
New-NetQosPolicy -Name "SMBDirect-RDMA" -SMB -PriorityValue8021Action 3

# Enable Priority Flow Control for priority 3 (RDMA traffic)
Enable-NetQosFlowControl -Priority 3

# Disable PFC on other priorities to avoid head-of-line blocking
Disable-NetQosFlowControl -Priority 0,1,2,4,5,6,7

# Configure Enhanced Transmission Selection - reserve bandwidth
New-NetQosTrafficClass -Name "RDMA" -Priority 3 -BandwidthPercentage 50 -Algorithm ETS
New-NetQosTrafficClass -Name "Default" -Priority 0 -BandwidthPercentage 50 -Algorithm ETS

# Enable DCB willing on the adapter
Set-NetQosDcbxSetting -Willing $false

# Apply DCB settings to RDMA adapter
Enable-NetAdapterQoS -Name "Slot3-Port1"
Enable-NetAdapterQoS -Name "Slot3-Port2"

# Verify DCB configuration
Get-NetQosPolicy | Select-Object Name, PriorityValue
Get-NetQosFlowControl | Select-Object Priority, Enabled
Get-NetQosTrafficClass | Select-Object Name, Priority, BandwidthPercentage

Testing RDMA Connectivity

# Test RDMA connectivity between two servers
# Run on the source server
Test-RDMAConnectivity -IfIndex (Get-NetAdapter -Name "Slot3-Port1").InterfaceIndex `
  -DestinationIPAddress "10.10.20.2" `
  -Verbose

# Or use the diskspd tool for performance testing
# Run a write test using SMB Direct
diskspd -b64K -d60 -F8 -w50 -t4 -o16 -Sh -L \server2SMBSharetestfile.dat

# Monitor RDMA statistics during the test
Get-Counter -Counter "RDMA Activity(*)*" -SampleInterval 2 -MaxSamples 10 |
  Select-Object -ExpandProperty CounterSamples |
  Where-Object { $_.CookedValue -gt 0 } |
  Select-Object Path, CookedValue | Format-Table -AutoSize

A well-configured RDMA network on Windows Server 2019 can achieve near line-rate throughput (25–100 Gbps with modern adapters) at near-zero CPU cost. This is critical for Hyper-V live migration performance, Scale-Out File Server throughput, and database storage access. Document your RDMA network topology including adapter models, firmware versions, switch configuration, and PFC/ETS settings to ensure consistent configuration across all nodes in a cluster.