How to Configure Windows Server 2019 SMB Direct

SMB Direct is an enhancement to the SMB 3 protocol that enables direct memory access transfers between servers using RDMA-capable network adapters. When available, SMB Direct allows data to be transferred between server memory over the network without involving the CPU for data movement, dramatically reducing CPU utilization and latency while increasing throughput. Windows Server 2019 supports SMB Direct automatically when RDMA-capable hardware is present. This guide covers enabling, configuring, and verifying SMB Direct.

Understanding SMB Direct Requirements

SMB Direct requires RDMA-capable network adapters. Supported RDMA technologies include iWARP (Internet Wide Area RDMA Protocol), RoCE (RDMA over Converged Ethernet) version 1 and 2, and InfiniBand. Common adapter families that support RDMA include Mellanox ConnectX series, Chelsio Terminator series, Intel X722, and QLogic FastLinQ. Both the server and the SMB client must have RDMA adapters for SMB Direct to be used. SMB 3 is available on Windows Server 2012 and later.

Checking for RDMA Adapter Support

# Check if any installed adapters support RDMA
Get-NetAdapterRdma | Select-Object Name, InterfaceDescription, Enabled

# Check adapter capabilities including RDMA
Get-NetAdapter | Where-Object { $_.Status -eq "Up" } | 
  Get-NetAdapterAdvancedProperty | 
  Where-Object { $_.DisplayName -like "*RDMA*" } |
  Select-Object Name, DisplayName, DisplayValue

# Check RDMA hardware status
Get-NetAdapterRdma | Select-Object Name, Enabled, OperationalState

# Verify SMB Direct is enabled
Get-SmbClientConfiguration | Select-Object EnableBandwidthThrottling, EnableLargeMtu, EnableMultiChannel
Get-SmbServerConfiguration | Select-Object EnableSMBDirect

Enabling and Configuring SMB Direct

# Enable SMB Direct on the server (enabled by default if RDMA hardware present)
Set-SmbServerConfiguration -EnableSMBDirect $true -Confirm:$false

# Enable RDMA on specific network adapters
Enable-NetAdapterRdma -Name "SLOT 3 Port 1"
Enable-NetAdapterRdma -Name "SLOT 3 Port 2"

# Verify RDMA is enabled on the adapter
Get-NetAdapterRdma -Name "SLOT 3 Port 1" | Select-Object Name, Enabled

# Check operational RDMA state
Get-NetAdapterRdma | Select-Object Name, Enabled, OperationalState

Configuring SMB Multichannel with RDMA

SMB Multichannel uses multiple network paths simultaneously, and when combined with RDMA adapters, provides maximum throughput. Configure and verify SMB Multichannel:

# Verify SMB Multichannel is enabled (enabled by default)
Get-SmbMultichannelConstraint
Get-SmbMultichannelConnection

# Enable Multichannel explicitly
Set-SmbClientConfiguration -EnableMultiChannel $true -Confirm:$false
Set-SmbServerConfiguration -EnableMultiChannel $true -Confirm:$false

# View active SMB Multichannel connections (shows RDMA usage)
Get-SmbMultichannelConnection | Select-Object ServerName, ClientInterfaceIndex, 
  ServerInterfaceIndex, TransportName, ThroughputConstraint, CurrentChannels

# Force specific network interfaces for SMB connections
New-SmbMultichannelConstraint -ServerName "fileserver01" -InterfaceIndex 2
New-SmbMultichannelConstraint -ServerName "fileserver01" -InterfaceIndex 3

Configuring iWARP Network Adapters for SMB Direct

# For iWARP adapters, configure QoS to prioritize RDMA traffic
# First, verify iWARP adapter is detected
Get-NetAdapterRdma | Where-Object { $_.InterfaceDescription -like "*iWARP*" }

# Set adapter MTU for iWARP (9000 for jumbo frames)
Set-NetAdapterAdvancedProperty -Name "iWARP Adapter 1" `
  -DisplayName "Jumbo Packet" -DisplayValue "9014 Bytes"

# Assign static IP to RDMA adapter (dedicated RDMA network best practice)
New-NetIPAddress -InterfaceAlias "iWARP Adapter 1" `
  -IPAddress "172.16.1.1" `
  -PrefixLength 24

# Disable IPv6 on RDMA interfaces (reduces complexity)
Set-NetAdapterBinding -Name "iWARP Adapter 1" `
  -ComponentID "ms_tcpip6" -Enabled $false

Configuring RoCE Adapters for SMB Direct

# For RoCE (RDMA over Converged Ethernet), configure PFC and ETS on the switch
# and on the adapter for lossless fabric

# Enable Priority Flow Control on the adapter
Set-NetAdapterQos -Name "RoCE Adapter 1" -Enabled $true

# Enable PFC for priority 3 (typically used for RDMA/RoCE)
Enable-NetAdapterQoS -Name "RoCE Adapter 1"

# Configure DCB (Data Center Bridging) for RoCE
# Priority 3 for SMB Direct / RDMA traffic
New-NetQosPolicy -Name "SMB-Direct" `
  -SMB `
  -PriorityValue8021Action 3

# Enable PFC on priority 3
Enable-NetQosFlowControl -Priority 3

# Disable PFC on other priorities
Disable-NetQosFlowControl -Priority 0,1,2,4,5,6,7

Verifying SMB Direct Performance

# Check SMB Direct performance counters
Get-Counter -Counter "SMB Direct Connection(*)*" -SampleInterval 1 -MaxSamples 5

# Check which SMB connections are using RDMA
Get-SmbConnection | Where-Object { $_.Dialect -ge "3.0" } |
  Select-Object ServerName, ShareName, Dialect, NumOpens

# Run a file copy and observe CPU usage (should be near 0% with RDMA)
$job = Start-Job {
    $start = Get-Date
    Copy-Item -Path "\192.168.1.100sharelargefile.vhdx" -Destination "D:Test" -Force
    $elapsed = (Get-Date) - $start
    Write-Output "Transfer completed in $($elapsed.TotalSeconds)s"
}

# Monitor CPU during transfer
while ($job.State -eq "Running") {
    $cpu = (Get-CimInstance Win32_Processor | Measure-Object LoadPercentage -Average).Average
    Write-Host "CPU: $cpu%"
    Start-Sleep -Seconds 2
}

Disabling SMB Direct for Troubleshooting

# Disable SMB Direct on the server for troubleshooting
Set-SmbServerConfiguration -EnableSMBDirect $false -Confirm:$false

# Disable RDMA on a specific adapter
Disable-NetAdapterRdma -Name "SLOT 3 Port 1"

# Re-enable after troubleshooting
Enable-NetAdapterRdma -Name "SLOT 3 Port 1"
Set-SmbServerConfiguration -EnableSMBDirect $true -Confirm:$false

SMB Direct with RDMA provides the most significant performance benefits for workloads that transfer large amounts of data between servers, such as live VM migrations, Scale-Out File Server access from Hyper-V hosts, and SQL Server storage access. Organizations that deploy RDMA infrastructure should monitor CPU savings and throughput improvements using Performance Monitor counters to quantify the return on investment from the hardware upgrade.