Introduction

IPsec Transport Mode encrypts traffic directly between two Windows Server 2016 hosts without requiring a VPN gateway or tunnel infrastructure. Unlike IPsec Tunnel Mode (used in site-to-site VPNs), Transport Mode encrypts only the data payload of each IP packet, leaving the original IP headers intact. This makes Transport Mode ideal for server-to-server communication within a data centre — protecting database traffic, replication streams, and administrative connections with authenticated, encrypted channels at the network layer.

IPsec Transport Mode vs Tunnel Mode

Understanding the difference is critical when choosing the correct IPsec mode. Transport Mode protects communication between two specific endpoints — each server negotiates the IPsec Security Association (SA) directly. The original source and destination IP headers are preserved, so the traffic can be routed normally. Tunnel Mode wraps the entire original IP packet in a new encrypted packet with different headers, typically used between VPN gateways across the Internet. Transport Mode has lower overhead and is appropriate for intra-datacenter traffic.

Configuring IPsec via Windows Firewall with Advanced Security

Create Connection Security Rules using the Windows Firewall with Advanced Security snap-in to enforce IPsec Transport Mode between two servers:

# Create an IPsec Transport Mode rule — require encryption between two servers
New-NetIPsecRule -DisplayName 'IPsec Transport DB-to-Web' `
    -Mode Transport `
    -LocalAddress '10.0.1.10' `
    -RemoteAddress '10.0.1.20' `
    -InboundSecurity Require `
    -OutboundSecurity Require `
    -Phase1AuthSet 'Default' `
    -Phase2AuthSet 'Default' `
    -QuickModeSecMethods 'ESP:SHA256-AES128+'

Configuring Authentication Methods

IPsec requires both endpoints to authenticate each other. Use computer certificate authentication for the strongest security:

# Create a custom Main Mode (Phase 1) auth set using computer certificates
New-NetIPsecAuthProposal -Machine -Cert `
    -Authority 'CN=Contoso Issuing CA 01,DC=contoso,DC=com' | `
    New-NetIPsecPhase1AuthSet -DisplayName 'IPsec-Cert-Auth'

# Create a Quick Mode (Phase 2) crypto set with AES-256 and SHA-256
New-NetIPsecQuickModeCryptoProposal -Encryption AES256 -ESPHash SHA256 `
    -PfsGroup Group2 | `
    New-NetIPsecQuickModeCryptoSet -DisplayName 'IPsec-AES256-Crypto'

# Apply the auth set and crypto set to the connection security rule
New-NetIPsecRule -DisplayName 'IPsec SQL Server Encryption' `
    -Mode Transport `
    -LocalAddress '10.0.1.10' `
    -RemoteAddress '10.0.1.30' `
    -InboundSecurity Require `
    -OutboundSecurity Require `
    -Phase1AuthSet 'IPsec-Cert-Auth' `
    -QuickModeSecMethods 'ESP:SHA256-AES256+'

Using Pre-Shared Keys for Lab Environments

For testing in lab environments without a PKI, pre-shared key authentication can be used (not recommended for production):

# Create PSK auth proposal
New-NetIPsecAuthProposal -Machine -Presharedkey 'Lab-PreSharedKey-2016!' |
    New-NetIPsecPhase1AuthSet -DisplayName 'IPsec-PSK-Auth'

# Apply to a connection security rule
New-NetIPsecRule -DisplayName 'IPsec PSK Test Rule' `
    -Mode Transport `
    -RemoteAddress '10.0.1.0/24' `
    -InboundSecurity Request `
    -OutboundSecurity Request `
    -Phase1AuthSet 'IPsec-PSK-Auth'

Verifying IPsec Security Associations

Confirm that IPsec SAs are being negotiated and traffic is being encrypted between the servers:

# List active IPsec Main Mode Security Associations
Get-NetIPsecMainModeSA | Select-Object LocalAddress,RemoteAddress,AuthenticationMethod,CipherAlgorithm

# List active Quick Mode (data encryption) SAs
Get-NetIPsecQuickModeSA | Select-Object LocalAddress,RemoteAddress,TransformSet,TrafficSelectorLocal

# View connection security rule status
Get-NetIPsecRule | Select-Object DisplayName,Enabled,Mode,InboundSecurity,OutboundSecurity

# Monitor IPsec traffic statistics
netsh ipsec dynamic show all

# Check IPsec event log
Get-WinEvent -LogName 'Security' -FilterXPath "*[System[EventID=4977 or EventID=4978 or EventID=5451]]" -MaxEvents 20 |
    Select-Object TimeCreated,Id,Message | Format-List

Exempting ICMP and Discovery Traffic

Ensure that ICMP and essential discovery traffic can bypass IPsec to avoid connectivity issues:

# Allow ICMP without IPsec (required for ping diagnostics)
New-NetFirewallRule -DisplayName 'ICMP IPsec Exempt' `
    -Protocol ICMPv4 -Action Allow -Profile Any

# Create an IPsec exemption for ICMP
New-NetIPsecRule -DisplayName 'ICMP Exemption' `
    -Mode Transport `
    -Protocol ICMPv4 `
    -InboundSecurity None `
    -OutboundSecurity None

Monitoring with Performance Monitor

Track IPsec performance counters to confirm traffic is being encrypted and no negotiation failures occur:

# Add IPsec counters to performance monitor
$counters = @(
    'IPsec IKEv1 IPv4Active Main Mode SAs',
    'IPsec IKEv1 IPv4Active Quick Mode SAs',
    'IPsec IKEv1 IPv4Authentication Failures',
    'IPsec IKEv2 IPv4Active Main Mode SAs'
)
Get-Counter -Counter $counters -SampleInterval 5 -MaxSamples 3

Summary

IPsec Transport Mode on Windows Server 2016 delivers transparent, policy-driven encryption between specific servers without any changes to application code. By using certificate-based authentication and AES-256 encryption, this approach protects sensitive inter-server traffic — such as database queries, replication streams, and management traffic — at the network layer. Once the connection security rules are in place, all matching traffic is automatically encrypted by the Windows TCP/IP stack, providing strong data-in-transit protection throughout the data centre.