How to Configure IPsec Transport Mode for Server-to-Server Encryption on Windows Server 2019

April 7, 2026
Tutorials
Comment off

Introduction to IPsec Transport Mode

IPsec (Internet Protocol Security) Transport Mode encrypts and authenticates the payload of IP packets while leaving the IP header visible. Unlike Tunnel Mode (used for VPNs), Transport Mode secures traffic between two specific servers across an existing network without changing the routing path. On Windows Server 2019, IPsec Transport Mode using Windows Firewall with Advanced Security provides strong, standards-based encryption for server-to-server communication—protecting database connections, inter-service API calls, and AD replication traffic even on a trusted internal network.

IPsec Authentication Methods

Windows Server 2019 supports three primary IPsec authentication methods: Kerberos V5 (default for domain members, no certificate required), X.509 certificate (cross-domain or workgroup servers), and pre-shared key (testing only, never use in production). For domain-joined servers communicating within the same forest, Kerberos is the correct and simplest choice.

Method 1: Configure IPsec via PowerShell (Domain Members)


# Create a Connection Security Rule requiring encryption between two specific servers
# This uses Kerberos for authentication and AES-256/SHA-256 for encryption

$server1 = '10.0.1.10'  # APP-SRV01
$server2 = '10.0.1.20'  # DB-SRV01

# On BOTH servers, run the following commands:

# Step 1: Create the IPsec phase 1 (IKE) proposal - key exchange parameters
$phase1 = New-NetIPsecPhase1AuthSet `
    -DisplayName 'KerberosAuth-P1' `
    -Proposal (New-NetIPsecAuthProposal -Machine -Kerberos)

# Step 2: Create the IPsec phase 2 (ESP) proposal - encryption and integrity
$phase2 = New-NetIPsecPhase2AuthSet `
    -DisplayName 'NoAuth-P2' `
    -Proposal (New-NetIPsecAuthProposal -None)

# Step 3: Define the encryption proposal (AES-256, SHA-256)
$cryptoSet = New-NetIPsecMainModeCryptoSet `
    -DisplayName 'AES256-SHA256-DH14' `
    -Proposal (New-NetIPsecMainModeCryptoProposal `
        -Encryption AES256 `
        -Hash SHA256 `
        -KeyExchange DH14)

# Step 4: Create quick mode crypto set
$qmCrypto = New-NetIPsecQuickModeCryptoSet `
    -DisplayName 'ESP-AES256-SHA256' `
    -Proposal (New-NetIPsecQuickModeCryptoProposal `
        -Encapsulation ESP `
        -ESPHash SHA256 `
        -Encryption AES256 `
        -PfsGroup DH14)

# Step 5: Create the Connection Security Rule
New-NetIPsecRule `
    -DisplayName 'Encrypt APP to DB Traffic' `
    -Mode Transport `
    -LocalAddress $server1 `   # Run on APP-SRV01 this is local, on DB-SRV01 use $server2
    -RemoteAddress $server2 `  # Reverse on the other server
    -RequireInRequireOut `
    -InboundSecurity Require `
    -OutboundSecurity Require `
    -Phase1AuthSet $phase1.Name `
    -Phase2AuthSet $phase2.Name `
    -QuickModeCryptoSet $qmCrypto.Name

Configuring IPsec for an Entire Subnet

Rather than server pairs, you can enforce encryption for all traffic within the server VLAN (10.0.1.0/24):


# Require IPsec for all traffic within the server subnet
New-NetIPsecRule `
    -DisplayName 'Encrypt All Server-to-Server Traffic' `
    -Mode Transport `
    -LocalAddress '10.0.1.0/24' `
    -RemoteAddress '10.0.1.0/24' `
    -RequireInRequireOut `
    -InboundSecurity Require `
    -OutboundSecurity Require `
    -Protocol Any `
    -Phase1AuthSet 'KerberosAuth-P1' `
    -QuickModeCryptoSet 'ESP-AES256-SHA256'

# Exempt ICMP from encryption (optional, for ping diagnostics)
New-NetIPsecRule `
    -DisplayName 'Exempt ICMP from IPsec' `
    -Mode Transport `
    -LocalAddress '10.0.1.0/24' `
    -RemoteAddress '10.0.1.0/24' `
    -Protocol ICMPv4 `
    -InboundSecurity None `
    -OutboundSecurity None

# Exempt domain controller traffic (Kerberos, LDAP, DNS must work before IPsec is established)
New-NetIPsecRule `
    -DisplayName 'Exempt DC Traffic from IPsec' `
    -Mode Transport `
    -LocalAddress '10.0.1.0/24' `
    -RemoteAddress '10.0.0.10','10.0.0.11' `  # DC IPs
    -InboundSecurity None `
    -OutboundSecurity None

Deploying IPsec Rules via Group Policy

For fleet-wide deployment, configure IPsec Connection Security Rules via Group Policy Object:


# Create a GPO for server-to-server IPsec
$gpo = New-GPO -Name 'Server IPsec Transport Encryption'
New-GPLink -Name 'Server IPsec Transport Encryption' `
    -Target 'OU=Servers,DC=corp,DC=local'

# Configure via GPO using PowerShell (requires GroupPolicy module)
# Key GPO path:
# Computer Configuration > Windows Settings > Security Settings >
#   Windows Firewall with Advanced Security > Connection Security Rules

# Or export from a configured server and import to GPO:
netsh advfirewall consec export 'C:IPsecipsec_rules.wfw'

# Import to GPO (done via GPMC UI > right-click Connection Security Rules > Import Policy)

# Verify GPO is applied on target servers
gpresult /R /SCOPE COMPUTER | Select-String -Pattern 'IPsec|Connection Security'

# After GPO applies, check that rules are active
Get-NetIPsecRule | Where-Object { $_.Enabled -eq 'True' } | 
    Select-Object DisplayName, Mode, InboundSecurity, OutboundSecurity

Verifying IPsec Security Associations


# Check active Main Mode Security Associations (Phase 1)
Get-NetIPsecMainModeSA | Select-Object LocalAddress, RemoteAddress, `
    FirstPacketSentTime, LocalFirstPacketSentTime, AuthenticationMethod

# Check active Quick Mode Security Associations (Phase 2 - actual data encryption)
Get-NetIPsecQuickModeSA | Select-Object LocalAddress, RemoteAddress, `
    Protocol, LocalPort, RemotePort, CipherAlgorithm, IntegrityAlgorithm

# Verify traffic is encrypted (packet capture shows ESP packets instead of plaintext)
# Run on the network interface
netsh trace start capture=yes ipv4.address=10.0.1.10 tracefile=C:Tracesipsec.etl

# After capturing some traffic:
netsh trace stop
# Open .etl file in Microsoft Message Analyzer - ESP packets indicate active encryption

# Check Windows Firewall security log for IPsec events
Get-WinEvent -LogName 'Security' |
    Where-Object { $_.Id -in 4650,4651,4652,4653,5451,5452 } |
    Select-Object TimeCreated, Id, Message -First 20

Using Certificate Authentication (Cross-Domain / Workgroup)

For servers that are not in the same domain, certificate-based IPsec authentication uses certificates from the internal CA:


# Request a computer certificate for IPsec authentication
$cert = Get-Certificate `
    -Template 'IPSECIntermediateOffline' `
    -CertStoreLocation 'Cert:LocalMachineMy' `
    -SubjectName "CN=$env:COMPUTERNAME.corp.local"

# Create Phase 1 auth using certificate (instead of Kerberos)
$certAuth = New-NetIPsecAuthProposal `
    -Machine `
    -Cert `
    -Authority ('CN=Corp Issuing CA 01, O=Corp, C=US')

$phase1Cert = New-NetIPsecPhase1AuthSet `
    -DisplayName 'CertAuth-P1' `
    -Proposal $certAuth

# Create Connection Security Rule using certificate auth
New-NetIPsecRule `
    -DisplayName 'Encrypt Cross-Domain Server Traffic' `
    -Mode Transport `
    -LocalAddress '10.0.1.0/24' `
    -RemoteAddress '172.16.5.0/24' `
    -InboundSecurity Require `
    -OutboundSecurity Require `
    -Phase1AuthSet $phase1Cert.Name `
    -QuickModeCryptoSet 'ESP-AES256-SHA256'

Monitoring and Troubleshooting


# View IPsec statistics
netsh ipsec dynamic show stats

# Check for SA negotiation failures
netsh ipsec dynamic show mmsas all
netsh ipsec dynamic show qmsas all

# Common troubleshooting commands
# Test Kerberos ticket availability (required for machine auth)
klist tickets -li 0x3e7   # SYSTEM's Kerberos tickets

# Verify the computer account has a valid Kerberos ticket
Get-WinEvent -LogName System |
    Where-Object { $_.Id -in 6, 7, 8, 14, 16 } |
    Select-Object TimeCreated, Id, Message -First 10

# Flush SA database and re-negotiate (useful during testing)
netsh advfirewall consec delete rule name=all
# Wait, then re-create rules
# SAs will be renegotiated on next connection attempt

Conclusion

IPsec Transport Mode on Windows Server 2019 delivers transparent, standards-based encryption for server-to-server communication without requiring application changes or VPN infrastructure. Deploying Connection Security Rules via Group Policy ensures consistent enforcement across the entire server fleet. Kerberos authentication for domain members and certificate authentication for cross-domain scenarios provide strong mutual authentication, while AES-256 encryption and SHA-256 integrity ensure the confidentiality and authenticity of every packet exchanged between protected servers.