How to Configure IPsec Transport Mode for Server-to-Server Encryption on Windows Server 2022

Understanding IPsec Transport Mode vs Tunnel Mode

IPsec (Internet Protocol Security) is a suite of protocols that authenticates and encrypts IP packets to provide secure communication over IP networks. Windows Server 2022 supports IPsec natively through Windows Defender Firewall with Advanced Security, and understanding the difference between transport mode and tunnel mode is foundational before any deployment.

In transport mode, only the payload of the IP packet is encrypted and authenticated. The original IP headers remain intact, meaning the source and destination addresses are visible to intermediate routers. This makes transport mode ideal for host-to-host or server-to-server encryption within the same network or between known endpoints, as routing is unaffected and overhead is minimal.

In tunnel mode, the entire original IP packet is encapsulated within a new IP packet with new headers. This is the mode used by VPN gateways to protect traffic between two networks across an untrusted medium, such as the internet. The original source and destination are hidden from intermediate routers.

For server-to-server encryption within a datacenter or corporate network, transport mode is the correct choice. It adds minimal overhead, keeps routing intact, and provides strong mutual authentication and payload encryption between specific servers.

Windows Firewall with Advanced Security and Connection Security Rules

Windows Defender Firewall with Advanced Security (WFAS) is the primary interface for configuring IPsec on Windows Server 2022. Within WFAS, IPsec is implemented through Connection Security Rules, not regular inbound or outbound firewall rules. Connection Security Rules define when and how IPsec negotiations occur between two endpoints.

You can open WFAS from Server Manager, by running wf.msc, or via PowerShell. There are five types of Connection Security Rules:

• Isolation – restricts connections based on authentication credentials
• Authentication Exemption – bypasses IPsec for specific computers
• Server-to-Server – authenticates connections between two specific endpoints
• Tunnel – secures connections between two gateway computers
• Custom – fully customizable rule

For server-to-server encryption, you use the Server-to-Server rule type. This ensures both endpoints authenticate before any data is exchanged and that the connection is encrypted using negotiated algorithms.

Creating IPsec Connection Security Rules with PowerShell

PowerShell provides the New-NetIPsecRule and New-NetIPsecMainModeRule cmdlets for managing IPsec. The following example creates a server-to-server IPsec transport mode rule requiring encryption between two specific servers.

First, define the IPsec phase 1 (Main Mode) and phase 2 (Quick Mode) proposals. Phase 1 establishes the authenticated, secure channel for key exchange. Phase 2 negotiates the actual data protection parameters.

# Create Main Mode (IKE Phase 1) crypto set
New-NetIPsecMainModeCryptoSet -DisplayName "Server-to-Server MM Crypto" `
    -Name "{MM-CryptoSet-001}" `
    -Proposal @(New-NetIPsecMainModeCryptoProposal `
        -Encryption AES256 `
        -KeyExchange DH14 `
        -Hash SHA256)

# Create Quick Mode (IKE Phase 2) crypto set
New-NetIPsecQuickModeCryptoSet -DisplayName "Server-to-Server QM Crypto" `
    -Name "{QM-CryptoSet-001}" `
    -Proposal @(New-NetIPsecQuickModeCryptoProposal `
        -Encryption AES256 `
        -ESPHash SHA256 `
        -Encapsulation Transport)

# Create authentication set using Kerberos
New-NetIPsecPhase1AuthSet -DisplayName "Kerberos Auth Set" `
    -Name "{Auth-Set-001}" `
    -Proposal @(New-NetIPsecAuthProposal -Machine -Kerberos)

Now create the Connection Security Rule linking these proposals to the specific server endpoints:

# Create the server-to-server Connection Security Rule
New-NetIPsecRule `
    -DisplayName "SQL-App Server-to-Server IPsec" `
    -Name "SrvToSrv-IPsec-001" `
    -InboundSecurity Require `
    -OutboundSecurity Require `
    -LocalAddress "10.10.1.20" `
    -RemoteAddress "10.10.1.30" `
    -Protocol TCP `
    -LocalPort Any `
    -RemotePort Any `
    -Mode Transport `
    -Phase1AuthSet "{Auth-Set-001}" `
    -QuickModeCryptoSet "{QM-CryptoSet-001}" `
    -MainModeCryptoSet "{MM-CryptoSet-001}"

The -InboundSecurity Require and -OutboundSecurity Require settings enforce that all traffic matching this rule must be IPsec-authenticated and encrypted. Setting these to Request instead creates a best-effort mode where IPsec is attempted but unprotected traffic is still allowed — useful during phased rollouts.

Authentication Methods: Kerberos, Certificates, and Pre-Shared Keys

IPsec authentication verifies the identity of both endpoints before establishing the security association. Windows Server 2022 supports three authentication methods for Connection Security Rules:

Kerberos v5 is the default for domain-joined computers. Both computers authenticate using their machine accounts against a domain controller. This is the simplest option for server-to-server IPsec within an Active Directory domain and requires no additional PKI infrastructure. The limitation is that both endpoints must be domain members in trusting domains.

Certificates provide the most flexible and secure option, especially for servers in different domains or workgroups. Both endpoints must have a valid machine certificate issued by a trusted CA. The certificate must be in the local computer’s certificate store and the CA must be trusted by both endpoints.

# Create certificate-based auth proposal
$certAuthProposal = New-NetIPsecAuthProposal `
    -Machine `
    -Cert `
    -Authority "CN=CorpRootCA,DC=corp,DC=example,DC=com" `
    -AuthorityType RootCA

New-NetIPsecPhase1AuthSet -DisplayName "Certificate Auth Set" `
    -Name "{Cert-Auth-Set-001}" `
    -Proposal $certAuthProposal

Pre-shared keys (PSK) should only be used for testing purposes. PSKs are stored in plaintext in the registry and provide no real security benefit over using a domain Kerberos setup. Never use PSK in production environments. To configure a PSK-based rule for testing:

$pskProposal = New-NetIPsecAuthProposal -Machine -PreSharedKey "TestOnly-NotForProd-12345"
New-NetIPsecPhase1AuthSet -DisplayName "PSK Auth Set (TEST ONLY)" `
    -Name "{PSK-Auth-Set-001}" `
    -Proposal $pskProposal

Requiring Encryption vs Requesting Encryption

The choice between Require and Request for inbound and outbound security is critical in production deployments. When set to Require, Windows drops any packet that is not IPsec-protected. When set to Request, Windows attempts IPsec negotiation but falls back to unencrypted communication if the remote endpoint does not support IPsec.

During migration or phased deployments, starting with Request on both sides allows you to verify IPsec is negotiating successfully before enforcing it. Once all servers are confirmed to be negotiating IPsec successfully, switch to Require.

# Switch an existing rule from Request to Require
Set-NetIPsecRule -Name "SrvToSrv-IPsec-001" `
    -InboundSecurity Require `
    -OutboundSecurity Require

Monitoring IPsec Security Associations

Once IPsec rules are in place and traffic is flowing, you can monitor active Security Associations (SAs) using PowerShell cmdlets. A Main Mode SA represents the authenticated IKE channel, while Quick Mode SAs represent the actual data protection tunnels.

# View all active Main Mode SAs (IKE Phase 1)
Get-NetIPsecMainModeSA | Format-List LocalAddress, RemoteAddress, `
    AuthenticationMethod, CipherAlgorithm, HashAlgorithm, `
    DHGroup, Lifetime

# View all active Quick Mode SAs (IKE Phase 2)
Get-NetIPsecQuickModeSA | Format-List LocalAddress, RemoteAddress, `
    Protocol, LocalPort, RemotePort, `
    CipherAlgorithm, IntegrityAlgorithm, `
    EncapsulationMode, OutboundSPI, InboundSPI

# Get a summary count
(Get-NetIPsecMainModeSA).Count
(Get-NetIPsecQuickModeSA).Count

If no SAs appear after configuring rules and generating traffic, IPsec negotiation has failed. Check that both endpoints have matching crypto proposals and authentication methods, and that no firewall rules are blocking UDP port 500 (IKE) or UDP port 4500 (NAT-T).

IPsec Exemptions

Certain types of traffic are automatically exempt from IPsec requirements to prevent bootstrap problems. By default, Windows exempts ICMP traffic and multicast/broadcast traffic from IPsec. You can also configure explicit authentication exemptions for specific IP addresses — for example, domain controllers used for initial Kerberos authentication.

# Create an IPsec exemption for a domain controller (so initial auth can occur)
New-NetIPsecRule `
    -DisplayName "Exempt Domain Controller" `
    -Name "Exempt-DC-001" `
    -InboundSecurity None `
    -OutboundSecurity None `
    -RemoteAddress "10.10.1.5" `
    -Enabled True

# View exemptions in WFAS
Get-NetIPsecRule | Where-Object { $_.InboundSecurity -eq "None" } | `
    Select-Object DisplayName, RemoteAddress

Without exempting domain controllers, Kerberos authentication cannot complete, which means IPsec cannot authenticate, resulting in all traffic being blocked. This is a common misconfiguration that leaves servers completely isolated.

Domain Isolation with GPO-Based IPsec

Domain isolation extends IPsec to all domain-joined computers by deploying Connection Security Rules via Group Policy. This ensures that only domain-authenticated computers can communicate with protected resources, effectively creating a logical boundary that excludes unmanaged devices.

To deploy IPsec rules via GPO, navigate to: Computer Configuration > Windows Settings > Security Settings > Windows Defender Firewall with Advanced Security > Connection Security Rules

Create a GPO with the following configuration for domain isolation:

# On a management workstation, create the GPO using PowerShell
Import-Module GroupPolicy

# Create the GPO
New-GPO -Name "Domain Isolation - IPsec Policy" -Comment "Requires IPsec for all domain traffic"

# Link to the domain
New-GPLink -Name "Domain Isolation - IPsec Policy" -Target "DC=corp,DC=example,DC=com"

# The Connection Security Rules must be created via WFAS GUI within the GPO
# or exported from a reference server and imported via GPO
# Export rules from a configured server:
$rules = Get-NetIPsecRule
$rules | Export-Clixml -Path "C:IPsecRules_Export.xml"

Troubleshooting IPsec with Network Monitor and Event Logs

When IPsec is not establishing SAs as expected, the Windows event logs contain detailed diagnostic information. The primary logs are under Applications and Services Logs > Microsoft > Windows > Windows Firewall with Advanced Security.

# View IPsec-related events from the last 24 hours
Get-WinEvent -LogName "Microsoft-Windows-Windows Firewall With Advanced Security/Firewall" `
    -MaxEvents 50 | Where-Object { $_.Id -in (5451, 5452, 5453, 5477, 5478) } | `
    Format-List TimeCreated, Id, Message

# Key event IDs:
# 5451 - Quick Mode SA established
# 5452 - Quick Mode SA ended
# 5453 - IPsec SA negotiation failed
# 5477 - Failed to add Quick Mode filter
# 5478 - IPsec Services started

# Enable IPsec audit policy
auditpol /set /subcategory:"IPsec Main Mode" /success:enable /failure:enable
auditpol /set /subcategory:"IPsec Quick Mode" /success:enable /failure:enable
auditpol /set /subcategory:"IPsec Extended Mode" /success:enable /failure:enable

For packet-level inspection, use netsh trace to capture IKE negotiation traffic:

# Start a trace capturing IPsec/IKE traffic
netsh trace start capture=yes provider=Microsoft-Windows-IPSEC-SRV `
    overwrite=yes maxsize=100 tracefile=C:ipsec_trace.etl

# Reproduce the issue, then stop the trace
netsh trace stop

# Convert to readable format (requires Message Analyzer or Microsoft Network Monitor)
# Or view directly in Event Viewer after converting with:
tracerpt C:ipsec_trace.etl -o C:ipsec_trace.xml -of XML

Additional diagnostic commands useful during IPsec troubleshooting:

# Verify IKE service is running
Get-Service PolicyAgent | Select-Object Status, StartType

# Check for IPsec policy mismatches
netsh ipsec dynamic show mmsas all
netsh ipsec dynamic show qmsas all

# Verify the Connection Security Rules are active
Get-NetIPsecRule -Enabled True | Format-Table DisplayName, `
    InboundSecurity, OutboundSecurity, Mode, LocalAddress, RemoteAddress

# Test connectivity and confirm IPsec is in use
Test-NetConnection -ComputerName 10.10.1.30 -Port 1433 -InformationLevel Detailed

IPsec transport mode on Windows Server 2022 provides robust server-to-server encryption without requiring a VPN appliance. By combining PowerShell-based rule management with GPO deployment and structured monitoring, you can enforce encrypted communication between specific servers or across your entire domain infrastructure.