Introduction to IPsec on Windows Server 2022

IPsec (Internet Protocol Security) is a suite of protocols that provides authentication and encryption at the IP layer, securing communications between hosts regardless of the application running above it. Windows Server 2022 includes a fully integrated IPsec implementation through Windows Firewall with Advanced Security (WFAS), allowing administrators to create connection security rules that enforce authenticated, encrypted communications between servers and clients. IPsec is used for domain isolation, server isolation, site-to-site VPN tunnels, and protecting lateral movement in the network.

IPsec Modes: Transport vs Tunnel

IPsec operates in two fundamental modes that determine how the protocol headers are applied to packets.

Transport Mode encrypts and authenticates only the payload (data portion) of each IP packet, leaving the original IP header intact. The original source and destination IP addresses remain visible in the header. Transport mode is used for host-to-host communication — for example, protecting SMB traffic between two domain-joined servers, or securing RPC calls between a management station and a remote server. This is the most common mode used with Windows Server 2022’s Connection Security Rules.

Tunnel Mode encapsulates the entire original IP packet (header plus payload) inside a new IP packet with new source and destination addresses. This hides the original endpoint addresses and is used for gateway-to-gateway VPN scenarios — such as connecting two branch offices over the public internet. Windows Server 2022 supports tunnel mode IPsec, but for site-to-site VPN, most organizations prefer to use a dedicated VPN appliance or DirectAccess/Always On VPN rather than raw IPsec tunnels.

Windows Firewall with Advanced Security — Connection Security Rules

Connection Security Rules (CSRs) are the mechanism in Windows Firewall with Advanced Security (WFAS) that configure IPsec. Unlike firewall rules (which permit or deny traffic), connection security rules define when and how Windows will negotiate IPsec with remote endpoints. A connection security rule does not block traffic by itself — it tells Windows to request or require that traffic matching the rule be protected by IPsec.

Open WFAS with:

wf.msc

Or use PowerShell:

Get-NetIPsecRule
New-NetIPsecRule

In the WFAS console, Connection Security Rules are listed in the left pane. Right-click to create a new rule. The rule wizard offers several types:

  • Isolation: Restricts connections based on domain membership or health status.
  • Authentication Exemption: Exempt specific subnets from IPsec requirements.
  • Server-to-Server: Authenticate and optionally encrypt traffic between specific endpoint pairs.
  • Tunnel: Create a gateway-to-gateway IPsec tunnel.
  • Custom: Full manual control over all parameters.

Creating a Server-to-Server IPsec Rule

The following steps create an IPsec rule that authenticates and encrypts traffic between two specific servers — for example, a web server at 10.0.1.20 and a database server at 10.0.1.30.

Using PowerShell on both servers:

# Create an IPsec Phase 1 (Main Mode) policy
$mmCrypto = New-NetIPsecMainModeCryptoProposal `
    -Encryption AES256 `
    -KeyExchange DH14 `
    -Hash SHA256

$mmPolicy = New-NetIPsecMainModeCryptoSet `
    -DisplayName "PAW-MM-AES256-SHA256-DH14" `
    -Proposal $mmCrypto

# Create an IPsec Phase 2 (Quick Mode) policy
$qmCrypto = New-NetIPsecQuickModeCryptoProposal `
    -Encapsulation ESP `
    -Encryption AES256 `
    -ESPHash SHA256

$qmPolicy = New-NetIPsecQuickModeCryptoSet `
    -DisplayName "PAW-QM-ESP-AES256-SHA256" `
    -Proposal $qmCrypto

# Create the connection security rule
New-NetIPsecRule `
    -DisplayName "Server-to-Server IPsec" `
    -InboundSecurity Require `
    -OutboundSecurity Require `
    -LocalAddress 10.0.1.20 `
    -RemoteAddress 10.0.1.30 `
    -Phase1AuthSet "ComputerKerberosSet" `
    -QuickModeCryptoSet $qmPolicy.Name `
    -MainModeCryptoSet $mmPolicy.Name

The InboundSecurity Require and OutboundSecurity Require settings mean that traffic matching this rule will be blocked if IPsec cannot be negotiated. Use Request instead of Require during testing to allow fallback to unencrypted traffic.

Authentication Methods

IPsec supports several authentication methods for establishing the identity of peers before exchanging encryption keys.

Kerberos V5 is the default and most convenient option for domain-joined Windows machines. Both machines authenticate using their computer accounts via the Kerberos protocol. No pre-configuration is needed beyond domain membership. This is the recommended method for domain isolation scenarios.

# View the default Kerberos authentication set
Get-NetIPsecPhase1AuthSet -DisplayName "Default"

Certificate-based authentication is used when machines are not in the same domain or when Kerberos is not available (e.g., workgroup servers, DMZ hosts). Each server must have a computer certificate from a trusted CA. Configure the authentication set:

$certAuthProposal = New-NetIPsecAuthProposal `
    -Machine `
    -Cert `
    -Authority "CN=CorpCA, DC=corp, DC=local" `
    -AuthorityType CA

$authSet = New-NetIPsecPhase1AuthSet `
    -DisplayName "CertAuth-Set" `
    -Proposal $certAuthProposal

Pre-shared Key (PSK) is the simplest to configure but the least secure — the same key is stored on both endpoints and does not scale. Use PSK only for testing or when no other option exists:

$pskProposal = New-NetIPsecAuthProposal `
    -Machine `
    -PreSharedKey "MyTestKey123!"

$pskAuthSet = New-NetIPsecPhase1AuthSet `
    -DisplayName "PSK-AuthSet" `
    -Proposal $pskProposal

Encryption Algorithms — AES-256 and SHA-256

Windows Server 2022 supports a wide range of cryptographic algorithms for IPsec. The recommended configuration for high-security environments is:

  • Key exchange: Diffie-Hellman Group 14 (2048-bit) or Group 20 (384-bit ECC) for Perfect Forward Secrecy
  • Main Mode encryption: AES-256
  • Main Mode integrity: SHA-256 or SHA-384
  • Quick Mode (data plane) encryption: AES-256 via ESP (Encapsulating Security Payload)
  • Quick Mode integrity: SHA-256 via ESP

Avoid DES, 3DES, MD5, and SHA-1 in new deployments. These algorithms are considered weak and are disabled by default in Windows Server 2022’s Schannel and IPsec stack in some configurations, but may still appear in legacy crypto sets.

Verify the active crypto configuration:

Get-NetIPsecMainModeCryptoSet | Format-List
Get-NetIPsecQuickModeCryptoSet | Format-List

IPsec Monitoring with netsh

The netsh command-line tool provides detailed IPsec statistics and active security association (SA) information.

Show all active Main Mode security associations:

netsh advfirewall monitor show mmsa

Show all active Quick Mode (data plane) security associations:

netsh advfirewall monitor show qmsa

Show connection security rule status:

netsh advfirewall monitor show consec

Using PowerShell for more detailed output:

# Show active Main Mode SAs
Get-NetIPsecMainModeSA | Format-List

# Show active Quick Mode SAs
Get-NetIPsecQuickModeSA | Format-List

# Get IPsec statistics
Get-NetIPsecStatistics

IPsec Troubleshooting — Event IDs 5451 and 5452

The Windows Security event log contains IPsec-related events that are essential for troubleshooting failed negotiations.

Event ID 5451 — IPsec Quick Mode security association established. This event is logged when a Quick Mode SA is successfully created, confirming that IPsec protection is active between two endpoints. Fields include the local and remote addresses, the selected cryptographic algorithms, and the SPI (Security Parameter Index) values.

Event ID 5452 — IPsec Quick Mode security association ended. Logged when a QM SA expires or is torn down. Normal SA teardown generates this event; unexpected teardowns may indicate a connectivity problem.

Other important IPsec event IDs:

  • 4650/4651: IPsec Main Mode security association established (4650 = no extended mode, 4651 = with extended mode)
  • 4652/4653: IPsec Main Mode negotiation failed
  • 4655: IPsec Main Mode security association ended
  • 5453: IPsec negotiation with a remote computer failed because the IKE and AuthIP IPsec Keying Modules (IKEEXT) service encountered a failure

Enable diagnostic logging for IKEEXT (the IKE key exchange service):

netsh advfirewall set global ipsec debuglog enable
netsh advfirewall set global ipsec debuglog file C:Logsipsec.log
netsh advfirewall set global ipsec debuglog size 20480

Check if the IKEEXT service is running:

Get-Service IKEEXT | Select-Object Status, StartType

IPsec with NAT Traversal

Standard IPsec ESP packets cannot pass through NAT devices because NAT modifies IP headers, breaking the integrity check on the outer IP packet. NAT Traversal (NAT-T) solves this by encapsulating IPsec packets inside UDP port 4500. Windows Server 2022 supports NAT-T automatically — when IKE detects a NAT device in the path during Main Mode negotiation, it switches to UDP encapsulation transparently.

NAT-T is enabled by default in Windows. Verify or configure it:

netsh advfirewall set global ipsec exemptions dhcp,neighbor,icmp,nodefaultexempt
netsh advfirewall show global

If NAT-T is not functioning, check that UDP port 500 (IKE) and UDP port 4500 (NAT-T) are not blocked by the firewall between the two endpoints. Also verify the IKEEXT service is running and the Windows Firewall service is active.

Domain Isolation with IPsec

Domain isolation is a scenario where IPsec is used to ensure that only domain-joined machines can communicate with each other, blocking unmanaged devices from accessing protected servers. This is implemented using a Connection Security Rule of type Isolation deployed via Group Policy to all domain members.

The isolation rule requires all inbound connections to be authenticated with Kerberos. Non-domain machines cannot authenticate and are therefore blocked from accessing protected resources. Exemptions are created for specific IP addresses (such as printers, legacy systems, or external monitoring agents) using Authentication Exemption rules.

Deploy via GPO:

Computer Configuration > Windows Settings > Security Settings > Windows Firewall with Advanced Security
  Connection Security Rules > New Rule > Isolation
  Requirements: Require authentication for inbound and outbound connections
  Authentication Method: Computer (Kerberos V5)

Test isolation effectiveness using:

# From an isolated machine, check if connections require IPsec
netsh advfirewall monitor show consec remote=10.0.1.30

IPsec in Windows Server Failover Clustering

Applying IPsec to a Windows Server Failover Cluster requires care because cluster communication uses several ports and IP addresses including the cluster IP, individual node IPs, and the cluster heartbeat network. Misconfigured IPsec rules can break cluster heartbeat communication, causing spurious failovers.

Best practices for IPsec with failover clustering:

  • Use Authentication Exemption rules for the dedicated cluster heartbeat network adapter — this network should be on an isolated, non-routed VLAN that is physically secure.
  • Ensure IPsec rules explicitly cover the cluster virtual IP addresses as well as the per-node management IPs.
  • Test IPsec rules in Request mode (not Require) first, then monitor the cluster health report before switching to Require.
  • Use Get-ClusterNetwork to enumerate cluster networks and ensure authentication exemptions cover the correct subnets.
# Exempt the cluster heartbeat network from IPsec
New-NetIPsecRule `
    -DisplayName "IPsec Exemption - Cluster Heartbeat" `
    -InboundSecurity None `
    -OutboundSecurity None `
    -LocalAddress 10.10.0.0/24 `
    -RemoteAddress 10.10.0.0/24

After deploying IPsec in a clustered environment, validate cluster health with Test-Cluster and review the cluster validation report for any network-related failures.

Summary

IPsec on Windows Server 2022 is a powerful, built-in mechanism for securing server-to-server communications without requiring application changes. Using AES-256 and SHA-256 with Kerberos authentication provides strong mutual authentication and encryption. Connection Security Rules deployed via Group Policy allow you to enforce IPsec at scale across all domain-joined machines. Understanding IPsec modes, monitoring with netsh and PowerShell, and correctly handling NAT traversal and cluster environments allows you to deploy IPsec confidently in production environments.