How to Configure IPsec on Windows Server 2025

How to Configure IPsec on Windows Server 2025

Internet Protocol Security (IPsec) is a suite of protocols that authenticates and encrypts IP packets at the network layer, providing transparent security for communications between hosts without requiring changes to applications. On Windows Server 2025, IPsec is deeply integrated into the Windows Firewall with Advanced Security (WFAS) engine and can be configured through the GUI, Group Policy, or PowerShell to protect inter-server traffic, isolate sensitive network segments, and enforce mutual authentication between systems. This guide covers creating IPsec connection security rules, selecting authentication methods and encryption algorithms, understanding transport versus tunnel mode, monitoring active security associations, and deploying IPsec policy across a domain environment using Group Policy Objects.

Prerequisites

• Windows Server 2025 with the latest cumulative update applied
• Local administrator or Domain Admin rights depending on whether you are configuring locally or via GPO
• Two or more Windows Server 2025 hosts for end-to-end testing
• An Active Directory domain with Kerberos available, or a PKI infrastructure for certificate-based authentication
• PowerShell 5.1 or later with the NetSecurity module (included in-box)
• Windows Firewall with Advanced Security enabled (enabled by default)

Step 1: Understand IPsec Modes and Rule Types

Before creating rules, it is important to understand the two IPsec operating modes and the types of connection security rules available in WFAS:

• Transport Mode: Encrypts only the payload of each IP packet while leaving the original IP header intact. Used for host-to-host communications within the same network. This is the most common mode for server-to-server protection inside a corporate network.
• Tunnel Mode: Encapsulates the entire original IP packet inside a new IP packet with a new header. Used primarily for site-to-site VPN gateways. In Windows WFAS, tunnel mode is configured on the connection security rule.
• Isolation Rules: Restrict connections to authenticated peers only.
• Authentication Exemption Rules: Exempt specific IP ranges or protocols from IPsec requirements.
• Server-to-Server Rules: Require IPsec between two specific endpoints.
• Tunnel Rules: Create a VPN tunnel between two gateways.

Step 2: Create an IPsec Connection Security Rule via PowerShell

The New-NetIPsecRule and New-NetIPsecPhase1AuthSet / New-NetIPsecPhase2AuthSet cmdlets provide full control over IPsec policy from PowerShell. The example below creates a server-to-server rule using Kerberos authentication with AES-256 encryption and SHA-256 integrity.

# Define the Phase 1 (Main Mode) authentication set — Kerberos
$p1Auth = New-NetIPsecAuthProposal -Machine -Kerberos
$mmAuthSet = New-NetIPsecPhase1AuthSet -DisplayName "IPsec-MM-Kerberos" -Proposal $p1Auth

# Define the Phase 2 (Quick Mode) cryptographic set — AES-256 + SHA-256
$p2Crypto = New-NetIPsecQuickModeCryptoProposal `
    -Encapsulation ESP `
    -ESPHash SHA256 `
    -Encryption AES256 `
    -KeyLifeTimeSeconds 3600 `
    -KeyLifeTimeKiloBytes 4608000

$qmCryptoSet = New-NetIPsecQuickModeCryptoSet `
    -DisplayName "IPsec-QM-AES256-SHA256" `
    -Proposal $p2Crypto

# Create the connection security rule (transport mode, server-to-server)
New-NetIPsecRule `
    -DisplayName "IPsec-Server-To-SQL01" `
    -InboundSecurity Require `
    -OutboundSecurity Require `
    -Phase1AuthSet $mmAuthSet.Name `
    -QuickModeCryptoSet $qmCryptoSet.Name `
    -RemoteAddress "10.10.5.20" `
    -Mode Transport `
    -Enabled True `
    -Profile Domain

Write-Host "IPsec rule created successfully."

This rule requires authenticated, encrypted IPsec for all traffic between this host and the SQL server at 10.10.5.20. If either endpoint does not support IPsec, the connection will be blocked.

Step 3: Configure Authentication Methods

Windows Server 2025 supports three authentication methods for IPsec Main Mode negotiation:

• Kerberos v5: The default in domain environments. Both machines must be domain members and able to reach a KDC. Easiest to configure — no certificates required.
• Certificate-based authentication: Requires a PKI. Each machine must have a computer certificate issued by a trusted CA. Best for cross-domain or workgroup-to-domain scenarios.
• NTLMv2: Fallback method. Less secure than Kerberos but available when Kerberos is not feasible. Should be used only when required.

# Certificate-based authentication example
# First, identify the root CA certificate thumbprint
$rootCAThumbprint = (Get-ChildItem Cert:LocalMachineRoot |
    Where-Object { $_.Subject -like "*CorpRootCA*" }).Thumbprint

$p1AuthCert = New-NetIPsecAuthProposal `
    -Machine `
    -Cert `
    -Authority ("CN=CorpRootCA,DC=corp,DC=example,DC=com") `
    -AuthorityType Root

$mmAuthSetCert = New-NetIPsecPhase1AuthSet `
    -DisplayName "IPsec-MM-Certificate" `
    -Proposal $p1AuthCert

# Combined auth: try Kerberos first, fall back to certificate
$combined = @(
    (New-NetIPsecAuthProposal -Machine -Kerberos),
    (New-NetIPsecAuthProposal -Machine -Cert -Authority "CN=CorpRootCA,DC=corp,DC=example,DC=com" -AuthorityType Root)
)
$mmAuthSetCombo = New-NetIPsecPhase1AuthSet -DisplayName "IPsec-MM-KerbOrCert" -Proposal $combined

Step 4: Create an Authentication Exemption Rule

Certain traffic — such as ICMP for troubleshooting, DHCP, DNS, and Kerberos itself (which is needed to bootstrap IPsec authentication) — must be exempted from IPsec requirements. Without these exemptions, IPsec can create circular dependency problems where authentication cannot complete because the authentication traffic itself is blocked.

# Exempt Kerberos (UDP/TCP 88) from IPsec requirements
New-NetFirewallRule `
    -DisplayName "IPsec Exemption - Kerberos" `
    -Direction Outbound `
    -Protocol UDP `
    -RemotePort 88 `
    -Action Allow `
    -Profile Domain

# Exempt LDAP for initial AD queries
New-NetFirewallRule `
    -DisplayName "IPsec Exemption - LDAP" `
    -Direction Outbound `
    -Protocol TCP `
    -RemotePort 389 `
    -Action Allow `
    -Profile Domain

# Create a connection security exemption for the domain controller IP range
New-NetIPsecRule `
    -DisplayName "IPsec Exempt - DC Subnet" `
    -RemoteAddress "10.10.1.0/24" `
    -InboundSecurity None `
    -OutboundSecurity None `
    -Mode Transport `
    -Enabled True

Step 5: Configure Tunnel Mode for Site-to-Site

If you are connecting two sites via IPsec tunnel mode — for example, two Windows Server 2025 RRAS gateways — the rule syntax differs slightly. You specify local and remote tunnel endpoints rather than host addresses.

# Tunnel mode: site A gateway 203.0.113.10  site B gateway 198.51.100.20
New-NetIPsecRule `
    -DisplayName "IPsec Tunnel SiteA-to-SiteB" `
    -Mode Tunnel `
    -LocalTunnelEndpoint "203.0.113.10" `
    -RemoteTunnelEndpoint "198.51.100.20" `
    -LocalAddress "192.168.1.0/24" `
    -RemoteAddress "192.168.2.0/24" `
    -InboundSecurity Require `
    -OutboundSecurity Require `
    -Phase1AuthSet $mmAuthSet.Name `
    -QuickModeCryptoSet $qmCryptoSet.Name `
    -Enabled True

Step 6: Monitor IPsec Security Associations

After deploying IPsec rules, verify that security associations (SAs) are being negotiated correctly. Main Mode SAs are negotiated first; Quick Mode SAs carry the actual encrypted data.

# View active Main Mode (Phase 1) Security Associations
Get-NetIPsecMainModeSA | Format-Table -AutoSize LocalAddress, RemoteAddress, AuthenticationMethod, CipherAlgorithm

# View active Quick Mode (Phase 2) Security Associations
Get-NetIPsecQuickModeSA | Format-Table -AutoSize LocalAddress, RemoteAddress, EncryptionAlgorithm, IntegrityAlgorithm, LifeTimeKiloBytes

# Count active SAs
$mmSAs = (Get-NetIPsecMainModeSA).Count
$qmSAs = (Get-NetIPsecQuickModeSA).Count
Write-Host "Main Mode SAs: $mmSAs | Quick Mode SAs: $qmSAs"

# Use netsh for legacy-style diagnostics
netsh ipsec dynamic show mmsas all
netsh ipsec dynamic show qmsas all

Step 7: Deploy IPsec Policy via Group Policy

For domain-wide enforcement, configure IPsec connection security rules in a GPO so they apply consistently to all servers in a given OU.

# Create and configure the IPsec GPO
$gpoName = "IPsec-Server-Isolation-Policy"
$domain  = (Get-ADDomain).DNSRoot
$targetOU = "OU=Servers,DC=corp,DC=example,DC=com"

New-GPO -Name $gpoName -Domain $domain
New-GPLink -Name $gpoName -Target $targetOU -LinkEnabled Yes -Enforced Yes

# Note: The WFAS connection security rules within the GPO are configured via the
# Group Policy Management Editor under:
# Computer Configuration → Windows Settings → Security Settings →
# Windows Firewall with Advanced Security → Connection Security Rules

# After manual GPO rule configuration, force update on servers:
Invoke-Command -ComputerName "SRV01","SRV02","SRV03" -ScriptBlock {
    gpupdate /force
    Write-Host "GPO updated on $env:COMPUTERNAME"
}

Step 8: Test and Verify IPsec Encryption

# Verify connectivity and check if traffic is IPsec-protected
# From the initiating server, ping and then inspect SAs
ping 10.10.5.20 -n 4

# After ping, check for Quick Mode SAs to confirm encryption is active
Get-NetIPsecQuickModeSA | Where-Object { $_.RemoteAddress -eq "10.10.5.20" } |
    Select-Object LocalAddress, RemoteAddress, EncryptionAlgorithm, IntegrityAlgorithm, ActiveDurationSeconds

# Use Network Monitor or Wireshark on the network segment to confirm
# traffic appears as ESP (protocol 50) packets rather than plaintext TCP/UDP

Conclusion

Configuring IPsec on Windows Server 2025 provides a network-layer security foundation that complements application-level TLS and firewall-based controls. By requiring mutual authentication and encrypting traffic between specific hosts or subnets, you ensure that even if an attacker gains access to the network segment, they cannot read or tamper with inter-server communications. Windows Server 2025’s integration of IPsec into Windows Firewall with Advanced Security makes policy management straightforward, and PowerShell cmdlets such as New-NetIPsecRule, Get-NetIPsecMainModeSA, and Get-NetIPsecQuickModeSA enable scripted deployment and real-time monitoring at scale. Deploy IPsec as part of a defense-in-depth strategy, particularly for protecting traffic between database servers, domain controllers, and sensitive management systems.