How to Configure IPsec Policies on Windows Server 2012 R2

How to Configure IPsec Policies on Windows Server 2012 R2

Internet Protocol Security (IPsec) provides cryptographic authentication and encryption at the IP layer, securing network communications between servers without requiring changes to applications. On Windows Server 2012 R2, IPsec policies can be configured to require encrypted communications between specific servers, authenticate traffic between network segments, or block unauthorized hosts entirely. IPsec in Windows is managed through Windows Firewall with Advanced Security, which provides both connection security rules (IPsec) and firewall rules in a unified interface. This guide covers configuring IPsec transport mode for server-to-server communication, authentication methods, and Group Policy deployment.

Prerequisites

Both communicating servers must be running Windows Server 2012 R2 or a compatible Windows version. For domain environments, Kerberos authentication (the simplest option) requires both machines to be domain-joined. Certificate-based authentication requires a PKI infrastructure. Pre-shared key authentication is available but not recommended for production as it requires storing the key on both endpoints. IPsec is managed through the Windows Firewall with Advanced Security MMC snap-in or via PowerShell. Domain Admin or local Administrator rights are required.

Understanding IPsec Modes and Protocols

IPsec on Windows Server 2012 R2 supports two modes and two security protocols:

• Transport mode — Encrypts only the payload of each IP packet; used for host-to-host communication
• Tunnel mode — Encapsulates the entire original IP packet; used for gateway-to-gateway VPN
• Authentication Header (AH) — Provides data integrity and authentication but no encryption
• Encapsulating Security Payload (ESP) — Provides confidentiality, integrity, and authentication (recommended)

For server hardening, transport mode with ESP (AES encryption + SHA authentication) is the standard configuration.

Creating IPsec Connection Security Rules via PowerShell

Windows Firewall with Advanced Security cmdlets provide full IPsec policy management. Create a connection security rule that requires IPsec between two specific servers:

# Require IPsec encryption between WebServer and DatabaseServer
# This rule is applied on the DatabaseServer to protect inbound SQL traffic

New-NetIPsecRule -DisplayName "Require IPsec from WebServer" `
    -InboundSecurity Require `
    -OutboundSecurity Request `
    -RemoteAddress "192.168.10.50" `
    -LocalAddress "192.168.10.100" `
    -Protocol TCP `
    -RemotePort 1433 `
    -Mode Transport `
    -Auth1 ComputerKerb `
    -ESP -ESPCipher AES256 -AESGCM 256 `
    -Description "Require IPsec for SQL Server traffic from WebServer"

Create a mutual authentication and encryption rule between all domain servers in a subnet:

# Require IPsec for all traffic within the server subnet
New-NetIPsecRule -DisplayName "Server Subnet IPsec Policy" `
    -InboundSecurity Require `
    -OutboundSecurity Require `
    -RemoteAddress "192.168.10.0/24" `
    -LocalAddress "192.168.10.0/24" `
    -Mode Transport `
    -Auth1 ComputerKerb `
    -Phase1AuthSet (New-NetIPsecAuthProposal -Machine -Kerberos | 
        New-NetIPsecPhase1AuthSet -DisplayName "Kerberos Auth").Name

Configuring IPsec Authentication Methods

Windows Server 2012 R2 IPsec supports multiple authentication methods. Configure the phase 1 (IKE main mode) and phase 2 (quick mode) authentication proposals:

# Create Phase 1 authentication set using Kerberos
$Phase1Auth = New-NetIPsecAuthProposal -Machine -Kerberos
$Phase1AuthSet = New-NetIPsecPhase1AuthSet -DisplayName "DomainServers-Phase1" `
    -Proposal $Phase1Auth

# Create Phase 2 authentication set (optional user auth)
$Phase2Auth = New-NetIPsecAuthProposal -Machine -Kerberos
$Phase2AuthSet = New-NetIPsecPhase2AuthSet -DisplayName "DomainServers-Phase2" `
    -Proposal $Phase2Auth

# Create a crypto set for phase 2 (ESP with AES-256 and SHA-256)
$QuickModeProposal = New-NetIPsecQuickModeCryptoProposal `
    -Encryption AES256 `
    -ESPHash SHA256 `
    -Encapsulation ESP

$QMCryptoSet = New-NetIPsecQuickModeCryptoSet -DisplayName "AES256-SHA256" `
    -Proposal $QuickModeProposal

# Apply to connection security rule
New-NetIPsecRule -DisplayName "Encrypted Server Traffic" `
    -InboundSecurity Require `
    -OutboundSecurity Require `
    -Phase1AuthSet $Phase1AuthSet.Name `
    -Phase2AuthSet $Phase2AuthSet.Name `
    -QuickModeCryptoSet $QMCryptoSet.Name `
    -Mode Transport

Configuring Certificate-Based IPsec Authentication

For servers not in the same domain or in workgroup configurations, certificate authentication provides a more scalable trust model:

# Get the CA certificate thumbprint
$CACert = Get-ChildItem -Path Cert:LocalMachineRoot | 
    Where-Object {$_.Subject -like "*YourCA*"} | 
    Select-Object -First 1

# Create Phase 1 auth using certificate
$CertAuth = New-NetIPsecAuthProposal -Machine -Cert `
    -Authority "CN=YourCA,DC=domain,DC=com" `
    -AuthorityType CA

$CertPhase1 = New-NetIPsecPhase1AuthSet -DisplayName "CertBased-Phase1" `
    -Proposal $CertAuth

# Create connection security rule with certificate auth
New-NetIPsecRule -DisplayName "Cross-Domain IPsec Rule" `
    -InboundSecurity Require `
    -OutboundSecurity Require `
    -Phase1AuthSet $CertPhase1.Name `
    -Mode Transport `
    -RemoteAddress "10.20.0.0/16"

Configuring IPsec via Windows Firewall with Advanced Security MMC

The GUI provides a wizard-driven approach for complex policies. Open Windows Firewall with Advanced Security via wf.msc. Navigate to Connection Security Rules and right-click to create a new rule. The wizard provides these rule types:

# Rule types available in the Connection Security Rule Wizard:
# - Isolation: Restricts connections based on domain membership or health status
# - Authentication Exemption: Exempt specific computers from authentication
# - Server-to-Server: Authenticate connections between two specific computers/subnets
# - Tunnel: Gateway-to-gateway tunnel (site-to-site VPN)
# - Custom: Full manual control over all parameters

Deploying IPsec Policy via Group Policy

For enterprise-wide deployment, configure IPsec connection security rules in Group Policy:

# Export local IPsec rules to a WFP policy file for import into GPO
netsh advfirewall consec export "C:IPsecPolicy.wfw"

# Import to another machine for verification
netsh advfirewall consec import "C:IPsecPolicy.wfw"

# View all connection security rules via netsh
netsh advfirewall consec show rule name=all

In Group Policy Management Console, navigate to Computer Configuration > Windows Settings > Security Settings > Windows Firewall with Advanced Security > Connection Security Rules. Right-click and create rules, or import from a .wfw file.

Monitoring IPsec Security Associations

Verify IPsec is functioning by checking active Security Associations (SAs):

# View main mode (Phase 1) security associations
Get-NetIPsecMainModeSA | Select-Object LocalAddress, RemoteAddress, 
    AuthenticationMethod, CipherAlgorithm | Format-Table -AutoSize

# View quick mode (Phase 2) security associations
Get-NetIPsecQuickModeSA | Select-Object LocalAddress, RemoteAddress, 
    Protocol, LocalPort, RemotePort | Format-Table -AutoSize

# View via netsh
netsh advfirewall monitor show mmsa all
netsh advfirewall monitor show qmsa all

Troubleshooting IPsec Connectivity

When IPsec negotiation fails, check the Windows Firewall event log:

# Check IPsec/IKE events
Get-WinEvent -LogName "Security" | Where-Object {$_.Id -in @(4650,4651,4652,4653,5451,5452)} |
    Select-Object TimeCreated, Id, Message | Format-List

# Enable IKEv2 diagnostic logging
netsh advfirewall set global ipsecdiagnostics enable

# Check for failed IKE negotiations in the System log
Get-WinEvent -FilterHashtable @{LogName='System'; ProviderName='IKEEXT'} |
    Where-Object {$_.Level -le 3} | Select-Object TimeCreated, Id, Message | Format-List

Verification

Confirm IPsec rules are applied and active:

# List all connection security rules
Get-NetIPsecRule | Select-Object DisplayName, Enabled, InboundSecurity, 
    OutboundSecurity, Mode | Format-Table -AutoSize

# Test connectivity to verify IPsec is negotiating properly
# A successful ping after adding IPsec rules means IKE negotiation succeeded
Test-NetConnection -ComputerName 192.168.10.100 -Port 1433

Summary

IPsec on Windows Server 2012 R2 provides network-layer encryption and authentication for server-to-server communications. Configure connection security rules to require IPsec between sensitive servers (web-to-database, management-to-server tiers), use Kerberos authentication for domain-joined machines and certificate authentication for cross-domain or workgroup scenarios. ESP with AES-256 and SHA-256 provides strong confidentiality and integrity. Deploy policies via Group Policy for consistency and monitor Security Associations to confirm IPsec is successfully negotiating encryption between expected endpoints.