How to Configure IPsec for Server-to-Server Encryption on Windows Server 2012 R2

November 18, 2025
Tutorials
Comment off

How to Configure IPsec for Server-to-Server Encryption on Windows Server 2012 R2

Internet Protocol Security (IPsec) provides authentication and encryption at the IP layer, ensuring that traffic between servers cannot be intercepted or tampered with even if an attacker has gained access to the network segment. On Windows Server 2012 R2, IPsec is configured through Windows Firewall with Advanced Security (WFAS), which integrates connection security rules with firewall rules in a single management interface. This guide covers creating IPsec connection security rules to require encrypted, authenticated traffic between specific servers using both command-line tools and Group Policy.

Prerequisites

Two or more Windows Server 2012 R2 servers that need encrypted communication
Both servers must be domain-joined (for Kerberos-based authentication) or have machine certificates from a shared CA (for certificate-based authentication)
Local Administrator or Domain Admin access
Windows Firewall enabled on both servers
A test window to verify connectivity after applying rules

Step 1: Understand IPsec Mode and Authentication Options

Windows Server 2012 R2 IPsec supports:

Transport mode — Encrypts only the payload, headers remain intact. Used for server-to-server communication within a LAN.
Tunnel mode — Encapsulates the entire IP packet. Used for site-to-site VPN scenarios.
Authentication methods: Kerberos (domain-joined, zero-config), Computer certificates (cross-domain or workgroup), Pre-shared key (testing only—not for production)
Encryption algorithms: AES-128, AES-192, AES-256 (avoid DES and 3DES)

Step 2: Create an IPsec Connection Security Rule via PowerShell

Create an IPsec rule on both servers requiring authenticated and encrypted communication between the two hosts:

# Run on BOTH servers involved in the encrypted communication

# Define server IP addresses
$localServer = "10.0.1.10"   # This server
$remoteServer = "10.0.1.20"  # The peer server

# Create a connection security rule requiring IPsec (ESP - Encapsulating Security Payload)
New-NetIPsecRule `
    -DisplayName "Require IPsec to Database Server" `
    -InboundSecurity Require `
    -OutboundSecurity Require `
    -RemoteAddress $remoteServer `
    -IntegrityAlgorithm SHA256 `
    -EncryptionAlgorithm AES256 `
    -KeyExchangeAlgorithm DH14 `
    -AuthenticationMethod Kerberos `
    -Description "Require ESP encryption for all traffic between App and DB servers"

# Verify the rule was created
Get-NetIPsecRule | Where-Object { $_.DisplayName -match "Database" }

Step 3: Create Connection Security Rule Using WFAS GUI

Open Windows Firewall with Advanced Security (wf.msc):

Click Connection Security Rules → New Rule
Rule Type: Isolation (for server isolation scenarios) or Server-to-Server
Endpoints: Endpoint 1 = local server IP(s); Endpoint 2 = remote server IP(s)
Requirements: Require authentication for inbound and outbound connections
Authentication Method: Computer and user (Kerberos V5) or Computer certificate
Profile: apply to all profiles (Domain, Private, Public)
Name: IPsec-AppServer-to-DBServer

Step 4: Configure Phase 1 (Main Mode) and Phase 2 (Quick Mode) Proposals

Customize the cryptographic algorithms for each IPsec phase:

# Configure Phase 1 (IKE Main Mode) proposal - key exchange and authentication
$mainModeProposal = New-NetIPsecMainModeCryptoProposal `
    -Encryption AES256 `
    -KeyExchange DH14 `
    -Hash SHA256

New-NetIPsecMainModeCryptoSet `
    -DisplayName "AES256-SHA256-DH14" `
    -Proposal $mainModeProposal

# Configure Phase 2 (Quick Mode / Child SA) proposal - data encryption
$quickModeProposal = New-NetIPsecQuickModeCryptoProposal `
    -Encryption AES256 `
    -ESPHash SHA256

New-NetIPsecQuickModeCryptoSet `
    -DisplayName "ESP-AES256-SHA256" `
    -Proposal $quickModeProposal

# Apply crypto sets to the connection security rule
Set-NetIPsecRule `
    -DisplayName "Require IPsec to Database Server" `
    -MainModeCryptoSet "AES256-SHA256-DH14" `
    -QuickModeCryptoSet "ESP-AES256-SHA256"

Step 5: Deploy IPsec Rules via Group Policy

For domain-wide deployment, configure IPsec via a GPO. Create a GPO named IPsec-ServerIsolation and navigate to:

Computer Configuration → Windows Settings → Security Settings → Windows Firewall with Advanced Security → Connection Security Rules

Right-click → New Rule → Server-to-Server, and fill in the same settings as Step 3. The GPO-deployed rules will be pushed to all computers in the linked OUs via Group Policy.

# Link the GPO to the servers OU
New-GPLink -Name "IPsec-ServerIsolation" -Target "OU=Servers,DC=corp,DC=example,DC=com"
gpupdate /force

Step 6: Verify IPsec Associations

After configuring rules on both endpoints and allowing time for negotiation, verify that IPsec security associations have been established:

# View active Main Mode security associations
Get-NetIPsecMainModeSA | Format-Table -AutoSize

# View active Quick Mode (data protection) security associations
Get-NetIPsecQuickModeSA | Format-Table -AutoSize

# View connection statistics
netsh advfirewall monitor show mmsa all
netsh advfirewall monitor show qmsa all

If no SAs are listed after initiating traffic between the servers, check the following:

# Check IPsec diagnostic events
Get-WinEvent -LogName "Microsoft-Windows-Windows Firewall With Advanced Security/ConnectionSecurity" -MaxEvents 50 |
    Select-Object TimeCreated, Id, Message | Format-List

Step 7: Configure IPsec Exemptions

Certain traffic types cannot use IPsec (ICMP for network diagnostics, DHCP for IP address acquisition). Create exemptions for essential infrastructure traffic:

# Exempt ICMP from IPsec requirement (allows ping and traceroute)
New-NetIPsecRule `
    -DisplayName "IPsec Exempt - ICMP" `
    -InboundSecurity None `
    -OutboundSecurity None `
    -Protocol ICMPv4 `
    -Description "Allow ICMP without IPsec for diagnostics"

# Exempt DHCP traffic
New-NetIPsecRule `
    -DisplayName "IPsec Exempt - DHCP" `
    -InboundSecurity None `
    -OutboundSecurity None `
    -Protocol UDP `
    -LocalPort 68 `
    -RemotePort 67

Step 8: Monitor IPsec Activity

Enable IPsec auditing to log successful and failed security associations:

auditpol /set /subcategory:"IPsec Main Mode" /success:enable /failure:enable
auditpol /set /subcategory:"IPsec Quick Mode" /success:enable /failure:enable
auditpol /set /subcategory:"IPsec Extended Mode" /success:enable /failure:enable

# Monitor for IPsec failures (Event ID 4653 = IPsec Main Mode negotiation failed)
Get-WinEvent -LogName Security -FilterXPath "*[System[EventID=4653 or EventID=4651]]" -MaxEvents 20 |
    Select-Object TimeCreated, Message

Summary

IPsec on Windows Server 2012 R2 provides transparent, network-layer encryption that protects server-to-server traffic without requiring changes to applications or services. By creating connection security rules requiring ESP encryption with AES-256 and SHA-256, deploying the configuration via Group Policy, and monitoring for negotiation failures, you have established a cryptographically protected communication channel. This is particularly valuable for database server traffic, internal API calls, and management traffic that carries sensitive data across network segments where eavesdropping is a concern.

How to Configure IPsec for Server-to-Server Encryption on Windows Server 2012 R2