How to Configure Windows Server 2016 IPsec

IPsec (Internet Protocol Security) is a suite of protocols that provides authentication, integrity, and encryption for IP network traffic. On Windows Server 2016, IPsec can be configured through Windows Firewall with Advanced Security to protect communications between servers, isolate network segments, and enforce secure communication policies. Common uses include server-to-server traffic encryption, network isolation, and protecting sensitive management traffic.

Understanding Windows IPsec Components

  • Connection Security Rules: Define when IPsec should be applied (request, require, or exempt).
  • IKE (Internet Key Exchange): Negotiates the IPsec session parameters and key material.
  • Authentication Header (AH): Provides authentication and integrity (no encryption).
  • Encapsulating Security Payload (ESP): Provides authentication, integrity, and encryption.
  • Main Mode: Establishes the secure channel between endpoints.
  • Quick Mode: Negotiates the actual data protection parameters.

Step 1: Create an IPsec Connection Security Rule

Request IPsec (but do not require it) between two specific servers:

New-NetIPsecRule -DisplayName "Request IPsec to DBServer" `
  -InboundSecurity Request `
  -OutboundSecurity Request `
  -RemoteAddress "10.0.0.20" `
  -Authentication "ComputerCertificates" `
  -Phase1AuthSet (New-NetIKEMainModeAuthProposal -Machine) `
  -Enabled True

Step 2: Create an Isolation Rule Requiring Authentication

Require IPsec authentication for all inbound connections (domain isolation):

New-NetIPsecRule -DisplayName "Domain Isolation - Require Inbound" `
  -InboundSecurity Require `
  -OutboundSecurity Request `
  -Authentication "Kerberos" `
  -Enabled True

Step 3: Configure IPsec Encryption Settings

Create a quick mode policy specifying encryption algorithms:

$esp = New-NetIPsecMainModeSA
New-NetIPsecQuickModeCryptoProposal -Encapsulation ESP `
  -ESPHash SHA256 `
  -Encryption AESGCM256

Create a comprehensive IPsec rule with specific crypto settings:

$quickModeSet = New-NetIPsecQuickModeCryptoSet -DisplayName "AES256-SHA256" `
  -Proposal (New-NetIPsecQuickModeCryptoProposal -Encapsulation ESP -ESPHash SHA256 -Encryption AES256)

New-NetIPsecRule -DisplayName "Encrypted Comms to DBServer" `
  -InboundSecurity Require `
  -OutboundSecurity Require `
  -RemoteAddress "10.0.0.20" `
  -QuickModeCryptoSet $quickModeSet.Name `
  -Enabled True

Step 4: Configure Main Mode Authentication

Use computer certificates for authentication (more scalable than Kerberos for cross-domain scenarios):

$certProposal = New-NetIKEMainModeAuthProposal -Machine `
  -MachineAuthSet "Certificate" `
  -Cert2CAName "CN=Corp-Root-CA,DC=corp,DC=local" `
  -Cert2Type CA

New-NetIKEMainModeCryptoSet -DisplayName "Main Mode Crypto" `
  -Proposal (New-NetIKEMainModeCryptoProposal -Encryption AES256 -Hash SHA256 -KeyExchange DH14)

Step 5: View Connection Security Rules

Get-NetIPsecRule | Select-Object DisplayName, Enabled, InboundSecurity, OutboundSecurity, PrimaryStatus

View active IPsec Security Associations (SAs):

Get-NetIPsecMainModeSA
Get-NetIPsecQuickModeSA

Step 6: Create an IPsec Exemption Rule

Exempt ICMP traffic from IPsec requirements:

New-NetIPsecRule -DisplayName "Exempt ICMP" `
  -InboundSecurity None `
  -OutboundSecurity None `
  -Protocol ICMPv4 `
  -Enabled True

Step 7: Deploy IPsec Rules via Group Policy

Connection Security Rules can be deployed via Group Policy for consistency across servers. Configure at:

Computer Configuration > Windows Settings > Security Settings > Windows Firewall with Advanced Security > Connection Security Rules

Export local IPsec rules for import into GPO:

netsh advfirewall export "C:IPsecPolicy.wfw"

Import into another machine or GPO baseline:

netsh advfirewall import "C:IPsecPolicy.wfw"

Step 8: Monitor IPsec with Netsh

netsh ipsec dynamic show all
netsh advfirewall monitor show consec rule name=all
netsh advfirewall monitor show mmsa all
netsh advfirewall monitor show qmsa all

Step 9: Troubleshoot IPsec Negotiation Failures

Enable IKE diagnostic logging:

netsh ras set tracing ikeext enabled
netsh ras set tracing iashlpr enabled

Check the IKE and AuthIP IPsec Keying Modules event log:

Get-WinEvent -LogName "Microsoft-Windows-IKEEXT-Operational" -MaxEvents 20 | Select-Object TimeCreated, Id, Message

Summary

IPsec on Windows Server 2016 provides a powerful mechanism for securing server-to-server communications at the network layer. By configuring connection security rules, requiring authentication through Kerberos or certificates, and enforcing AES-256 encryption, you can protect sensitive traffic without modifying applications. Deploying IPsec via Group Policy ensures consistent enforcement across your server fleet.