How to Configure IPsec Transport Mode for Server-to-Server Encryption on Windows Server 2025

Protecting east-west traffic between servers on the same network is often overlooked, yet it is one of the most critical layers of a defence-in-depth strategy. IPsec Transport Mode encrypts only the payload of each IP packet while leaving the original IP header intact, making it the ideal choice for server-to-server communication within a single network segment. Unlike Tunnel Mode, which wraps an entirely new IP header around the packet and is used for VPN gateways, Transport Mode adds minimal overhead and integrates cleanly with Windows Firewall with Advanced Security (WFAS). On Windows Server 2025, the combination of AES-256-GCM encryption, modern IKEv2 negotiation, and PowerShell-driven automation makes deploying IPsec Transport Mode a straightforward and auditable process. This tutorial walks you through designing, implementing, and verifying an IPsec Transport Mode policy between two domain-joined Windows Server 2025 hosts.

Prerequisites

  • Two Windows Server 2025 hosts (e.g., SRV-APP01 and SRV-DB01) on the same subnet, both domain-joined.
  • Active Directory domain with Kerberos working correctly (or a PKI if using certificate authentication).
  • Administrator or Domain Admin credentials on both servers.
  • PowerShell 5.1 or later (included in Windows Server 2025).
  • Windows Firewall with Advanced Security enabled and set to its default active profile.
  • Wireshark installed on a third host or a port mirror configured on your switch if you want to verify ESP traffic.

Step 1: Plan Your IPsec Policy

Before writing a single PowerShell command, define the scope of traffic you want to protect. For a typical application-to-database scenario you want to encrypt all TCP traffic on port 1433 (SQL Server) between SRV-APP01 (192.168.1.10) and SRV-DB01 (192.168.1.20). You will also need exemption rules so that ICMP (ping) and DHCP do not get blocked waiting for SA negotiation. Sketch out the following before proceeding:

  • Source and destination IP addresses or subnets.
  • Protocols and ports to protect.
  • Authentication method: Kerberos (domain computers) or certificates (cross-domain or workgroup).
  • Encryption suite: ESP with AES-256-GCM (provides both confidentiality and integrity in a single pass).
  • Key lifetime: default 480 minutes for Main Mode SA, 60 minutes for Quick Mode SA.

Step 2: Create Exemption Rules for ICMP and DHCP

Connection security rules that require IPsec will block traffic that cannot negotiate an SA. Create exemptions first so that management and infrastructure traffic is never interrupted.

# Run on BOTH servers
# Exempt ICMP from IPsec negotiation
New-NetFirewallRule -DisplayName "IPsec Exempt - ICMPv4" `
    -Direction Inbound `
    -Protocol ICMPv4 `
    -Action Allow `
    -Profile Any

New-NetFirewallRule -DisplayName "IPsec Exempt - ICMPv4 Outbound" `
    -Direction Outbound `
    -Protocol ICMPv4 `
    -Action Allow `
    -Profile Any

# Exempt DHCP (UDP 67/68)
New-NetFirewallRule -DisplayName "IPsec Exempt - DHCP" `
    -Direction Inbound `
    -Protocol UDP `
    -LocalPort 67,68 `
    -Action Allow `
    -Profile Any

Step 3: Define the IPsec Proposal (Encryption and Integrity Algorithms)

Windows Server 2025 supports AES-256-GCM for ESP, which combines encryption and authentication in a single AEAD cipher. Create a phase-2 (Quick Mode) crypto set that mandates this algorithm.

# Create a Quick Mode (Phase 2) crypto set using AES-256-GCM
$qmCryptoSet = New-NetIPsecQuickModeCryptoSet `
    -DisplayName "QM-AES256GCM" `
    -Proposal (New-NetIPsecQuickModeCryptoProposal `
        -Encryption AES256GCM `
        -ESPHash None `
        -Encapsulation ESP)

# Create a Main Mode (Phase 1) crypto set using AES-256 + SHA-384 + DH Group 20 (384-bit ECC)
$mmCryptoSet = New-NetIPsecMainModeCryptoSet `
    -DisplayName "MM-AES256-SHA384-DH20" `
    -Proposal (New-NetIPsecMainModeCryptoProposal `
        -Cipher AES256 `
        -Hash SHA384 `
        -KeyExchange DH20)

Step 4: Create the Connection Security Rule

The connection security rule enforces IPsec negotiation for the target traffic. Setting both -InboundSecurity and -OutboundSecurity to Require means the rule will drop packets that cannot be secured. Use Kerberos for authentication on domain-joined servers — no certificate infrastructure is needed.

# Run on BOTH servers — identical rule on each end
New-NetIPsecRule `
    -DisplayName "IPsec-Transport-APP-DB" `
    -Mode Transport `
    -InboundSecurity Require `
    -OutboundSecurity Require `
    -LocalAddress 192.168.1.0/24 `
    -RemoteAddress 192.168.1.0/24 `
    -Protocol TCP `
    -LocalPort Any `
    -RemotePort 1433 `
    -Phase1AuthSet (Get-NetIPsecPhase1AuthSet -DisplayName "Default" | Select-Object -First 1).Name `
    -Phase2AuthSet (Get-NetIPsecPhase2AuthSet -DisplayName "Default" | Select-Object -First 1).Name `
    -QuickModeCryptoSet $qmCryptoSet.Name `
    -MainModeCryptoSet $mmCryptoSet.Name `
    -Enabled True

# For certificate-based authentication (cross-domain or workgroup), replace the
# Phase1AuthSet with a custom set referencing your CA certificate thumbprint:
# New-NetIPsecAuthProposal -Machine -Cert -Authority "CN=MyCA,DC=contoso,DC=com" `
#     -AuthorityType RootCA

Step 5: Configure Main Mode and Quick Mode SA Lifetimes

Key lifetimes control how often keying material is refreshed. Shorter lifetimes improve forward secrecy but increase negotiation overhead. The following settings are appropriate for a high-security environment.

# Set Main Mode SA lifetime to 8 hours (480 minutes) — applied globally
Set-NetIPsecMainModeCryptoSet -DisplayName "MM-AES256-SHA384-DH20" `
    -MaxMinutes 480 `
    -MaxSessions 0

# Set Quick Mode SA lifetime to 1 hour (60 minutes)
Set-NetIPsecQuickModeCryptoSet -DisplayName "QM-AES256GCM" `
    -MaxSeconds 3600 `
    -MaxKilobytes 250000

Step 6: Verify Security Associations Are Established

After applying the rules on both servers, trigger traffic across the protected port and then inspect the SAs. A successful negotiation will show both a Main Mode SA and a Quick Mode SA.

# On SRV-APP01 — initiate a connection to SQL Server to trigger SA negotiation
Test-NetConnection -ComputerName SRV-DB01 -Port 1433

# Check Main Mode SAs (IKE Phase 1)
Get-NetIPsecMainModeSA | Format-List LocalAddress, RemoteAddress, AuthenticationMethod, `
    CipherAlgorithm, HashAlgorithm, Lifetime

# Check Quick Mode SAs (IKE Phase 2 — actual traffic encryption)
Get-NetIPsecQuickModeSA | Format-List LocalAddress, RemoteAddress, `
    Protocol, LocalPort, RemotePort, EncryptionAlgorithm, IntegrityAlgorithm

# List active connection security rule matches
Get-NetIPsecMainModeSA | Where-Object { $_.RemoteAddress -eq "192.168.1.20" }

You should see entries with EncryptionAlgorithm : AES256GCM and both LocalAddress and RemoteAddress matching your two servers. If no SAs appear, check that both servers have identical rules and that no upstream firewall is blocking UDP 500 (IKE) or UDP 4500 (NAT-T).

Step 7: Monitor Traffic with Wireshark

To confirm that traffic is actually encrypted on the wire, capture packets on the server NIC or a switch mirror port. Wireshark will show ESP (Encapsulating Security Payload) packets rather than plaintext TCP segments when IPsec Transport Mode is working correctly.

  • Filter: esp — shows all ESP-encapsulated packets.
  • Filter: isakmp — shows IKE negotiation handshake packets (UDP 500).
  • You should see no plaintext port 1433 traffic between the two servers after the SA is established.
  • ESP packets will display protocol 50 at the IP layer with the payload opaque.

Step 8: Audit and Export the Policy

# Export current connection security rules to a file for documentation
$rules = Get-NetIPsecRule | Where-Object { $_.DisplayName -like "IPsec-Transport*" }
$rules | Export-Csv -Path "C:IPsec-Policy-Audit.csv" -NoTypeInformation

# Review the full policy in human-readable form
$rules | Format-List *

# Delete an SA manually to force renegotiation (useful during testing)
Remove-NetIPsecMainModeSA -All
Remove-NetIPsecQuickModeSA -All

Conclusion

IPsec Transport Mode on Windows Server 2025 provides hardware-accelerated, authenticated encryption for server-to-server traffic with minimal configuration overhead when integrated with Active Directory Kerberos authentication. By requiring both inbound and outbound security on your connection security rules and specifying AES-256-GCM as the ESP algorithm, you ensure that sensitive data in transit — such as database queries, file share traffic, or internal API calls — cannot be read or tampered with even by a compromised host on the same VLAN. The Get-NetIPsecMainModeSA and Get-NetIPsecQuickModeSA cmdlets give you real-time visibility into negotiated SAs, while Wireshark confirms encryption at the wire level. Combine this policy with Windows Defender Firewall rules and network segmentation to build a layered, zero-trust-aligned internal network architecture.