How to Configure an SMTP Relay on Windows Server 2012 R2

An SMTP relay server acts as an intermediary mail gateway, accepting email from authorised internal systems (applications, printers, monitoring tools, scripts) and forwarding it to its destination — either a smart host (such as Exchange or Office 365) or directly via MX lookup. Every organisation running line-of-business applications, network devices, or monitoring systems needs an SMTP relay to consolidate outbound email flow, apply consistent relay policies, and avoid configuring SMTP credentials on every individual device.

Windows Server 2012 R2 includes the SMTP Server feature as part of IIS 6 Management Compatibility. While not a full mail server, it is entirely adequate as an internal relay. This guide covers installation, configuration, relay restrictions, TLS, and smart host forwarding.

Prerequisites

  • Windows Server 2012 R2 with static IP address.
  • TCP port 25 open in Windows Firewall (inbound) and to the smart host (outbound).
  • DNS MX/A records if receiving inbound email.
  • Smart host details (if forwarding to Exchange or Office 365): hostname and port.
  • Administrator account.

Step 1: Install the SMTP Server Feature

# Install the SMTP Server feature and IIS 6 management tools
Install-WindowsFeature -Name SMTP-Server -IncludeManagementTools

# Also install IIS 6 Management Compatibility (required to manage SMTP via GUI)
Install-WindowsFeature -Name Web-Mgmt-Compat, Web-Metabase, Web-WMI

# Verify installation
Get-WindowsFeature -Name SMTP-Server | Select-Object Name, InstallState

# Check the Simple Mail Transfer Protocol service
Get-Service -Name SMTPSVC | Select-Object Name, Status, StartType

# Start and set SMTP service to Automatic
Set-Service -Name SMTPSVC -StartupType Automatic
Start-Service -Name SMTPSVC

Step 2: Configure the SMTP Virtual Server via IIS 6 Manager

The SMTP feature in Windows Server 2012 R2 is managed through the legacy Internet Information Services (IIS) 6.0 Manager — note this is distinct from the modern IIS Manager. Open it via Server Manager → Tools → Internet Information Services (IIS) 6.0 Manager.

Expand the server node, right-click [SMTP Virtual Server #1] and select Properties.

On the General tab: set the IP address to the server’s static IP (not All Unassigned) and set the maximum connections as appropriate.

On the Access tab: configure relay restrictions (see Step 3).

On the Delivery tab: configure smart host forwarding (see Step 4).

Step 3: Configure Relay Restrictions via Registry/CMD

The most critical security configuration is relay restrictions — limiting which IP addresses are allowed to submit email through the relay. An open relay will be discovered and abused by spammers within hours.

# View current SMTP virtual server configuration via WMI
$smtp = Get-WmiObject -Namespace "rootMicrosoftIISv2" -Class "IIsSmtpVirtualServer"
$smtp | Select-Object Name, ServerComment

# Configure relay restrictions: allow only specific IPs
# This is done via IIS 6 Manager GUI:
# SMTP Virtual Server Properties -> Access tab -> Relay... button
# Select "Only the list below" and add:
# 127.0.0.1 (localhost)
# 192.168.1.0/255.255.255.0 (your internal subnet)
# Specific application server IPs

# Alternatively, configure via ADSUTIL.VBS (legacy IIS admin tool)
cscript C:WindowsSystem32iesutil.vbs SET W3SVC/1/SmtpSvc/1/RelayIPList 127.0.0.1

# Set the connection control to grant access to specific IPs only
# (done through IIS 6 Manager -> Access -> Connection button)

# Verify the SMTP pickup directory
dir "C:inetpubmailrootPickup"
dir "C:inetpubmailrootQueue"
dir "C:inetpubmailrootBadmail"

Step 4: Configure Smart Host Forwarding

A smart host is an upstream mail server that handles final delivery. Configuring a smart host means the SMTP relay doesn’t need to perform direct MX lookup delivery — it just forwards everything to Exchange, Office 365, or your ISP’s mail relay.

# Smart host configuration is done in IIS 6 Manager:
# SMTP Virtual Server Properties -> Delivery tab -> Advanced -> Smart host field
# Enter: [mail.corp.example.com] (brackets force IP lookup bypass)
# Or: smtp.office365.com for Office 365

# For Office 365 smart host with TLS:
# Smart host: smtp.office365.com
# Port: 587 (requires Outbound Security button -> TLS)

# Configure delivery retry intervals (in IIS 6 Manager -> Delivery tab)
# First retry: 15 minutes
# Second retry: 30 minutes  
# Third retry: 60 minutes
# Subsequent retries: 240 minutes
# Expiry timeout: 2 days

# Set the fully qualified domain name (FQDN) for the relay
# Delivery tab -> Advanced -> Fully qualified domain name: relay.corp.example.com

Step 5: Send a Test Email via PowerShell

Once configured, verify the relay works by sending a test message through it.

# Send a test email through the local SMTP relay
Send-MailMessage `
    -To "[email protected]" `
    -From "[email protected]" `
    -Subject "SMTP Relay Test" `
    -Body "This is a test message from the SMTP relay on Windows Server 2012 R2." `
    -SmtpServer "127.0.0.1" `
    -Port 25

# Send with credentials (if smart host requires auth)
$cred = Get-Credential
Send-MailMessage `
    -To "[email protected]" `
    -From "[email protected]" `
    -Subject "SMTP Relay Test with Auth" `
    -Body "Test message via authenticated smart host." `
    -SmtpServer "smtp.office365.com" `
    -Port 587 `
    -UseSsl `
    -Credential $cred

Step 6: Monitor and Troubleshoot the SMTP Service

# Check the SMTP queue directories
$pickup = "C:inetpubmailrootPickup"
$queue = "C:inetpubmailrootQueue"
$badmail = "C:inetpubmailrootBadmail"
$drop = "C:inetpubmailrootDrop"

# Count items in each directory
(Get-ChildItem $pickup).Count
(Get-ChildItem $queue).Count
(Get-ChildItem $badmail).Count

# Check the SMTP log
Get-Content "C:WindowsSystem32LogFilesSMTPSVC1*.log" -Tail 50

# If messages are stuck in queue, check DNS resolution
nslookup -type=MX corp.example.com
nslookup smtp.office365.com

# Restart SMTP service
Restart-Service -Name SMTPSVC

# Check SMTP connectivity to smart host
Test-NetConnection -ComputerName smtp.office365.com -Port 587

# Enable SMTP logging (IIS 6 Manager -> Virtual Server Properties -> General -> Enable logging)

Sending Email from Applications via the Relay

# Example: Configure an application to use the relay
# In any application's SMTP settings use:
# SMTP Server: 192.168.1.10  (the relay server's IP)
# Port: 25
# Authentication: None (authentication happens at the application server IP level)
# TLS: Optional (configure within IIS 6 Manager if needed)

# For .NET applications, add to web.config or app.config:
# 
#   
#     
#       
#     
#   
# 

# Test from CMD using telnet
telnet 192.168.1.10 25
# Then type:
# EHLO testclient
# MAIL FROM: 
# RCPT TO: 
# DATA
# Subject: Telnet Test
#
# Test body
# .
# QUIT

Summary

Configuring an SMTP relay on Windows Server 2012 R2 using the built-in SMTP Server feature provides a simple, reliable mail forwarding service for internal applications and devices. The essential configuration steps are: install the SMTP Server feature with IIS 6 management tools, configure relay restrictions to allow only your internal subnets and deny all others (to prevent open relay abuse), set a smart host to forward mail to your Exchange or cloud mail platform, and test delivery with Send-MailMessage. Monitor the queue and badmail directories regularly — messages accumulating in either directory indicate delivery problems that need investigation. For high-volume or business-critical mail relay, consider deploying Exchange Server or a dedicated commercial relay solution instead.