Table of Contents

URL: https://www.progressiverobot.com/zero-downtime-migration-from-ingress-to-gateway/

Introduction

The Ingress NGINX controller is being deprecated. This guide provides a step-by-step process to migrate to the Gateway API on Kubernetes (DOKS) with Cilium, ensuring zero downtime for your workloads. You’ll learn how to handle TLS certificates, safely switch over DNS, and configure the the cloud provider LoadBalancer for the new gateway. The process allows you to run both Ingress and Gateway side by side, giving you time to validate production readiness before DNS cutover.

If you’re unfamiliar with Gateway concepts or best practices, review our tutorials on Gateway API with Cilium and HTTPS Traffic Routing. These resources will help you understand the new API model and configuration patterns, and highlight important differences from the traditional Ingress approach. Planning and proper testing are crucial for a smooth migration.

Important: This guide is tested on DOKS version 1.33.x. If you're on an older version, upgrade your cluster before attempting migration.

Key Takeaways

gateway illustration for: Key Takeaways
  • Zero downtime is achievable when migrating from Ingress NGINX to Gateway API on DOKS by running both controllers side by side and performing a controlled DNS cutover.
  • LoadBalancer endpoints will change: The Gateway API creates a new LoadBalancer with a new IP; plan for a short period with dual LoadBalancers and adjust DNS only after validating production traffic.
  • Annotation migration is required: Ingress NGINX and Gateway API use different approaches for configuration. the cloud provider LoadBalancer annotations move to spec.infrastructure.annotations in your Gateway resources, not metadata.annotations.
  • Certificates must be explicit: Instead of Ingress annotations, Gateway API requires separate Certificate resources. Cert-manager's solver must be configured for Gateway, not Ingress.
  • HTTP to HTTPS redirects shift to filters: Gateway API does not honor NGINX redirect annotations; use an explicit HTTPRoute with a RequestRedirect filter.
  • Test thoroughly before cutover: Use the Gateway LoadBalancer IP to validate readiness before updating DNS. Only remove Ingress after confirming stability.
  • Budget for temporary dual resources: You will be running two LoadBalancers for the duration of testing and cutover, so factor this into your migration plan.

Prerequisites

Key Migration Considerations

LoadBalancer IP Changes

Gateway creates a new the cloud provider LoadBalancer with a new IP address.

Solution: Run both LoadBalancers simultaneously, switch DNS to the new Gateway LoadBalancer, then remove the old Ingress LoadBalancer after validation.

Annotation Mapping

Ingress NGINX Gateway API Location
kubernetes.io/ingress.class: nginx spec.gatewayClassName: cilium
cert-manager.io/cluster-issuer Explicit Certificate resource
external-dns.alpha.kubernetes.io/* HTTPRoute metadata.annotations
nginx.ingress.kubernetes.io/force-ssl-redirect Separate HTTPRoute with redirect filter
service.beta.kubernetes.io/do-loadbalancer-* Gateway spec.infrastructure.annotations

the cloud provider LoadBalancer annotations must be in spec.infrastructure.annotations, not metadata.annotations. This is a common mistake that prevents the LoadBalancer from being created correctly. See the Gateway API Infrastructure documentation for details on infrastructure annotations.

Certificate Management

  • Create explicit Certificate resources (no annotation-based approach)
  • ClusterIssuer must use gatewayHTTPRoute solver instead of ingress solver

HTTP to HTTPS Redirect

Gateway API requires a separate HTTPRoute resource with RequestRedirect filter (no annotation).

Step-by-Step Migration Guide

Blue-green approach: Deploy Gateway alongside Ingress, test via IP, execute DNS cutover, monitor for stability, then cleanup. Both systems run concurrently for zero downtime. You'll pay for two LoadBalancers temporarily (typically 24-48 hours). This approach ensures you can validate the Gateway configuration before switching production traffic.

Phase 1: Prepare Gateway API Stack

Step 1: Enable Gateway API in cert-manager

To enable cert-manager to work with the Gateway API, you need to configure it to support certificate issuance for Gateway-managed routes. The following Helm upgrade command updates your cert-manager installation with the required flag:

				
					helm upgrade cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --reuse-values \
  --set extraArgs="{--enable-gateway-api=true}"

What this does:

  • helm upgrade cert-manager jetstack/cert-manager: Upgrades (or installs) cert-manager using the Jetstack Helm chart.
  • --namespace cert-manager: Targets the cert-manager namespace.
  • --reuse-values: Keeps your existing cert-manager configuration; only new settings are changed.
  • --set extraArgs="{--enable-gateway-api=true}": Adds an argument to cert-manager so it recognizes and manages Gateway API resources, not just traditional Kubernetes Ingress resources.

Make sure to include this flag alongside any existing extraArgs you have set previously. This step is required for cert-manager to issue certificates to Gateway-managed HTTPS routes.

Step 2: Create Gateway Resource

The following gateway.yaml example demonstrates how to define a Kubernetes Gateway resource for the Gateway API. Here's what each section does:

  • apiVersion/kind/metadata: Specifies the resource type (Gateway), the API group, and the resource name (my-gateway).
  • spec.gatewayClassName: Tells Kubernetes which GatewayClass to use (here, cilium).
  • spec.infrastructure.annotations: Adds the cloud provider LoadBalancer annotations. These configure the resulting Load Balancer's name, size, and health check path in the the cloud provider cloud. These annotations must be migrated from your old NGINX Ingress resource.
  • spec.listeners: Defines how the Gateway listens for network traffic:
  • Two listeners are set up—one for HTTP (port 80) and one for HTTPS (port 443)—both on the hostname www.example.com.
  • The HTTPS listener includes a TLS configuration in mode: Terminate, which tells the Gateway to decrypt incoming HTTPS traffic. It references a Kubernetes Secret (named www-tls) containing your TLS certificate.

You should update hostname, any annotations, and the referenced TLS secret name (www-tls) to match your application's configuration.

				
					apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: my-gateway
spec:
  gatewayClassName: cilium
  infrastructure:
    annotations:
      # Copy the cloud provider LoadBalancer annotations from your Ingress
      service.beta.kubernetes.io/do-loadbalancer-name: "gateway-api-lb"
      service.beta.kubernetes.io/do-loadbalancer-size-unit: "2"
      service.beta.kubernetes.io/do-loadbalancer-healthcheck-path: "/"
  listeners:
  - name: http
    protocol: HTTP
    port: 80
    hostname: "www.example.com"
  - name: https
    protocol: HTTPS
    port: 443
    hostname: "www.example.com"
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        name: www-tls

This YAML manifests a new Gateway resource that provisions a the cloud provider LoadBalancer capable of handling both HTTP and HTTPS traffic, using custom annotations and referencing the appropriate TLS certificate for secure connections.

Apply and verify:

				
					kubectl apply -f gateway.yaml
kubectl get gateway my-gateway  # Should show PROGRAMMED: True, ADDRESS assigned

Step 3: Create a ClusterIssuer for Automated HTTPS Certificates with Gateway API

What this step does: This step sets up a ClusterIssuer resource for cert-manager using the Gateway API's HTTPRoute solver. The ClusterIssuer tells cert-manager how to request and manage Let's Encrypt certificates for your domains routed through the new Gateway (instead of NGINX Ingress). By leveraging the gatewayHTTPRoute solver, certificate challenges are solved via HTTP on your Gateway, enabling automated certificate issuance and renewal for routes managed by Gateway API.

How to do it:

  • Create a file called cluster-issuer-gateway.yaml.
  • In this file, configure a ClusterIssuer resource as shown below.
  • Update email to your real email address.
  • Make sure the metadata.name is unique and does not conflict with existing ClusterIssuers.
  • The parentRefs must match the name and namespace of your Gateway from previous steps.
				
					apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod-gateway
spec:
  acme:
    email: your-email@example.com   # <- Replace with your real email address
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod-gateway-key
    solvers:
      - http01:
          gatewayHTTPRoute:
            parentRefs:
              - name: my-gateway       # The Gateway name you defined earlier
                namespace: default     # The namespace of your Gateway
                kind: Gateway
  • Apply the ClusterIssuer resource to your cluster:
				
					kubectl apply -f cluster-issuer-gateway.yaml

After applying, cert-manager will be able to issue and renew certificates using HTTP-01 challenges routed through your new Gateway, automating HTTPS for your workloads managed by the Gateway API.

Step 4: Copy Existing TLS Certificate

Copy your Ingress TLS secret for Gateway use. Since DNS still points to Ingress at this stage, you cannot issue a new certificate yet. This step allows the Gateway to use your existing certificate temporarily.

First, find your Ingress TLS secret name:

				
					kubectl get ingress <your-ingress-name> -o jsonpath='{.spec.tls[0].secretName}'

The Gateway TLS secret name should match what you specified in your Gateway resource's spec.listeners[].tls.certificateRefs[].name field (e.g., www-tls from Step 2).

Copy the secret:

				
					INGRESS_TLS_SECRET=<your-ingress-tls-secret>  # From the command above
GATEWAY_TLS_SECRET=www-tls  # Must match Gateway spec.listeners[].tls.certificateRefs[].name

kubectl get secret $INGRESS_TLS_SECRET -o yaml | \
  sed "s/name: $INGRESS_TLS_SECRET/name: $GATEWAY_TLS_SECRET/" | \
  sed '/uid:/d' | sed '/resourceVersion:/d' | sed '/creationTimestamp:/d' | \
  kubectl apply -f -

Verify:

				
					kubectl get secret $GATEWAY_TLS_SECRET  # Should show kubernetes.io/tls type

Create a Certificate resource in Phase 4 (after DNS cutover) for proper cert-manager management and renewal. Without this, your certificate will expire after 90 days and won't renew automatically.

Step 5: Create HTTPRoute for HTTPS Traffic

The following httproute.yaml example demonstrates how to define a Kubernetes HTTPRoute resource for the Gateway API. Here's what each section does:

  • apiVersion/kind/metadata: Specifies the resource type (HTTPRoute), the API group, and the resource name (www-https).
  • metadata.annotations: Adds ExternalDNS annotations for hostname and TTL. These are optional but recommended for DNS propagation tracking.
  • spec.parentRefs: References the Gateway resource (my-gateway) and specifies the section name (https) to match the Gateway listener.
  • spec.hostnames: Lists the hostnames this route will match (here, www.example.com).
  • spec.rules: Defines the routing rules for the HTTPRoute:
  • matches: Defines the path prefix (/) to match incoming requests.
  • backendRefs: References the backend service (my-www-service) and specifies the port (80) to route traffic to.

Create httproute.yaml (customize hostname and backend service):

				
					apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: www-https
  annotations:
    # Add if using ExternalDNS
    external-dns.alpha.kubernetes.io/hostname: www.example.com
    external-dns.alpha.kubernetes.io/ttl: "300"
spec:
  parentRefs:
  - name: my-gateway
    sectionName: https
  hostnames:
  - www.example.com
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: my-www-service
      port: 80
				
			
				
					kubectl apply -f httproute.yaml

Step 6: Create HTTP to HTTPS Redirect

The following code block defines a Kubernetes HTTPRoute resource that performs an HTTP to HTTPS redirect for your domain. Here's what each part does:

  • apiVersion/kind/metadata: Specifies that this is an HTTPRoute named http-redirect.
  • spec.parentRefs: Links this route to the my-gateway Gateway resource and associates it with the http section (listener) – typically listening on port 80.
  • spec.hostnames: Targets traffic for www.example.com.
  • spec.rules: Adds a rule with a RequestRedirect filter. This filter instructs the Gateway to automatically redirect any HTTP request matching this route to HTTPS (by setting scheme: https). The statusCode: 301 ensures a permanent redirect (Moved Permanently).

This resource tells the Gateway to catch all HTTP requests to your domain and send them to the secure HTTPS endpoint, improving security and ensuring consistent access over TLS.

Here's the YAML manifest for the redirect route:

				
					apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: http-redirect
spec:
  parentRefs:
  - name: my-gateway
    sectionName: http
  hostnames:
  - www.example.com
  rules:
  - filters:
    - type: RequestRedirect
      requestRedirect:
        scheme: https
        statusCode: 301

Apply the redirect by running:

				
					kubectl apply -f http-redirect.yaml

Step 7: Validate HTTPRoutes are Attached

				
					kubectl get httproute  # Should show both www-https and http-redirect
kubectl get gateway my-gateway  # Should show PROGRAMMED: True
kubectl describe httproute www-https
kubectl describe httproute http-redirect

In the describe output, check Status.Parents.Conditions for:

  • Type: Accepted with Status: True
  • Type: ResolvedRefs with Status: True

Common issues:

  • ResolvedRefs shows Status: False → Check backend Service name exists and port is correct
  • Accepted shows Status: False → Verify sectionName and hostnames match Gateway listeners

Phase 2: Validate Gateway Stack

Test via IP before DNS cutover:

				
					GATEWAY_IP=$(kubectl get gateway my-gateway -o jsonpath='{.status.addresses[0].value}')

# Test HTTP redirect (expect 301 to HTTPS, server: envoy)
curl -I --resolve www.example.com:80:$GATEWAY_IP http://www.example.com

# Test HTTPS traffic (expect 200, server: envoy)
curl -k -I --resolve www.example.com:443:$GATEWAY_IP https://www.example.com

# Verify content
curl -k --resolve www.example.com:443:$GATEWAY_IP https://www.example.com

Verify all tests return expected results before proceeding.

Phase 3: Execute DNS Cutover

See the section relevant to how you manage DNS records, either Manually or via ExternalDNS

Manual DNS Update

				
					# Optional: Lower TTL for faster rollback
doctl compute domain records update example.com --record-id <a-record-id> --record-ttl 60
sleep 300  # Wait for old TTL to expire

# Update A record to Gateway IP
doctl compute domain records update example.com --record-id <a-record-id> --record-data "$GATEWAY_IP"

# Monitor DNS propagation (Ctrl+C to stop)
while true; do echo "$(date): $(dig +short www.example.com)"; sleep 5; done

ExternalDNS with TXT Ownership Transfer

Deploy second ExternalDNS for Gateway API (customize secretKeyRef as needed):

				
					helm install external-dns-gateway external-dns/external-dns \
  --namespace external-dns \
  --set provider=the cloud provider \
  --set sources[0]=gateway-httproute \
  --set txtOwnerId=gateway \
  --set interval=1m \
  --set env[0].name=DO_TOKEN \
  --set env[0].valueFrom.secretKeyRef.name=external-dns \
  --set env[0].valueFrom.secretKeyRef.key=token

Transfer TXT ownership (triggers DNS cutover): Ensure to replace example.com domains in these commands with your domain hosted on the cloud provider.

				
					# Find TXT record. replace update pattern to find your record: a-<hostname>
TXT_RECORD_ID=$(doctl compute domain records list example.com --format ID,Type,Name --no-header | grep "TXT.*a-<hostname>" | awk '{print $1}')

# Validate TXT_RECORD_ID has a value
echo $TXT_RECORD_ID

# Transfer ownership from default to gateway
doctl compute domain records update example.com \
  --record-id $TXT_RECORD_ID \
  --record-data "heritage=external-dns,external-dns/owner=gateway,external-dns/resource=gateway/default/my-gateway"

Monitor cutover (Ctrl+C to stop):

				
					while true; do echo "$(date): $(dig +short www.example.com)"; sleep 5; done

Once the IP changes to the Gateway IP, verify with:

				
					curl -I https://www.example.com  # Should return 200, server: envoy

Phase 4: Post Migration

Step 1: Establish Certificate Management (Critical)

Create Certificate resource for proper cert-manager management and renewal:

				
					apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: www-tls-gateway
spec:
  secretName: www-tls
  issuerRef:
    name: letsencrypt-prod-gateway
    kind: ClusterIssuer
  dnsNames:
  - www.example.com
				
			
				
					kubectl apply -f certificate.yaml
kubectl get certificate www-tls-gateway -w  # Wait for READY: True

Without this Certificate resource, certificate renewal will fail after 90 days. The Certificate resource tells cert-manager to automatically renew the certificate before expiration using the ClusterIssuer you created in Phase 1, Step 3.

Step 2: Monitor Stability

Monitor for 24-48 hours before removing Ingress. Watch traffic volume, certificate READY status, error rates, and response times. This monitoring period ensures the Gateway is handling production traffic correctly before you remove the safety net of the old Ingress setup.

Step 3: Cleanup

Verify certificate management before cleanup:

				
					kubectl get certificate www-tls-gateway  # Must show READY = True
# Must show letsencrypt-prod-gateway or the name of the ClusterIssuer you created in Phase 1 Step 3
kubectl get certificate www-tls-gateway -o jsonpath='{.spec.issuerRef.name}'

Execute cleanup:

				
					helm uninstall external-dns -n external-dns  # If using ExternalDNS
helm uninstall ingress-nginx -n ingress-nginx
kubectl delete namespace ingress-nginx

Verify:

				
					curl -I https://www.example.com  # Should return 200, server: envoy
doctl compute load-balancer list --format ID,Region,Name,IP  # Should show only Gateway LoadBalancer

Rollback Procedure

Rollback if the Gateway becomes unreachable, certificate failures occur, error rates increase significantly, or critical feature gaps are discovered. The rollback process is quick (2-6 minutes) since your Ingress NGINX setup remains intact during the monitoring period.

ExternalDNS rollback:

This command updates the TXT record managed by ExternalDNS for your domain (example.com) to transfer DNS ownership and traffic routing back to the Ingress NGINX controller. It uses doctl to set the TXT record's data to reference the original Ingress resource, ensuring ExternalDNS will update the domain's A record to point back to the old Ingress LoadBalancer IP.

				
					doctl compute domain records update example.com \
  --record-id $TXT_RECORD_ID \
  --record-data "heritage=external-dns,external-dns/owner=default,external-dns/resource=ingress/default/sample-nginx"
# Wait ~60s for DNS update

Manual DNS rollback:

				
					INGRESS_IP=$(kubectl get svc -n ingress-nginx ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
doctl compute domain records update example.com --record-id <a-record-id> --record-data "$INGRESS_IP"
# Wait for DNS propagation (~60-300s)

Validate: dig +short www.example.com should show Ingress IP. Rollback time: 2-6 minutes total.

Frequently Asked Questions

1. Can I run Ingress NGINX and Gateway API simultaneously during migration?

Yes, this is the recommended approach for zero-downtime migration. Both systems can run concurrently. The Gateway API creates a new the cloud provider LoadBalancer with a different IP address, allowing you to test the Gateway configuration before switching DNS. Once you've validated the Gateway is working correctly, you update DNS to point to the new LoadBalancer IP, then remove the old Ingress NGINX setup after monitoring for stability.

2. What happens to my existing TLS certificates during migration?

Your existing TLS certificates from Ingress NGINX can be copied to the Gateway API initially. In Step 3 of Phase 1, you copy the TLS secret from your Ingress to the Gateway. However, after DNS cutover, you must create an explicit Certificate resource (as shown in Phase 4, Step 1) for proper cert-manager management and automatic renewal. Without this Certificate resource, your certificate will expire after 90 days and won't renew automatically.

3. Why do I need a separate ClusterIssuer for Gateway API?

Gateway API requires a different certificate challenge solver than Ingress. While Ingress uses the ingress solver, Gateway API uses the gatewayHTTPRoute solver. This is why you need to create a new ClusterIssuer with the gatewayHTTPRoute configuration pointing to your Gateway resource. The cert-manager must also have the --enable-gateway-api=true flag enabled to support this.

4. How long does the migration process take?

The technical migration can be completed in a few hours, but you should plan for a 24-48 hour monitoring period before removing the old Ingress NGINX setup. The actual cutover time depends on DNS propagation, which typically takes 60-300 seconds after updating DNS records. The dual LoadBalancer period (when you're paying for both) typically lasts 24-48 hours while you monitor stability.

5. What if I need to rollback after migration?

The rollback procedure is straightforward. If you're using ExternalDNS, you transfer the TXT ownership record back to the original Ingress. For manual DNS, you update the A record to point back to the Ingress LoadBalancer IP. The rollback process takes 2-6 minutes total, depending on DNS propagation. Your Ingress NGINX setup remains intact during the monitoring period, making rollback simple if needed.

Conclusion

You've successfully migrated from Ingress NGINX to Gateway API on DOKS with zero downtime. Your Gateway leverages Cilium's built-in controller and provides modern, extensible traffic management with explicit configuration and advanced routing capabilities. The Gateway API offers better separation of concerns, role-based access control, and more expressive routing options compared to the traditional Ingress API.

Next Steps

Now that you've completed the migration, explore these resources to deepen your understanding and expand your Gateway API implementation:

For production deployments, consider implementing monitoring and observability for your Gateway API setup. You can also explore advanced Gateway API features like header-based routing, traffic splitting, and integration with service meshes for more sophisticated traffic management patterns.