Graceful Shutdown and Hitless Upgrades

Envoy Gateway enables zero-downtime deployments through graceful connection draining during pod termination.

Overview

The shutdown manager sidecar coordinates graceful connection draining during pod termination, providing:

• Zero-downtime rolling updates
• Configurable drain timeouts
• Automatic health check failure to remove pods from load balancer rotation

Shutdown Process

1. Kubernetes sends SIGTERM to the pod’s containers and marks the pod as terminating.
2. Shutdown manager starts Envoy listener drain.
   • Drain is initiated directly via /drain_listeners?graceful&skip_exit or indirectly via /healthcheck/fail when no health check failure delay is configured.
   • Envoy continues to serve accepted connections while listeners are draining.
3. Shutdown manager fails health checks via /healthcheck/fail, causing the pod’s readiness probe to fail.
   • By default this happens immediately and also starts listener drain.
   • When healthCheckFailureDelay is configured, this step is delayed without delaying listener drain, connection monitoring, or the drain timeout.
4. Connection monitoring begins, polling server.total_connections
5. Process exits when connections reach zero or drain timeout is exceeded

Configuration

Graceful shutdown behavior includes default values that can be overridden using the EnvoyProxy resource. The EnvoyProxy resource can be referenced in two ways:

1. Gateway-level: Referenced from a Gateway via infrastructure.parametersRef
2. GatewayClass-level: Referenced from a GatewayClass via parametersRef

Default Values:

• drainTimeout: 60 seconds - Maximum time for connection draining
• minDrainDuration: 10 seconds - Minimum wait before allowing exit
• healthCheckFailureDelay: 0 seconds - Optional delay before failing health checks after drain starts

healthCheckFailureDelay does not extend the drain sequence or keep the pod’s containers running. If the drain completes before healthCheckFailureDelay elapses, /healthcheck/fail is not called. This can happen when connections drop below exitAtConnections after minDrainDuration, or when healthCheckFailureDelay is greater than or equal to drainTimeout.

• Gateway-Level Configuration
• GatewayClass-Level Configuration

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg
spec:
  gatewayClassName: eg
  infrastructure:
    parametersRef:
      group: gateway.envoyproxy.io
      kind: EnvoyProxy
      name: graceful-shutdown-config
  listeners:
  - name: http
    port: 80
    protocol: HTTP
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: graceful-shutdown-config
spec:
  shutdown:
    drainTimeout: "90s"              # Override default 60s
    minDrainDuration: "15s"          # Override default 10s
    healthCheckFailureDelay: "40s"   # Override default 0s

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
  parametersRef:
    group: gateway.envoyproxy.io
    kind: EnvoyProxy
    name: graceful-shutdown-config
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: graceful-shutdown-config
spec:
  shutdown:
    drainTimeout: "90s"              # Override default 60s
    minDrainDuration: "15s"          # Override default 10s
    healthCheckFailureDelay: "40s"   # Override default 0s