Unix Signals and Graceful Shutdown Patterns

Introduction#

When Kubernetes terminates a pod, or a user presses Ctrl+C, the OS sends a signal to your process. How your application responds determines whether in-flight requests complete cleanly or get abruptly cut off. Graceful shutdown is a basic reliability requirement, but it requires explicit implementation.

Signal Basics#

Signals are asynchronous notifications sent to a process. Common signals:

Signal	Number	Default Action	Common Use
`SIGTERM`	15	Terminate	Polite shutdown request
`SIGINT`	2	Terminate	Ctrl+C
`SIGKILL`	9	Terminate (unblockable)	Force kill
`SIGHUP`	1	Terminate	Reload config
`SIGQUIT`	3	Core dump	Debug dump
`SIGUSR1/2`	10/12	Terminate	Application-defined

SIGKILL and SIGSTOP cannot be caught, blocked, or ignored — they are handled by the kernel directly.

Kubernetes Shutdown Sequence#

Understanding signals in Kubernetes:

Pod receives delete request
Kubernetes sends SIGTERM to PID 1 in the container
Kubernetes waits terminationGracePeriodSeconds (default 30s)
If process still running: sends SIGKILL

Your application must complete in-flight work and exit cleanly within the grace period.

Graceful Shutdown in Python (FastAPI)#

import asyncio
import signal
import logging
from contextlib import asynccontextmanager
from fastapi import FastAPI
import uvicorn

logger = logging.getLogger(__name__)

# Track active requests
active_requests = 0
shutdown_event = asyncio.Event()

@asynccontextmanager
async def lifespan(app: FastAPI):
    # Startup
    logger.info("Application starting")
    yield
    # Shutdown: wait for active requests to complete
    logger.info("Shutdown initiated, waiting for active requests")
    while active_requests > 0:
        await asyncio.sleep(0.1)
    logger.info("All requests complete, shutting down")

app = FastAPI(lifespan=lifespan)

@app.middleware("http")
async def track_requests(request, call_next):
    global active_requests
    active_requests += 1
    try:
        response = await call_next(request)
        return response
    finally:
        active_requests -= 1

@app.get("/health")
async def health():
    if shutdown_event.is_set():
        # Return 503 during shutdown so load balancer stops routing
        from fastapi import Response
        return Response(status_code=503, content="shutting down")
    return {"status": "ok"}

def handle_sigterm(sig, frame):
    logger.info("SIGTERM received")
    shutdown_event.set()

signal.signal(signal.SIGTERM, handle_sigterm)
signal.signal(signal.SIGINT, handle_sigterm)

if __name__ == "__main__":
    config = uvicorn.Config(app, host="0.0.0.0", port=8000)
    server = uvicorn.Server(config)
    server.run()

Graceful Shutdown in Go#

package main

import (
    "context"
    "log"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"
)

func main() {
    mux := http.NewServeMux()
    mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        // Simulate work
        time.Sleep(2 * time.Second)
        w.Write([]byte("OK"))
    })

    server := &http.Server{
        Addr:    ":8080",
        Handler: mux,
    }

    // Start server in a goroutine
    go func() {
        log.Println("Server starting on :8080")
        if err := server.ListenAndServe(); err != http.ErrServerClosed {
            log.Fatalf("Server error: %v", err)
        }
    }()

    // Wait for shutdown signal
    quit := make(chan os.Signal, 1)
    signal.Notify(quit, syscall.SIGTERM, syscall.SIGINT)
    <-quit

    log.Println("Shutdown signal received")

    // Give in-flight requests 30 seconds to complete
    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()

    if err := server.Shutdown(ctx); err != nil {
        log.Fatalf("Forced shutdown: %v", err)
    }

    log.Println("Server stopped gracefully")
}

http.Server.Shutdown stops accepting new connections and waits for active handlers to return.

Signal Handling in Containers: PID 1 Problem#

Shell scripts and some base images run your application as a subprocess of a shell. Shells do not forward signals to child processes by default.

# BAD: shell form — /bin/sh is PID 1, does not forward SIGTERM
CMD python app.py

# GOOD: exec form — python is PID 1, receives signals directly
CMD ["python", "app.py"]

If you need a shell wrapper for environment variable substitution:

#!/bin/bash
# Use exec to replace the shell with your process
# This makes your app PID 1 in the container
exec python app.py

Or use tini as a minimal init process:

FROM python:3.12-slim
RUN apt-get install -y tini
ENTRYPOINT ["/usr/bin/tini", "--"]
CMD ["python", "app.py"]

tini properly forwards signals and reaps zombie processes.

SIGHUP for Config Reload#

Many servers reload configuration on SIGHUP without restarting.

import signal
import json
import logging

config = {}

def load_config():
    global config
    with open("/etc/app/config.json") as f:
        config = json.load(f)
    logging.info("Config reloaded")

def handle_sighup(sig, frame):
    logging.info("SIGHUP received — reloading config")
    load_config()

signal.signal(signal.SIGHUP, handle_sighup)
load_config()

# Reload config without restart
kill -HUP $(pgrep -f "python app.py")

# Or in Kubernetes: trigger pod restart by updating a ConfigMap annotation
kubectl patch deployment my-app -p \
  '{"spec":{"template":{"metadata":{"annotations":{"restart-time":"'$(date +%s)'"}}}}}'

Kubernetes Lifecycle Hooks#

Kubernetes provides preStop hooks to add a delay before SIGTERM — useful when the load balancer takes time to deregister the pod.

spec:
  containers:
  - name: app
    lifecycle:
      preStop:
        exec:
          # Sleep to allow load balancer to drain connections
          # before SIGTERM is sent
          command: ["/bin/sleep", "5"]
    terminationGracePeriodSeconds: 60

Conclusion#

Graceful shutdown requires: catching SIGTERM, stopping acceptance of new requests, completing in-flight work, then exiting. In containers, ensure your process is PID 1 or use an init wrapper. The Kubernetes preStop hook gives you a window to deregister from load balancers before the signal arrives.