Use this HTTP 5xx troubleshooting runbook to move from an alert to a verified fix. It covers 500, 502, 503, and 504 errors across load balancers, gateways, application services, containers, and dependencies, while giving your team a repeatable set of variables to track during and after every incident.
Overview
HTTP 5xx responses indicate that a server-side component could not successfully fulfill a request. The status code is useful, but it rarely identifies the failing layer by itself. A 502 may come from an unavailable upstream service, a 503 may reflect capacity or readiness problems, and a 504 may indicate that a request exceeded a timeout somewhere in the chain. A 500 often points to an unhandled application exception, although infrastructure and dependency failures can surface through the application as well.
The first objective is to establish scope: which users, routes, regions, versions, and dependencies are affected? The second is to identify the first component that failed, rather than stopping at the component that reported the error. A gateway may return 502 while the application behind it is crashing. An application may return 500 because a database connection pool is exhausted. Treat the visible status code as an entry point for investigation, not as a complete diagnosis.
During an incident, preserve a timeline containing the alert time, first observed error, deployment or configuration changes, mitigation attempts, and recovery. Assign one person to coordinate, another to investigate logs and traces, and another to communicate status when the incident is large enough to require separate roles. Keep hypotheses explicit and testable: for example, “5xx errors began immediately after version X reached production and are limited to route Y.”
What to track
Track the variables that help distinguish application faults from routing, capacity, timeout, and dependency problems. At minimum, build dashboards or saved queries for the following dimensions:
- Error rate: Count 5xx responses and calculate their proportion of total requests. Break the view down by service, route, method, status code, region, and deployment version.
- Latency: Compare successful and failed request duration. A growing latency tail before 504 responses often suggests saturation, queueing, or a timeout mismatch.
- Request volume: A sudden traffic increase can expose capacity limits, while a normal request rate with rising errors points toward a change, dependency issue, or internal failure.
- Resource pressure: Review CPU, memory, disk, network, process, connection-pool, and thread or worker utilization. For Kubernetes workloads, also check restarts, readiness status, pending pods, and throttling.
- Dependency health: Track database, cache, queue, identity provider, object storage, and external API errors, latency, timeouts, and connection failures.
- Release and configuration markers: Annotate dashboards with deployments, feature-flag changes, infrastructure updates, certificate changes, and scheduled jobs.
- Trace context: Capture a request ID or trace ID from the alerting path. Distributed traces can show whether the failure began at the edge, gateway, application, or downstream service.
Use structured logs rather than searching only for the text “500.” Filter by timestamp, service, route, status, request ID, exception type, and upstream response. A useful log query should answer: what failed, on which instance, for which request, and after how long? Redact secrets and personal data before sharing examples in incident channels.
For Kubernetes-related failures, inspect workload events and container logs alongside application telemetry. A pod that is repeatedly restarting may explain a 503 from a service, while a healthy pod with failed readiness checks may indicate a configuration or dependency problem. The Kubernetes CrashLoopBackOff Troubleshooting Runbook provides a focused follow-up for restart loops. Resource symptoms should be compared with the workload’s configured requests and limits; see the Kubernetes Requests and Limits guide for that review.
Cadence and checkpoints
Use the runbook at three different speeds. During an incident, check high-signal data in short, consistent intervals rather than changing several variables at once. Start with the last 15 to 30 minutes, then compare it with a longer baseline such as the previous hour or a normal operating period. Record each observation and action so responders do not repeat failed mitigations.
After mitigation, keep monitoring long enough to confirm that the error rate, latency, saturation, and dependency signals have returned to an acceptable range. Verify more than the single endpoint that triggered the alert. Test representative read and write paths, authentication flows, background processing, and critical integrations where applicable.
Review recurring 5xx data weekly or monthly, depending on service criticality. Look for routes that repeatedly fail, errors clustered around deployments, and dependencies that create long or unreliable request paths. On a quarterly reliability review, validate alert thresholds, dashboard ownership, runbook links, escalation contacts, timeout settings, and the availability of trace and request identifiers. If a deployment strategy is changing, compare the error behavior of rolling, blue-green, or canary releases using the guidance in Blue-Green vs Canary vs Rolling Deployments.
How to interpret changes
| Symptom | Likely investigation path |
|---|---|
| 500 errors increase in one route after a release | Compare the new version’s logs, stack traces, feature flags, input validation, and dependency calls with the previous version. Consider rollback or disabling the changed feature if the evidence supports it. |
| 502 errors appear at the gateway | Check upstream reachability, service discovery, TLS or protocol negotiation, connection resets, container availability, and gateway-to-service timeout settings. |
| 503 errors affect a service with unhealthy instances | Inspect readiness checks, pod or process restarts, capacity, deployment state, and whether the service has enough healthy endpoints to receive traffic. |
| 504 errors rise while latency increases | Trace the slow request across each hop. Check queues, database queries, connection pools, downstream latency, and mismatched timeout values. Raising a timeout may hide saturation rather than fix it. |
| Errors occur across several unrelated services | Investigate shared infrastructure, networking, identity, DNS, certificates, regional capacity, and common dependencies before changing individual applications. |
Do not infer causation from timing alone. A deployment that overlaps with an incident is an important lead, but confirm it through version-level error rates, traces, logs, or a controlled rollback. Likewise, a drop in 5xx responses after a restart may be temporary if the underlying leak, queue buildup, or dependency failure remains.
When handing an issue to another team, provide a compact evidence packet: affected service and routes, start time and timezone, status-code breakdown, request volume, example request or trace IDs, relevant log excerpts, recent changes, suspected dependency, actions already taken, and the specific decision needed. This is more useful than forwarding an unfiltered dashboard or saying that “the API is down.”
When to revisit
Revisit this runbook after every material 5xx incident, deployment regression, dependency outage, gateway change, or infrastructure migration. Also schedule a monthly or quarterly review even when no major incident occurred. Recurring low-volume errors can reveal a reliability problem before they become a visible outage.
During each review, update the service map, ownership details, escalation paths, dashboard links, alert thresholds, log fields, and example queries. Confirm that the runbook still matches the current gateway, cloud load balancer, framework, container platform, and observability tools. If traces are incomplete, use the OpenTelemetry Implementation Guide to identify instrumentation and collection gaps.
Finish every review with three practical outputs: one change that reduces the chance of recurrence, one improvement that shortens detection or diagnosis, and one verification step that proves the fix works. Link the resulting action items to the incident record and assign owners and due dates. For release-related failures, pair this runbook with the Secure Deployment Checklist and your team’s deployment verification process. A runbook is valuable only when its signals, contacts, and commands remain accurate when the next alert arrives.
