Kubernetes CrashLoopBackOff Troubleshooting Runbook: Causes, Commands, and Fixes

Overview

CrashLoopBackOff is a pod status that indicates Kubernetes has repeatedly started a container, observed it exit or fail, and is waiting progressively longer before trying again. The status describes the restart pattern, not the underlying cause. A failing application, invalid configuration, unsuccessful health probe, missing dependency, resource limit, or image-related problem can all produce a Kubernetes pod restart loop.

Begin with the pod rather than changing the deployment immediately. Capture its namespace, node, container names, restart count, recent events, termination reason, and logs. This preserves evidence that may disappear when a pod is replaced. If the workload is managed by a Deployment, StatefulSet, or another controller, use the controller to understand the desired configuration, but use the individual pod to investigate the current failure.

Set an incident scope before making changes. Confirm whether one replica is affected or whether several workloads on the same node, namespace, or cluster are failing. A single pod often points to application or configuration behavior. A broader pattern can indicate a shared secret, configuration source, dependency, node, admission rule, or platform issue.

kubectl get pods -n <namespace> -o wide
kubectl describe pod <pod-name> -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp
kubectl get deployment <deployment-name> -n <namespace> -o yaml

What to track

Pod state, restart count, and termination details

Record the pod's current state, the number of restarts, the age of the pod, and the container's last termination information. A nonzero exit code can help distinguish an application failure from an external termination. Look for fields such as reason, exitCode, signal, and whether the container was terminated by an out-of-memory condition.

kubectl get pod <pod-name> -n <namespace> \
  -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,RESTARTS:.status.containerStatuses[*].restartCount,REASON:.status.containerStatuses[*].lastState.terminated.reason,EXIT:.status.containerStatuses[*].lastState.terminated.exitCode
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.status.containerStatuses[*].lastState.terminated}'

Current and previous container logs

Always check both the current log and the previous container instance. In a restart loop, the previous log often contains the startup error that caused the last exit.

kubectl logs <pod-name> -n <namespace> -c <container-name>
kubectl logs <pod-name> -n <namespace> -c <container-name> --previous
kubectl logs <pod-name> -n <namespace> -c <container-name> --previous --timestamps

If logs are empty, the process may exit before it initializes logging, write logs to a file inside the container, or fail before the selected container starts. Check the container command, entrypoint, mounted files, and pod events rather than assuming the absence of logs means the application is healthy.

Command, arguments, image, and configuration

Compare the running pod specification with the expected release configuration. Inspect the image reference, command, arguments, environment variables, ConfigMaps, Secrets, volume mounts, and working directory. Common configuration failures include a missing key, an incorrectly named environment variable, an invalid file path, a malformed configuration file, or a command that does not exist in the image.

kubectl get pod <pod-name> -n <namespace> -o yaml
kubectl get configmap <configmap-name> -n <namespace> -o yaml
kubectl get secret <secret-name> -n <namespace> -o yaml
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[*].image}'

Do not paste Secret values into tickets or chat. Verify that the expected key exists and that the consuming application receives the right name and format without exposing the value.

Health probes and resources

Review liveness, readiness, and startup probes separately. A readiness failure normally removes a pod from service, while a liveness failure can cause the kubelet to restart the container. A startup probe can give a slow-starting process time to initialize before other probe behavior is applied. Check the probe path, port, scheme, command, initial delay, timeout, period, and failure threshold against the application's actual startup behavior.

kubectl describe pod <pod-name> -n <namespace>
kubectl top pod <pod-name> -n <namespace>

Resource symptoms also matter. An out-of-memory termination points toward memory limits, workload behavior, or both. CPU pressure may produce slow startup and probe failures rather than a direct crash. Review requests and limits alongside recent application changes and node conditions. Avoid raising limits blindly: first determine whether the process has a leak, an unexpectedly large workload, or a configuration that increases startup demand.

Dependencies and scheduling context

Some applications exit when they cannot reach a database, queue, configuration service, or external endpoint. Test the dependency from an appropriate diagnostic pod when possible, and check service endpoints, DNS behavior, network policies, credentials, and TLS configuration. Also inspect the node and namespace context when more than one pod is affected.

kubectl get nodes
kubectl describe node <node-name>
kubectl get svc,endpoints,endpointslices -n <namespace>
kubectl get networkpolicy -n <namespace>

Cadence and checkpoints

Use this runbook during an incident, then turn its observations into recurring checks. At minimum, review CrashLoopBackOff events after every deployment, configuration change, image update, or node maintenance activity. A lightweight weekly or biweekly review can identify workloads with rising restart counts before users report an outage.

On a monthly or quarterly cadence, review these checkpoints:

• Workloads with repeated restarts, even if they eventually become ready.
• Deployments whose probes have been edited or whose startup time has changed.
• Images, base images, entrypoints, and runtime versions that changed recently.
• Resource requests and limits compared with observed usage and workload growth.
• Configuration and Secret keys that are required at startup.
• Runbook commands, dashboards, alert links, and ownership information.

Keep a small incident record containing the first observed time, affected workload, restart count, last termination reason, relevant log excerpt, change made, and verification result. This creates a useful history without retaining sensitive configuration or credentials.

How to interpret changes

A restart count that increases quickly after a release, with an application error in --previous logs, usually warrants comparing the release image, command, arguments, and configuration with the last known-good version. If the process exits cleanly but immediately, check whether the command is a short-lived job being run under a controller that expects a continuously running process.

Events mentioning failed probes shift attention to probe configuration and application startup timing. Confirm the endpoint manually inside the pod or through a suitable diagnostic method, then determine whether the probe is checking the correct port and path. Do not weaken a probe simply to make the status green; align it with a meaningful health signal.

An out-of-memory termination shifts attention to memory behavior and limits. Compare the affected pod with a healthy replica, inspect recent traffic or input-size changes, and determine whether the limit is intentional. If several pods fail after a shared configuration change, investigate the common dependency before editing each workload separately.

When evidence is incomplete, make one reversible change at a time and observe the result. Record what changed and what result you expected. If the workload is managed declaratively, fix the source manifest, Helm values, or deployment pipeline rather than relying on an ad hoc live edit that the next rollout may overwrite.

After remediation, verify more than the pod status:

kubectl rollout status deployment/<deployment-name> -n <namespace>
kubectl get pods -n <namespace> -w
kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> -c <container-name> --tail=100

Confirm that the desired number of replicas is available, restart counts stop increasing, readiness is stable, recent events are not reporting new failures, and application-level checks succeed. For broader visibility, connect pod events and restart alerts with your observability tooling; an OpenTelemetry implementation guide can help structure application telemetry for cloud-native systems.

When to revisit

Revisit this runbook whenever a workload enters CrashLoopBackOff, after a failed rollout, or when restart counts rise without an obvious outage. Schedule a monthly or quarterly maintenance review for the commands, ownership contacts, dashboards, probe assumptions, and known failure modes. Update it immediately when your cluster version, deployment method, image layout, secret-management approach, or observability stack changes.

For the next incident, follow this short sequence:

1. Identify the namespace, controller, pod, container, node, and scope of impact.
2. Capture describe output, recent events, current logs, and --previous logs.
3. Check termination details, command and arguments, image, configuration, probes, resources, and dependencies.
4. Compare the affected workload with the last known-good revision.
5. Apply the smallest declarative, reversible fix and document the reason.
6. Verify rollout, readiness, restart stability, events, and application behavior.
7. Convert any recurring finding into a preventive test, alert, checklist item, or deployment guardrail.

Keeping this sequence current turns Kubernetes troubleshooting from a collection of guesses into a repeatable incident response practice.