CI/CD Pipeline Failure Triage Playbook: Step-by-Step Runbook + Provider Templates

Overview

Pipeline failures usually fall into a small set of categories: a change in application code, a dependency or toolchain change, an unhealthy agent, a network or service interruption, a stale cache, a missing artifact, or an authentication and permission problem. The visible error is often downstream from the original cause. For example, a package installation timeout may later appear as a failed test, and a missing environment variable may surface as an application startup error.

Use this order when you need to debug CI pipeline failures quickly:

1. Identify the first failed job and the first meaningful error.
2. Determine whether the failure is reproducible on the same commit.
3. Compare the failing run with the last successful run.
4. Classify the failure as code, environment, infrastructure, dependency, or access related.
5. Apply the smallest reversible fix, then rerun only the necessary scope.
6. Record the cause, evidence, fix, and follow-up action in the runbook or incident record.

Do not begin by deleting every cache, increasing every timeout, or adding retries to all steps. Those actions can hide the cause and make future failures harder to diagnose. Capture logs and identify the failing boundary first.

What to track

1. Failure location and first error

Record the workflow, pipeline, stage, job, step, commit, branch, runner or agent, and execution time. Look for the first non-wrapper error rather than the final message such as “job failed” or “command exited with code 1.” Save a short log excerpt that includes the command and its output.

A useful failure record looks like this:

Repository: example-service
Commit: abc1234
Workflow/job: test / unit-tests
Runner: linux container
First error: dependency download timed out
Last successful run: def5678
Reproduction: fails on rerun; passes locally with a clean dependency directory
Suspected class: network or package mirror

2. Common root causes and telltale evidence

3. Provider-specific evidence

Provider interfaces differ, but the evidence to collect is similar. In GitHub Actions, open the failed run and expand the first failing step. If the repository uses the GitHub CLI, list recent runs and inspect a failed run with commands such as:

gh run list --limit 10
 gh run view RUN_ID --log-failed

For a GitHub Actions failing build, check the event that started the run, the selected runner, matrix parameters, permissions, environment variables, action versions, and cache key. A failure limited to one matrix entry often points to an operating-system, runtime, or version-specific condition.

In GitLab CI, inspect the job trace, commit, runner tags, image, variables, and artifacts. With the GitLab CLI where it is configured for the project, useful starting points include:

glab ci status
 glab ci view
 glab ci trace JOB_ID

For GitLab CI pipeline debug steps, verify that the job is assigned to a compatible runner and that the selected image contains the expected shell, package manager, runtime, and certificates. If a job is pending rather than failing, investigate runner availability and tags before changing the script.

In Jenkins, open the build’s Console Output and identify the first failed shell, stage, or plugin action. On an agent, these commands can establish basic conditions:

df -h
 free -m
 uname -a
 docker info

The exact Jenkins log location depends on the installation and logging configuration, so use the controller or agent configuration rather than assuming a fixed path. For a Jenkins pipeline error checklist, review recent plugin, JDK, agent image, credential-binding, and workspace changes. A workspace left by an interrupted build can also affect later jobs; clean it deliberately and record that action.

In CircleCI, inspect the job step where the command failed, then compare the executor, resource class, environment variables, contexts, and cache keys. If the CircleCI CLI is configured, begin with:

circleci config validate
 circleci local execute --job JOB_NAME

Local execution is useful for configuration and command reproduction, but it may not reproduce hosted networking, contexts, machine images, or service dependencies. Treat a local pass as evidence, not proof that the hosted job is healthy.

Cadence and checkpoints

Pipeline triage is reactive, but pipeline quality improves when recurring variables are reviewed on a schedule. Use a lightweight monthly review for active repositories and a deeper quarterly review for shared runners, deployment workflows, and organization-wide templates.

At every failure

• Capture the first meaningful error and the failing commit.
• Mark whether the failure reproduced, passed on retry, or passed after a clean environment.
• Record the affected provider, runner, image, dependency source, and deployment target.
• Separate the immediate mitigation from the permanent corrective action.

Monthly checkpoint

• Review repeated failures by job, repository, runner, and error category.
• Inspect jobs that pass only after retries or cache deletion.
• Check dependency, runtime, container image, action, plugin, and orb changes.
• Review average queue time and unusually long build stages.
• Remove obsolete secrets, permissions, branches, and workflow paths where appropriate.

Quarterly checkpoint

• Test the failure and rollback portions of the release runbook.
• Review retry policies so transient failures are retried without multiplying load or delaying clear failures.
• Validate cache keys against operating system, runtime, lockfile, and toolchain inputs.
• Confirm artifact naming, retention, provenance, and promotion rules.
• Reproduce a representative pipeline on a clean runner or image.
• Update provider commands and log locations after material platform or plugin changes.

How to interpret changes

A rising failure count does not automatically mean the application is becoming less reliable. First divide failures into deterministic and intermittent groups. Deterministic failures reproduce on the same commit and environment; intermittent failures vary by runner, timing, network, concurrency, or external service.

Use these patterns as diagnostic signals:

• One commit fails across every runner: inspect the code, lockfile, build configuration, or test change.
• Many unrelated repositories fail together: investigate a shared runner image, package registry, credential, network route, or provider incident.
• Only one runner fails: compare disk, memory, tool versions, clock, workspace state, and image digest with a healthy runner.
• Only scheduled jobs fail: compare event context, permissions, secrets, branch state, and time-dependent data.
• Retries pass frequently: classify the underlying transient condition instead of treating retry success as resolution.
• Failures begin after a cache change: restore a clean path, then make the cache key include every input that affects the output.

Track failure rate alongside queue time, duration, retry rate, and time to recovery. A pipeline that fails less often but takes much longer to run may still be creating delivery risk. Conversely, a short-lived test failure may be less urgent than a deployment job that silently produces or promotes the wrong artifact.

When the issue reaches production, connect the CI evidence to operational evidence. Deployment failures may require checking service health, HTTP 5xx responses, container startup, or telemetry. The HTTP 5xx troubleshooting runbook can support the post-deployment investigation, while the secure deployment checklist helps review release permissions and configuration controls.

When to revisit

Revisit this playbook monthly while a repository or provider is changing rapidly, and at least quarterly for stable pipelines. Update it sooner when a provider changes its CLI, runner behavior, log format, authentication model, artifact handling, or cache implementation. The same applies when a widely used action, plugin, container image, package manager, or build tool changes behavior.

Use a simple change log with four fields: date, observed pattern, runbook change, and owner. Add a provider template when a team adopts a new CI system. Remove commands that no longer match the installed CLI version, and label commands that require optional tooling or local credentials.

After the next failure, follow this one-page checklist:

1. Pause unnecessary reruns and preserve the failed run.
2. Write down the commit, job, stage, runner, event, and first meaningful error.
3. Compare the last success with the current run.
4. Classify the issue: code, dependency, agent, network, cache, artifact, or permission.
5. Test one hypothesis with the smallest safe change.
6. Rerun the affected job or pipeline scope and verify the result.
7. Document the root cause and add a preventive check, alert, or test if needed.
8. Schedule the next monthly or quarterly review if the failure exposed a recurring risk.

A maintained triage record turns emergency debugging into institutional knowledge. Keep provider-specific details close to the general workflow, review recurring patterns rather than isolated incidents, and update the runbook whenever the pipeline’s inputs or execution environment materially change.