CI/CD and What a "Run" Really Is

41.0 What this chapter gives you#

1. You will be able to explain what continuous integration is as a practice, and say clearly why it is not the same thing as a tool that runs tests.
2. You will be able to state the difference between continuous delivery and continuous deployment in one sentence each, without muddling them.
3. You will be able to describe what a runner is, what it has installed when it starts, and why a command that works on your laptop fails there.
4. You will be able to read a complete GitHub Actions workflow file line by line, and translate the same pipeline into GitLab CI.
5. You will be able to name every common trigger, and explain why pull_request_target is a security trap while pull_request is not.
6. You will be able to explain the single most important idea here: a run is attached to one commit hash, never to a branch name.
7. You will be able to verify continuous integration by run ID against an exact head hash, using real commands, and list six ways a branch-level green tick can lie to you.
8. You will be able to explain what a merge queue is, what problem it solves, and what every mergeStateStatus value means and implies.
9. You will be able to lay out a sensible pipeline in order, choose a deployment strategy, and handle secrets without leaking them.
10. You will be able to read a failed run methodically instead of guessing.

41.1 What continuous integration means and the problem it solves#

PLAIN41.1.1 in simple words#

1. Several people work on the same program at the same time.
2. Each person has their own copy, and each changes different files.
3. Sooner or later all those changes must be joined into one program.
4. Joining them is called integration. It is the moment of truth.
5. If everyone waits weeks before joining, the changes have drifted far apart and joining them is slow, painful and full of surprises.
6. Old teams had a word for this: integration hell. It could take days.
7. Continuous integration is the opposite habit. Everyone joins their work into the shared copy very often, at least once a day.
8. Small joins are easy. Big joins are hard. So you make the joins small.
9. But joining often is only safe if you also check often.
10. So every time anyone joins, a machine automatically builds the program and runs the tests, and tells you within minutes if it broke.
11. That habit, plus that automatic check, is continuous integration.
12. Notice that the habit comes first. The machine only enforces the habit.

PLAIN41.1.2 a picture in your head#

1. Five students must write one report together, one chapter each.
2. Plan A: everyone writes alone for three weeks, then meets the night before to glue the chapters together.
3. That night is chaos. Two people defined the same term differently. Three people numbered their figures from one. The introduction promises a section that nobody wrote.
4. Plan B: every evening, each student pastes what they have into the one shared document, and one person reads the whole thing top to bottom.
5. Under plan B a clash is found the same evening it is created, while both students still remember what they meant.
6. Now replace “one person reads it top to bottom” with a machine that reads it every single time anyone pastes anything, in two minutes.
7. That machine is the continuous integration server. The evening paste is the practice.

Where this comparison breaks: a report can be a bit inconsistent and still be readable. A program cannot. One mismatched function name and nothing runs at all. Also, a human reader gets bored and misses things on the tenth read. A test suite does not get bored, but it only checks what somebody wrote a test for, which is always less than everything.

PLAIN41.1.3 a worked example#

1. Take a team of five people. Each writes about 200 changed lines a day.
2. Under plan A they integrate once every three weeks, so 15 working days.
3. Lines waiting to be joined: 5 people x 200 lines x 15 days = 15,000 lines.
4. Under plan B they integrate once a day: 5 x 200 x 1 = 1,000 lines.
5. The pain of a merge does not grow in a straight line with size. Two changes can clash, and the number of possible clashing pairs grows roughly with the square of the number of changes.
6. So a batch fifteen times bigger can feel far worse than fifteen times worse.

7. The numbers in that table are illustrative, not measured. The shape is what matters: smaller batches, cheaper joins.

PLAIN41.1.4 what is really happening inside#

1. You finish a small piece of work and commit it locally.
2. You push it to the shared server. The push is just data transfer.
3. The server notices that a branch moved and fires an event.
4. That event is delivered to whatever is listening. On a hosted service it goes to an internal queue. For an external tool it goes out as a webhook, which is a small message posted to a web address you registered.
5. The listener picks a free machine, or starts a fresh one.
6. That machine downloads the exact commit that was pushed. Not the branch. The commit.
7. It installs the tools the project needs, builds the program, and runs the tests.
8. It records success or failure, and posts that result back, attached to the same commit hash it downloaded.
9. The web page you look at then draws a tick or a cross next to that commit.
10. The whole loop is usually two to twenty minutes. Under ten minutes is the target most teams aim at, because longer than that and people stop waiting for the answer and start guessing.

TECHNICAL41.1.5 the engineer’s version#

1. The phrase “continuous integration” was coined by Grady Booch in his 1991 book Object-Oriented Design With Applications, where it described the habit of integrating incrementally rather than in a big bang.
2. Extreme Programming, developed by Kent Beck on the Chrysler C3 project from 1996 and set out in Extreme Programming Explained in 1999, made it one of its named practices, with the rule that the team integrates and builds many times a day.
3. Martin Fowler and Matthew Foemmel published the article Continuous Integration in 2000, revised by Fowler in 2006 and again in 2024. That article is where most of the working definition comes from.
4. CruiseControl, written at ThoughtWorks and first released on 30 March 2001, was the first widely used continuous integration server. It polled the version control system on a timer, because webhooks did not exist yet.
5. Hudson was started by Kohsuke Kawaguchi at Sun Microsystems in 2004 and first released in 2005. After a dispute with Oracle, a rename vote was called on 11 January 2011 and approved on 29 January 2011. The renamed project is Jenkins.

6. Travis CI launched in 2011 and popularized the idea of a build configuration file living inside the repository. CircleCI launched the same year. GitLab CI began as a separate application in 2012 and was folded into GitLab itself in version 8.0 in September 2015.
7. GitHub Actions was announced as a general automation product in October
   2018. Its continuous integration capability was announced in beta on 8 August 2019 and became generally available on 13 November 2019.
8. The practice, in its strict form, has three testable properties: everyone merges to the mainline at least daily, every merge triggers an automated build and test, and a broken mainline is fixed immediately and takes priority over new work.
9. Say this plainly: a team running Jenkins on a branch that merges to main once a quarter is not doing continuous integration. It owns a continuous integration tool. The practice and the tool are different things.
10. Observing tools: gh run list, gh run view, git log --oneline --since='1 day ago' origin/main to see integration frequency, and any dashboard that plots merges to trunk per day.

WORDS41.1.6 remember these#

1. Integration — joining everyone’s work into one copy — merging feature branches into the mainline and producing a buildable tree.
2. Integration hell — the painful big join at the end — the superlinear cost of merging long-lived divergent branches.
3. Continuous integration — join often, check automatically every time — the practice of at-least-daily mainline merges verified by an automated build.
4. Webhook — a message the server sends when something happens — an HTTP POST to a registered endpoint carrying a JSON event payload.
5. Mainline or trunk — the one branch everybody joins into — usually main, the integration target for all short-lived branches.
6. Build — turning source into a runnable thing — compile, link, package, including dependency resolution.

41.2 Continuous delivery and continuous deployment#

PLAIN41.2.1 in simple words#

1. Continuous integration answers one question: does the joined code work.
2. There is a second question: could we give it to users right now.
3. Continuous delivery means the answer is always yes. Every change that passes the checks is packaged, tested in a realistic place, and left in a state where one button press would put it in front of users.
4. A human still decides when to press the button.
5. Continuous deployment goes one step further. There is no button. If the checks pass, the change goes live automatically.
6. So the difference is exactly one thing: whether a human approves the last step.
7. Continuous delivery is a capability. Continuous deployment is a policy you can only adopt once you have that capability.
8. Both are abbreviated CD, which is why people confuse them constantly.
9. If somebody says “we do CI/CD”, you have learned almost nothing. Ask which D they mean, and ask what the last manual step is.

PLAIN41.2.2 a picture in your head#

1. Think of a bakery that supplies a shop.
2. Continuous integration is the taste test after mixing. Is the dough right.
3. Continuous delivery is: the bread is baked, wrapped, labelled, loaded onto the van, and the van is parked outside with the engine running.
4. Any moment the manager says go, it leaves. Nothing more to prepare.
5. Continuous deployment is: the van leaves the moment the loading is finished. No manager, no decision.
6. Notice that a bakery can be excellent at delivery and still choose to send the van only twice a day, for business reasons.
7. That is a real and respectable choice. The point of delivery is that the delay is a decision, not a limitation.

Where this comparison breaks: bread cannot be un-delivered. Software can be rolled back, sometimes. Also the van either goes or does not, whereas software can go to one percent of users first, which has no bakery equivalent.

PLAIN41.2.3 a worked example#

1. Here is one change moving through a delivery pipeline, with times.

2. Total machine time is 19 minutes. Under continuous delivery the change sits at “ready to release” until a person approves it.
3. Under continuous deployment, the row “Ready to release” is deleted and the change is live 19 minutes after the commit.
4. The test suites did not change. The pipeline did not change. Only the gate changed.

PLAIN41.2.4 what is really happening inside#

1. The pipeline is a chain of stages. Each stage takes the output of the one before, not a fresh copy of the source.
2. That is the key rule: you build the artifact once, and the same bytes are what gets tested and what gets deployed.
3. If you rebuild before deploying, you have tested something other than what you shipped, and the test result no longer applies.
4. Early stages are fast and cheap and run on everything.
5. Later stages are slow and expensive and only run on things that passed.
6. Each stage either passes the artifact forward or stops it dead.
7. The pipeline itself records which commit each artifact came from, so you can always answer “what is running in production, exactly”.
8. Continuous deployment simply removes the human gate from the last hop, and compensates with automatic checks after the deploy: health checks, error rate alarms, and automatic rollback.

TECHNICAL41.2.5 the engineer’s version#

1. The deployment pipeline as a named pattern comes from Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation by Jez Humble and David Farley, published in 2010.
2. Its core rules are: build binaries once, deploy the same way to every environment, smoke-test every deployment, and stop the line if any stage fails.
3. Continuous delivery: every change that passes automated checks is a release candidate that could be deployed on demand. Continuous deployment: every such change is deployed without human intervention.
4. The DORA research programme, published in the State of DevOps reports from 2014 and in the 2018 book Accelerate by Nicole Forsgren, Jez Humble and Gene Kim, measures four things.

5. The consistent finding across those reports is that speed and stability move together rather than trading off. That is a statistical association in survey data, not a controlled experiment. Treat it as strong evidence, not proof.
6. There is genuine expert disagreement about whether continuous deployment is right for every domain. Web services with cheap rollback lean yes. Firmware, medical devices and payment cores lean no, because the cost of a bad release is not symmetric with the cost of a slow one.
7. The honest version: most organizations that say “we do CD” mean continuous delivery to a staging environment and manual promotion to production. That is a fine place to be. It is just not continuous deployment.

WORDS41.2.6 remember these#

1. Continuous delivery — always ready to ship — every passing change is a deployable release candidate, promoted on demand.
2. Continuous deployment — ships by itself — every passing change is deployed to production without manual approval.
3. Deployment pipeline — the chain of stages a change walks — an automated implementation of the path from commit to release, with binary reuse.
4. Release candidate — a version that could go live — an immutable artifact that has passed all preceding pipeline stages.
5. Promotion — moving an artifact to the next environment — deploying the same artifact to a higher environment without rebuilding.
6. Lead time for changes — how long from writing to live — the DORA metric measuring commit timestamp to production deploy timestamp.

41.3 What a runner actually is#

PLAIN41.3.1 in simple words#

1. Something has to actually execute your build. That something is a computer.
2. In continuous integration that computer is called a runner, or an agent, or an executor, depending on the product.
3. It can be a whole machine, a virtual machine, or a container. From your side it does not matter much: it is a place with a filesystem and a shell.
4. When work arrives, the runner downloads your code at one exact commit and runs your steps one after another, in order, in a shell.
5. It reports the exit code of each step. Zero means success. Anything else means failure, and by default the job stops there.
6. Two kinds exist. Hosted runners are supplied by the service, created fresh for your job, and destroyed afterwards. Self-hosted runners are machines you own, running an agent program that asks for work.
7. The most important fact about a fresh hosted runner: it contains only what the image ships with. Nothing of yours. No files, no settings, no tools you installed last year and forgot about.
8. That is exactly why builds fail on the runner and pass on your laptop.

PLAIN41.3.2 a picture in your head#

1. Imagine a hotel kitchen you can rent by the minute.
2. It is spotless. Every surface wiped. Standard ovens, standard knives.
3. Whatever you want to cook, you must bring the recipe and the ingredients, or ask the hotel for them at the door.
4. Your own kitchen at home has a jar of your grandmother’s spice mix on the top shelf. You use it without thinking. It is not in your recipe.
5. In the hotel kitchen, the dish comes out wrong, and you cannot see why, because the missing thing was never written down anywhere.
6. When your work is finished, the hotel demolishes the kitchen and builds a new one for the next customer. Nothing you left behind survives.

Where this comparison breaks: a real kitchen is never truly identical twice. Runner images are updated every week or two, so tool versions drift under you. And a self-hosted runner is more like your own kitchen: it does keep the spice jar, which is convenient right up until it becomes the bug.

PLAIN41.3.3 a worked example#

1. This step works perfectly on the reader’s macOS laptop and fails on a Linux runner.

python script.py --config Config.json

2. On macOS the default filesystem is APFS, set up case-insensitive. The file on disk is config.json, and Config.json finds it anyway.
3. On the Linux runner the filesystem is case-sensitive. Config.json does not exist. The script exits non-zero. The job fails.
4. The error message is FileNotFoundError, which sounds like the file was not committed, so the first hour is spent looking in the wrong place.
5. Here are the usual suspects, all of the same shape.

PLAIN41.3.4 what is really happening inside#

1. A job goes through the same seven phases every time. Learn them and you can place any failure.

 1 QUEUE     job waits for a free runner
 2 ALLOCATE  a runner is chosen or a VM is created
 3 PROVISION boot image, start agent, set environment
 4 CHECKOUT  fetch the repository at one exact commit
 5 RESTORE   pull caches: packages, compiler output
 6 STEPS     run each step in order, stop on failure
 7 UPLOAD    save artifacts and logs
 8 CLEAN     destroy the machine or wipe the workspace

2. Phase 1 costs nothing but wall-clock time. If jobs sit here, you have hit a concurrency limit, not a bug.
3. Phase 3 is where the operating system image is decided. That image is a product of the service and it changes.
4. Phase 4 usually does a shallow fetch of depth one: only the one commit, not the whole history, because history is large and rarely needed.
5. Phase 5 restores a cache keyed on something like the hash of your lock file. A cache miss is slow but correct. A wrong cache hit is fast and wrong, which is worse.
6. Phase 6 runs each step as a separate shell invocation. Variables set with plain export in one step do not survive to the next step, because the shell exited.
7. Phase 8 is why “it left a file behind last time” is never an explanation on a hosted runner. There is no last time.

TECHNICAL41.3.5 the engineer’s version#

1. GitHub-hosted runners are ephemeral virtual machines in Microsoft Azure, with a fresh instance per job, running the open-source actions/runner agent. Figures below were taken from GitHub documentation in August 2026 and change over time.

2. For private repositories the standard Linux and Windows runners are smaller, at 2 vCPU and 8 GB RAM as of August 2026. Larger runners are a paid option.
3. Hard limits for GitHub Actions, as documented in August 2026: a job may run up to 6 hours on a hosted runner and up to 5 days on a self-hosted runner; a workflow run is cancelled after 35 days; a matrix can generate at most 256 jobs per run; a self-hosted job may sit queued for 24 hours.
4. Concurrent job limits by plan: 20 on Free, 40 on Pro, 60 on Team and 500 on Enterprise. The GITHUB_TOKEN is rate limited to 1,000 API requests per hour per repository.
5. GitLab uses a separate gitlab-runner binary with pluggable executors: docker, shell, kubernetes, docker-machine and others. Jenkins calls the same idea an agent, historically a “slave”, connected by JNLP or SSH.
6. Self-hosted runners on a public repository are a documented security hazard: a fork can propose a workflow change that executes on your hardware. GitHub documents this and recommends self-hosted runners only for private repositories, or with ephemeral, isolated, single-use runners.
7. Runner images are built in the open. The actions/runner-images repository publishes the exact contents of each image and a changelog per release, which is the correct place to check whether a tool version moved under you.
8. Observing tools: runs-on in the workflow file, the “Set up job” log group which prints the image version and the resolved runner name, uname -a, df -h, nproc and env | sort as diagnostic steps.

WORDS41.3.6 remember these#

1. Runner — the machine that executes your build — an ephemeral or persistent host running an agent that claims and executes jobs.
2. Hosted runner — supplied and destroyed by the service — a per-job ephemeral VM from a published image, with no state between jobs.
3. Self-hosted runner — your own machine doing the work — a long-lived or ephemeral host running the agent, with persistent state unless wiped.
4. Job — one unit of work on one runner — a named collection of steps sharing a workspace, filesystem and environment.
5. Step — one command or one action inside a job — a separate shell invocation or a container/JavaScript action execution.
6. Workspace — the folder your code is checked out into — the working directory, exposed as GITHUB_WORKSPACE or CI_PROJECT_DIR.
7. Exit code — the number a command returns — 0 for success, non-zero for failure; the runner uses it to decide whether to continue.

41.4 The workflow file#

PLAIN41.4.1 in simple words#

1. Somewhere you must write down what the machine should do. That written description is the workflow file.
2. It is a plain text file in YAML, a format that uses indentation instead of brackets to show what belongs to what.
3. On GitHub it lives in the folder .github/workflows/ inside your repository, and any file there ending in .yml or .yaml is picked up.
4. On GitLab it is a single file called .gitlab-ci.yml at the top of the repository.
5. It sits inside the repository, next to the code, in the same history. That is the important part.
6. Because it is in the history, the workflow that ran for an old commit is the workflow as it existed at that commit. Not today’s version.
7. So changing your tests and changing how they run is one change, one review, one commit. They can never drift apart.

PLAIN41.4.2 a picture in your head#

1. Think of a recipe card taped inside the lid of the box that holds the ingredients.
2. If you change the ingredients, you open the box, so you see the recipe and change it too.
3. Compare with a recipe pinned on a noticeboard in another room. It slowly becomes a description of a dish nobody makes any more.
4. A configuration stored in a web interface on the build server is the noticeboard. A workflow file in the repository is the card inside the lid.
5. And if you open a box from three years ago, the card inside tells you exactly how that batch was made.

Where this comparison breaks: the recipe card is not fully self-contained. It says “use action X at version 4”, and version 4 is fetched fresh at run time from somewhere else. So an old commit can still build differently today. Section 41.13 explains how pinning fixes that.

PLAIN41.4.3 a worked example#

1. Here is a complete, working GitHub Actions workflow. Every line is explained underneath.

name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

permissions:
  contents: read

concurrency:
  group: ci-${{ github.ref }}
  cancel-in-progress: true

env:
  CI: "true"

jobs:
  lint:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "22"
          cache: npm
      - run: npm ci
      - name: Static checks
        run: npm run lint

  test:
    needs: [lint]
    runs-on: ${{ matrix.os }}
    timeout-minutes: 20
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, macos-latest]
        node: ["20", "22"]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node }}
          cache: npm
      - run: npm ci
      - name: Unit tests
        run: npm test
        env:
          TZ: UTC
      - name: Save the report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: junit-${{ matrix.os }}-${{ matrix.node }}
          path: reports/junit.xml

2. name is the label shown in the user interface. Cosmetic only.
3. on lists the events that start a run. Here: a push to main, and any pull request whose target is main.
4. permissions sets what the automatic token may do. contents: read means the job can read the repository and nothing else.
5. concurrency groups runs. Two runs in the same group cannot overlap, and cancel-in-progress: true kills the older one. This stops five pushes in a minute from starting five full pipelines.
6. env at the top sets environment variables for every job.
7. jobs holds one or more jobs. lint and test are job IDs, chosen by you.
8. runs-on picks the runner label. ubuntu-latest means a hosted Linux VM.
9. timeout-minutes kills a job that hangs. Without it a stuck job burns until the six-hour limit.
10. steps is an ordered list. Each item is either uses or run.
11. uses runs a prepackaged action, here actions/checkout at major version 4, which clones the repository at the right commit.
12. with passes named inputs to that action.
13. run executes a shell command on the runner.
14. name on a step is the label in the log. Give every non-obvious step one, because that label is what you will search for when it fails.
15. needs: [lint] makes test wait for lint to succeed. Without needs, jobs run in parallel.
16. strategy.matrix expands one job into several. Two operating systems by two Node versions gives four jobs.
17. fail-fast: false says: if one matrix cell fails, let the others finish. The default, true, cancels the siblings, which is faster but tells you less.
18. if: always() on the upload step means “run this even if a previous step failed”, which is how you get the test report from a failing build.
19. ${{ ... }} is an expression. It is evaluated by the service before the step runs, and substituted as text.

PLAIN41.4.4 what is really happening inside#

1. Here is the same pipeline written for GitLab CI, in .gitlab-ci.yml.

stages: [lint, test]

default:
  image: node:22-bookworm
  cache:
    key:
      files: [package-lock.json]
    paths: [.npm/]

variables:
  CI: "true"

lint:
  stage: lint
  script:
    - npm ci --cache .npm --prefer-offline
    - npm run lint

unit-test:
  stage: test
  needs: [lint]
  parallel:
    matrix:
      - NODE_TAG: ["20", "22"]
  image: node:${NODE_TAG}-bookworm
  script:
    - npm ci --cache .npm --prefer-offline
    - npm test
  artifacts:
    when: always
    reports:
      junit: reports/junit.xml
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

2. The differences are worth naming, because they are differences of model, not of syntax.
3. GitLab has a first-class concept of stages: named phases that run in order, with all jobs in a stage running together. GitHub has no stages, only needs edges between jobs, which form a graph.
4. GitLab jobs run inside a container image by default, set with image. GitHub jobs run directly on the runner VM, and containers are opt-in.
5. GitLab has no uses. Reuse comes from include, extends and YAML anchors, plus CI/CD components. GitHub’s marketplace of actions has no direct GitLab equivalent.
6. GitLab checks out the code for you automatically. GitHub requires an explicit actions/checkout step, and forgetting it is a classic first-day mistake.
7. GitLab’s rules replaces GitHub’s on plus if, combining trigger and condition into one list evaluated top to bottom.
8. Both put the file in the repository, and both therefore get the same benefit: the pipeline definition is versioned, reviewable and bisectable.

TECHNICAL41.4.5 the engineer’s version#

1. GitHub reads every file matching .github/workflows/*.yml and .github/workflows/*.yaml on the branch that the event applies to. This is a documented behaviour, not a convention.
2. For push events the workflow file used is the one at the pushed commit. For pull_request events it is the one at the pull request’s head. For pull_request_target and schedule it is the one on the base or default branch. That asymmetry is the root of section 41.5’s security discussion.
3. Available contexts inside ${{ }} include github, env, vars, secrets, job, steps, runner, strategy, matrix, needs and inputs. Context values are substituted before the shell sees them, which is why interpolating untrusted text directly into a run block is a script injection hole. Pass it through env: instead.
4. Each run block is written to a temporary script and executed with the default shell: bash -e on Linux and macOS, pwsh on Windows. Because -e is set, the first failing command ends the step. Add set -o pipefail if you use pipes, since a pipeline’s exit code is the last command’s.
5. YAML has a real trap here. In YAML 1.1, the bare word on is a boolean true, so strict linters flag on: as the key true. GitHub’s parser special-cases it. Quoting as "on": also works. This is an implementation detail worth knowing when a linter shouts at a file that runs fine.
6. Reuse mechanisms: composite actions (steps bundled in an action.yml), reusable workflows called with uses at job level and triggered by workflow_call, and Docker container actions. Reusable workflows can be nested up to four levels deep.
7. GitLab equivalents: include:local, include:project, include:remote, extends, and CI/CD components introduced in GitLab 16.6 in late 2023.
8. Because the file is versioned, git log -p .github/workflows/ answers “when did this check start being required” and git bisect can walk a pipeline regression the same way it walks a code regression.
9. A practical consequence the reader met directly: a GitHub personal access token needs the workflow scope to push a commit that adds or edits a file under .github/workflows/. Without it the push is refused, even though the same token can push ordinary code. The workflow file is treated as more dangerous than code, because it is.

WORDS41.4.6 remember these#

1. Workflow file — the written plan for the machine — a YAML document defining triggers, jobs and steps, stored in the repository.
2. YAML — a text format that uses indentation — a data serialization language; GitHub Actions and GitLab CI both use it for pipeline definitions.
3. Action — a reusable prepackaged step — a JavaScript, Docker or composite unit referenced by uses: owner/repo@ref.
4. Matrix — one job definition expanded into many — a cross product of variable values generating up to 256 jobs per run.
5. Stage — a named phase in GitLab — an ordered grouping of jobs; GitHub models the same idea as needs dependencies instead.
6. Context — the data available to expressions — named objects such as github, matrix and secrets resolved before the step runs.
7. Script injection — untrusted text becoming code — interpolating attacker controlled values into a run block, mitigated by passing through env.

41.5 Triggers: what starts a run#

PLAIN41.5.1 in simple words#

1. A run never starts on its own. Something has to ask for it.
2. The thing that happens is called an event. The rule in your file that listens for it is called a trigger.
3. The commonest event is a push: you send new commits to the shared server.
4. The next commonest is a pull request: you propose joining one branch into another.
5. A clock can be the event: run every night at two in the morning.
6. A person can be the event: somebody clicks a “Run workflow” button.
7. Another program can be the event: an outside system sends a message asking for a run.
8. Creating a tag, which is a permanent name pinned to one commit, can be the event.
9. Publishing a release, which is a tag plus notes and downloadable files, can be the event.
10. Every event carries facts with it: which repository, which branch, which commit, and who caused it.
11. Your workflow file lists the events it cares about. Every other event is ignored by that file.
12. You can narrow an event further: only on this branch, only when these files changed.
13. Narrowing matters because every run costs minutes and money, and most runs on most changes are wasted work.

PLAIN41.5.2 a picture in your head#

1. Picture a workshop with a row of bells along one wall.
2. Bell one rings when a delivery van drops parts at the door. That is push.
3. Bell two rings when somebody posts a proposed change to the plans. That is pull_request.
4. Bell three is wired to a timer and rings itself at 2 a.m. every night. That is schedule.
5. Bell four has a rope hanging down, and any worker may pull it whenever they like. That is workflow_dispatch.
6. Bell five is a phone line from the factory next door, so they can call you. That is repository_dispatch.
7. Beside each bell hangs a small card saying which deliveries are worth getting up for.
8. One card says “steel only, ignore paint”. That is a path filter.
9. Another says “only deliveries addressed to the main workshop”. That is a branch filter.

Where this comparison breaks: bells in a workshop ring one at a time. Real events overlap. A single push to a branch that already has an open pull request rings two bells and can start two nearly identical runs on nearly the same code. Also a bell carries no information, while an event carries a whole document describing exactly what happened, which your job can read.

PLAIN41.5.3 a worked example#

1. Here is one on: block using most of the common triggers at once.

on:
  push:
    branches:
      - main
      - 'release/**'
    tags:
      - 'v*.*.*'
    paths:
      - 'src/**'
      - 'package-lock.json'
  pull_request:
    types: [opened, synchronize, reopened, ready_for_review]
    branches: [main]
  schedule:
    - cron: '17 2 * * *'
  workflow_dispatch:
    inputs:
      environment:
        description: 'Where to deploy'
        type: choice
        options: [staging, production]
        default: staging
      dry_run:
        type: boolean
        default: true
  repository_dispatch:
    types: [deploy-request]
  release:
    types: [published]

2. push.branches lists branch name patterns. release/** matches release/2026-08 and release/hotfix/db.
3. push.tags lists tag patterns. v*.*.* matches v2.4.1. If both branches and tags are present, a match on either one starts a run.
4. push.paths says: only start if at least one changed file matches. A commit that only edits README.md starts nothing.
5. pull_request.types lists the sub-kinds of the event. synchronize means “the pull request branch got a new push”, which is the one people forget.
6. ready_for_review fires when a draft pull request stops being a draft. It is not in the default list, so you must ask for it.
7. schedule.cron is five fields: minute, hour, day of month, month, day of week. 17 2 * * * is 02:17 every day, in UTC always.
8. The 17 is deliberate. Everyone writes 0 * * * *, so the top of the hour is the busiest queue on the service. Pick an odd minute.
9. workflow_dispatch.inputs builds a small form in the web page and also accepts the same values from the command line.
10. repository_dispatch.types filters on a label chosen by the outside caller, so one repository can serve several kinds of external request.
11. release.types: [published] fires when a release is published, which is the usual moment to build and upload artifacts.

PLAIN41.5.4 what is really happening inside#

1. Now the confusing pair. pull_request and pull_request_target look almost the same and behave very differently.
2. When somebody outside your team proposes a change from their own copy, that code is a stranger’s code. You have not read it yet.
3. A run started by pull_request does check out the stranger’s code, but it runs with almost no power.
4. It gets a read-only token, and it gets none of your secrets.
5. So even if the stranger’s code is hostile, it can make a mess inside its own throwaway machine and steal nothing.
6. That is the safe default. It is also why a first-time contributor often sees “waiting for a maintainer to approve running workflows”.
7. But it is inconvenient. A run with no secrets cannot post a comment, cannot upload to your storage, and cannot add a label.
8. So a second trigger exists: pull_request_target.
9. pull_request_target reads the workflow file from your own base branch, not from the stranger’s branch.
10. Because the file is yours and was reviewed by you, the service trusts it and hands it a write token and your secrets.
11. Here is the trap, and it is a sharp one. The workflow file is yours. The code being proposed is still theirs.
12. If your trusted workflow checks out the pull request’s head and then executes anything from it, you have just handed a stranger your keys.
13. “Executes anything” is wider than it sounds. It includes npm test. It also includes npm install, because a dependency can run an install script.
14. It includes a build tool that reads a config file naming a plugin, a Makefile, a test fixture, or a linter with a custom rule.
15. GitHub’s own security team named this family of bug pwn requests in an article published in August 2020.
16. The rule is one sentence. With pull_request_target, do not check out the head; and if you truly must, do not execute one byte of it.

              pull_request        pull_request_target
              ------------        -------------------
workflow file  PR head branch      base branch
checked out    PR merge result     base branch by default
token          read-only (fork)    read and write
secrets        none (fork)         all of them
safe by        having no power     you not running their code

TECHNICAL41.5.5 the engineer’s version#

1. GitHub Actions documents more than thirty webhook-backed events plus the non-webhook triggers schedule, workflow_dispatch and workflow_call.

2. The default activity types for pull_request are opened, synchronize and reopened. synchronize fires on every new push to the head branch.
3. schedule uses five-field POSIX cron, always evaluated in UTC, never in local time. The shortest documented interval is every 5 minutes.
4. Scheduled runs are queued, not guaranteed punctual. Delays of many minutes at the top of the hour are normal and documented.
5. In a public repository, a scheduled workflow is disabled automatically after 60 days with no repository activity. This is documented behaviour and a very common cause of “our nightly build silently stopped”.
6. workflow_dispatch accepts at most 10 top-level inputs, of types string, choice, boolean and environment, with a total payload limit of 65,535 characters. Start it with gh workflow run or POST /repos/{owner}/{repo}/actions/workflows/{id}/dispatches.
7. repository_dispatch is POST /repos/{owner}/{repo}/dispatches carrying event_type and an optional client_payload JSON object, readable as github.event.client_payload. It only starts workflows whose file lives on the default branch, which surprises people testing it from a branch.
8. Filter keys are branches, branches-ignore, tags, tags-ignore, paths and paths-ignore. You cannot use a key and its -ignore partner for the same event in the same workflow.
9. Glob syntax in these filters: * matches any characters except /, ** matches any characters including /, ? matches one character, + and * act as repeaters, [] is a character class, and a leading ! negates.
10. A documented caveat with real consequences: if a push contains more than 1,000 commits, or GitHub cannot compute the diff because it timed out, path filters are ignored and the workflow runs anyway.
11. The opposite caveat matters more. A path-filtered workflow that does not run produces no check at all, and a required status check that never arrives blocks the pull request forever. Section 41.8 covers the fix.
12. The precise difference between the two pull request triggers:

13. For a fork-originated pull_request, the GITHUB_TOKEN is issued with read permissions only, and no repository, environment or organization secret is exposed. Public repositories can additionally require maintainer approval before a first-time contributor’s workflow runs at all.
14. The dangerous pattern, written out so you can recognize it in review:

# DANGEROUS when combined with pull_request_target
- uses: actions/checkout@v4
  with:
    ref: ${{ github.event.pull_request.head.sha }}
- run: npm ci && npm test    # their code, your secrets

15. The safe pattern when you genuinely need write access on a fork pull request: split into two workflows. The pull_request workflow does the untrusted work with no secrets and uploads an artifact. A second workflow on workflow_run has permissions, downloads that artifact, and treats its contents as data, never as code.
16. workflow_run carries the same hazard in a different coat: it also runs the file from the default branch with a full token, so an artifact produced by an untrusted run must never be executed or expanded into a path you trust.
17. GitLab expresses all of this through rules and the CI_PIPELINE_SOURCE variable, whose values include push, merge_request_event, schedule, web, api, trigger and pipeline. GitLab further distinguishes merge request pipelines from merged results pipelines, which test the combined result rather than the source branch.
18. Observing tools: gh run list --json event,headBranch,headSha, the event name shown at the top of any run page, and gh api repos/OWNER/REPO/hooks to see which webhooks a repository sends to outside systems.

WORDS41.5.6 remember these#

1. Event — the thing that happened — a webhook payload describing a repository or platform change.
2. Trigger — the rule that listens — the on: key naming events, activity types and filters.
3. Activity type — the sub-kind of an event — such as opened or synchronize for pull_request.
4. Path filter — only run when these files changed — paths or paths-ignore globs matched against the push diff.
5. workflow_dispatch — the manual run button — a non-webhook trigger with up to 10 typed inputs, also callable by API.
6. repository_dispatch — an outside system asking for a run — a POST to the repository’s dispatches endpoint with an event_type.
7. Pwn request — a stranger’s code running with your keys — arbitrary code execution obtained by checking out untrusted head under pull_request_target.

41.6 The most important idea: a run belongs to a commit, not a branch#

PLAIN41.6.1 in simple words#

1. This is the idea the whole chapter is built around. Read it twice.
2. A branch name is not a box holding commits. It is a label stuck on exactly one commit.
3. Chapter 38 put it this way: a branch is a moving pointer. Add a commit and the label moves forward to the new one.
4. A test run is not stuck to a label. It is stuck to one exact commit.
5. The machine downloaded one commit, tested those exact bytes, and wrote the answer against that commit’s fingerprint.
6. That fingerprint is the commit hash, a long string of hexadecimal digits computed from the contents.
7. So the sentence “CI passed on main” is short for something much longer.
8. In full it means: “CI passed on whichever commit main happened to point at when that run started”.
9. If main has moved since, the sentence stays true and becomes useless. It is a fact about a commit nobody is looking at any more.
10. The tick you see beside a branch name on a web page is a summary drawn at the moment the page loaded. It is not a promise about now.
11. So never accept a tick beside a branch name as evidence that anything in particular was tested.
12. Get the exact commit hash first. Then ask whether that hash passed.

PLAIN41.6.2 a picture in your head#

1. Think of a hotel with a brass plate on one door reading MANAGER.
2. A health inspector visits, inspects the room behind that plate, and issues a certificate.
3. The certificate does not say MANAGER. It says room 412, and it lists what was in room 412 on that day.
4. Months later the manager moves office. Somebody unscrews the brass plate and screws it onto room 509.
5. Now walk up to the door marked MANAGER and ask: is this room certified?
6. The plate is not the room. The certificate belongs to 412. The door in front of you is 509, and nobody has inspected it.
7. The plate is the branch name. The room number is the commit hash. The certificate is the run.

Where this comparison breaks: a hotel room keeps its contents when the plate moves, so you could at least argue about it. A commit is defined by its contents, so it cannot change at all. Alter one byte and it becomes a different commit with a different hash and no certificate whatsoever. Git is stricter than the hotel, not looser.

PLAIN41.6.3 a worked example#

1. During the reader’s outage, the machine reported origin/main as 0a95cc8.
2. That was a cached remote-tracking value written by the last successful fetch, not a live fact about the server.
3. Now watch the sequence that makes a green tick lie. The clock times are invented; the shape is exactly real.

09:00  main -> 0a95cc8
09:01  run 1187 starts, head_sha = 0a95cc8
09:07  run 1187 finishes, conclusion = success
09:07  branch page shows:  main  [tick]

09:40  a pull request is merged
09:40  main -> 3f1d02c          (the label moved)
09:40  branch page still shows: main  [tick]   <- nothing
                                                  tested 3f1d02c
09:41  run 1194 starts, head_sha = 3f1d02c
09:53  run 1194 finishes, conclusion = failure

4. Between 09:40 and 09:41 the page showed a tick beside main while nothing at all had tested 3f1d02c.
5. At 09:53 the honest summary is: 0a95cc8 passed, 3f1d02c failed, and main is 3f1d02c.
6. If at 09:40 somebody asked “did main pass?”, the correct answer is that the question is badly formed.
7. The well-formed question is “did 3f1d02c pass?”, and at 09:40 the answer was “no run for that commit exists yet”.
8. There is a second way to get burned where nobody does anything wrong.
9. The reader rebased a local branch onto a moved main, and the branch’s hash changed.
10. Rebase does not carry commits across. It builds new commit objects with new parents, and new parents mean new hashes.
11. Every run recorded against the old hashes is still there, still correct, and now attached to commits that no branch points at.
12. The branch label now sits on brand new commits with zero runs. The tick you remember belonged to commits that are effectively gone.

PLAIN41.6.4 what is really happening inside#

1. A run record on the server stores many fields. Two of them matter here.
2. head_sha is the exact commit the run checked out. This field is the truth.
3. head_branch is the branch name as it read at trigger time. This field is a label, and labels move.
4. When you open a branch page, the service does roughly this: find the commit the branch points at right now, then look up the statuses and check runs attached to that commit.
5. That is correct behaviour, and it gives a right answer for the instant the page was rendered.
6. So where does the lie come from? Three places, none of them a bug.
7. Caching in your own tools. The reader’s machine reported 0a95cc8 from a file on disk, either .git/refs/remotes/origin/main or an entry in .git/packed-refs, written by the last fetch.
8. Human memory. You looked at the tick five minutes ago, then merged three pull requests, and then trusted the memory.
9. Frozen evidence. A screenshot or a chat message saying “CI is green, see attached” is a statement about a moment, not about now.
10. The fix is mechanical and takes one extra command.
11. Never carry a branch name into the verification step. Resolve it to a hash once, from the server, and carry the hash.
12. If somebody pushes between your resolve and your check, your conclusion is still exactly true, because it was a statement about the hash you named.
13. That is the whole difference between a claim you can defend and a claim you merely believe.

TECHNICAL41.6.5 the engineer’s version#

1. Step one: get the head SHA from the server, not from your local cache.

# live from the remote, no local cache involved
git ls-remote origin refs/heads/main
# 3f1d02c9d5f0a1b2c3d4e5f60718293a4b5c6d7e refs/heads/main

# or through the API
export SHA=$(gh api repos/OWNER/REPO/commits/main --jq .sha)
echo "$SHA"

2. git rev-parse origin/main does not do this. It reads the remote-tracking ref from disk and answers from cache. That is what produced 0a95cc8 during the reader’s outage while the real server value was unknown.
3. Step two: list the runs filtered by that exact SHA, never by branch.

gh run list --commit "$SHA" \
  --json databaseId,workflowName,event,status,conclusion,headSha

4. The same query in raw REST, which is what gh calls underneath:

gh api "repos/OWNER/REPO/actions/runs?head_sha=$SHA" \
  --jq '.workflow_runs[]
        | {id, name, event, status, conclusion, head_sha}'

5. Step three: look at the one run you care about, by its numeric ID, and read both its conclusion and its head SHA.

gh run view 11947722301 \
  --json status,conclusion,headSha,headBranch,event,url

6. Step four: assert rather than eyeball. A human reading two long hex strings side by side will miss a mismatch.

gh run view "$RUN_ID" --json headSha,conclusion \
  --jq 'if .headSha == env.SHA and .conclusion == "success"
        then "VERIFIED" else "NOT VERIFIED" end'

7. Two commit-level views give the rolled-up answer for that SHA, one per API:

# newer checks API, one entry per check run
gh api "repos/OWNER/REPO/commits/$SHA/check-runs" \
  --jq '.check_runs[] | {name, status, conclusion}'

# older commit status API, already rolled up
gh api "repos/OWNER/REPO/commits/$SHA/status" \
  --jq '{state, total_count}'

8. For a pull request, ask for the head object ID directly rather than deriving it, and pull the rollup in the same call:

gh pr view 42 --json headRefOid,statusCheckRollup \
  --jq '{sha: .headRefOid,
         checks: [.statusCheckRollup[]
                  | {name, conclusion}]}'

9. Field naming differs between the two interfaces and it trips people up. The gh --json output uses camelCase such as headSha and databaseId. REST uses snake_case such as head_sha and id. Same values, two spellings.
10. GET /repos/{owner}/{repo}/actions/runs accepts head_sha, branch, event, status, actor, created, check_suite_id and exclude_pull_requests as query parameters.
11. One commit can carry many runs: one per workflow file, plus every re-run, plus a separate run for the push event and the pull_request event. So “did this SHA pass” is incomplete without “in which workflow”.
12. A subtlety worth memorizing. For a pull_request event the run’s head_sha is the pull request branch head, but the code checked out is the ephemeral merge of that head into the base, which is what GITHUB_SHA holds. The run is filed under one hash and tested another.
13. The hash itself is a SHA-1 digest over the commit object in classic repositories, and SHA-256 in repositories created with the newer object format, which Git has supported experimentally since version 2.29 in October 2020. The commit object names its tree and its parents, so the hash transitively covers the entire content and history.
14. Because of that, verifying against a hash is verifying against exact bytes. There is no room left for “roughly the same code”.
15. Rebase consequence, stated precisely: rebasing writes new commit objects, so check runs and statuses remain attached to the abandoned hashes. They are not migrated, and there is no mechanism that would migrate them. git reflog still finds the old commits locally until garbage collection.
16. The sentence to memorize: a run R proves a property about commit R.head_sha, and proves nothing whatsoever about any ref.

WORDS41.6.6 remember these#

1. Commit SHA — the fingerprint of one exact snapshot — a 40-hex SHA-1, or 64-hex SHA-256, digest over the commit object.
2. Ref — a name pointing at a commit — a file under .git/refs or a line in .git/packed-refs.
3. head_sha — the commit the run really tested — the field on a workflow run record, spelled headSha in gh --json output.
4. Remote-tracking ref — your last known copy of the server’s branch — refs/remotes/origin/main, updated only by fetch, never live.
5. git ls-remote — ask the server what it has right now — a live ref advertisement over the transport, bypassing every local cache.
6. Stale tick — a pass mark for a commit that is no longer the head — a status attached to a superseded SHA and displayed next to a moved branch name.

41.7 Why verifying by run ID against a head SHA is the stronger claim#

PLAIN41.7.1 in simple words#

1. In the reader’s session, continuous integration was verified by run ID against the exact head hash.
2. It was not verified by glancing at a tick beside a branch name. That is the difference between proof and belief.
3. Here is the procedure in words, four steps.
4. Ask the server what the branch points at right now. Write that hash down.
5. List the runs whose tested commit is that hash.
6. Pick the run you care about and note its number.
7. Read that run’s result, and read the commit that run tested, and confirm it equals the hash you wrote down.
8. Only then say “this commit passed”.
9. That claim survives questioning, because every step names something that cannot move.
10. A hash cannot move. A run number cannot move. A branch name moves all day.
11. The extra cost is about ten seconds and three commands.
12. The cost it avoids is a bad deploy at six o’clock on a Friday.
13. There is a second reason this method is stronger, and it is the reason engineers care most about.
14. It fails loudly. If nothing tested your commit, the list comes back empty and you cannot mistake an empty list for a pass.
15. A tick beside a branch name never comes back empty. It always shows something, and that is exactly the problem.

PLAIN41.7.2 a picture in your head#

1. A school posts a notice on the board: “Class 10B: PASS”.
2. You are in class 10B. Are you finished?
3. Not yet. The board is a summary about a group, at a moment, written by somebody in a hurry.
4. Your own certificate carries your roll number. That number was never reassigned to anybody else.
5. Ask for the certificate bearing your roll number, and read the result printed on it.
6. Then compare the roll number on the certificate with the roll number on your identity card.
7. Two documents, one shared identifier, one comparison. That is the entire method, in school and in continuous integration.

Where this comparison breaks: a class notice can only be wrong in one way, by summarizing. A branch tick can be wrong in at least six different ways, and several of them look identical on the screen. Worse, a school certificate is issued by one office, while a commit status can be posted by anything holding a token, including a person typing an API call by hand.

PLAIN41.7.3 a worked example#

1. Start from the branch name and refuse to trust it.

$ export SHA=$(gh api repos/OWNER/REPO/commits/main --jq .sha)
$ echo "$SHA"
3f1d02c9d5f0a1b2c3d4e5f60718293a4b5c6d7e

$ gh run list --commit "$SHA" \
    --json databaseId,workflowName,conclusion,headSha
[]

2. An empty list. Nothing has ever tested this commit, even though the branch page a moment ago showed a green tick.
3. That tick belonged to 0a95cc8, the previous head. The claim “main is green” was true about a commit nobody is deploying.
4. Now the good case, after the run has actually happened.

$ gh run list --commit "$SHA" \
    --json databaseId,workflowName,conclusion,headSha
[
  {
    "databaseId": 11947722301,
    "workflowName": "ci",
    "conclusion": "success",
    "headSha": "3f1d02c9d5f0a1b2c3d4e5f60718293a4b5c6d7e"
  }
]

5. Now name the run and read it back on its own, so the answer does not depend on the filter having worked.

$ gh run view 11947722301 \
    --json conclusion,headSha,status,event,runAttempt
{
  "conclusion": "success",
  "headSha": "3f1d02c9d5f0a1b2c3d4e5f60718293a4b5c6d7e",
  "status": "completed",
  "event": "push",
  "runAttempt": 1
}

6. Four things were checked and all four matter.
7. status is completed, so the run finished rather than still running.
8. conclusion is success, so it finished well.
9. headSha equals the hash you resolved, so it tested the right commit.
10. runAttempt is 1, so nobody re-ran it until it went green.
11. Now the sentence you may say out loud: commit 3f1d02c passed workflow ci in run 11947722301 on the first attempt.

PLAIN41.7.4 what is really happening inside#

1. Six ways a branch-level tick can lie. Each is a real, ordinary situation, not a bug and not sabotage.
2. The run was for an older commit. The branch moved after the run. The page shows the newest result it can find, and briefly that is the old one.
3. The run tested the merge result, not your head. For a pull request the service builds a temporary merge of your branch into the target and tests that. Your head commit was never tested alone.
4. A required check was skipped, not passed. A job that never ran can report in a way that satisfies a rule, and a workflow that never started reports nothing at all.
5. The status came from something other than the run you have in mind. The name “build” on the screen is just a text label. Anything with a token can post that label.
6. A re-run replaced the result. Somebody pressed re-run until it went green. The run ID is the same, the number of attempts is not.
7. The check belongs to a merge commit that no longer exists. After a squash or rebase merge, the commit that was tested is not the commit that landed on the target branch.
8. Notice what all six have in common. In every case the tick is truthful about something, and that something is not the thing you are about to deploy.
9. The method in 41.7.1 defeats all six because it never asks a question whose subject can move.

TECHNICAL41.7.5 the engineer’s version#

1. The failure modes and the exact field to inspect for each.

2. Older commit. gh run list --commit "$SHA" returns an empty array when nothing tested that hash. Treat an empty array as a hard failure of verification, never as an absence of bad news.
3. Merge result. If event is pull_request, the run’s head_sha is the pull request branch head, but the job checked out refs/pull/N/merge, and GITHUB_SHA inside the job was that ephemeral merge commit. GitHub’s own troubleshooting page states that when the test merge commit has a status, it is the test merge commit that must pass, not the head commit.
4. Skipped versus passed. This is the sharpest edge in the whole area, and GitHub documents three different outcomes:

5. Read that table again. A job you skipped on purpose reports success and satisfies a required check. A workflow that never started reports nothing, stays pending, and blocks forever. Same English word, opposite consequences.
6. The documented mitigation for the fourth row is to add if: always() to a final summary job that depends on the others and explicitly fails if any dependency did not succeed.
7. Wrong poster. Any token with write access can create a commit status with any context and state, including success:

gh api -X POST repos/OWNER/REPO/statuses/$SHA \
  -f state=success -f context=build \
  -f description='looks fine to me'

8. That call is legitimate API use and is indistinguishable on the branch page from a real build result. Inspect the poster:

gh api "repos/OWNER/REPO/commits/$SHA/check-runs" \
  --jq '.check_runs[] | {name, conclusion, app: .app.slug}'

gh api "repos/OWNER/REPO/commits/$SHA/status" \
  --jq '.statuses[] | {context, state, creator: .creator.login}'

9. Re-runs. A re-run does not create a new run ID. It increments run_attempt on the same run, and the interface shows the latest attempt. Fetch an earlier attempt explicitly:

gh api repos/OWNER/REPO/actions/runs/11947722301/attempts/1 \
  --jq '{attempt: .run_attempt, conclusion, head_sha}'

10. “Re-run failed jobs” is more subtle still: it produces a new attempt in which the previously successful jobs are not executed again. The green you see is assembled from two different points in time.
11. Vanished merge commit. A squash merge creates one brand new commit on the target with a new hash and no checks. A rebase merge creates new commits for the same reason. Only a true merge commit preserves the tested parent.
12. So after any squash or rebase merge, the correct next step is to resolve the new head of the target branch and verify that hash on its own. What passed before the merge is evidence about a commit that is no longer in the first parent line of your branch.
13. Two further modes worth knowing, beyond the six:
14. continue-on-error: true on a step or job makes a real failure report as success at the job or run level. Read the step conclusions, not only the run conclusion.
15. A strategy.matrix job cancelled by fail-fast: true reports cancelled, which is not failure, and people read the absence of red as green.
16. A script for the whole verification, safe to put in a release checklist:

set -euo pipefail
REPO=OWNER/REPO
SHA=$(gh api "repos/$REPO/commits/main" --jq .sha)
RUN=$(gh run list --repo "$REPO" --commit "$SHA" \
        --workflow ci --limit 1 --json databaseId \
        --jq '.[0].databaseId')
test -n "$RUN" || { echo "no run for $SHA"; exit 1; }
gh run view "$RUN" --repo "$REPO" \
  --json conclusion,headSha,status,runAttempt \
  --jq --arg sha "$SHA" '
    if .status=="completed" and .conclusion=="success"
       and .headSha==$sha
    then "VERIFIED \($sha) run \(.runAttempt) attempt(s)"
    else error("NOT VERIFIED") end'

17. GitLab equivalent concepts: a pipeline has a sha, a status, and an id. glab ci status and the pipelines API filtered by sha do the same job. Jenkins records the built revision in the build metadata, reachable through the api/json endpoint of a build.

WORDS41.7.6 remember these#

1. Verification by run ID — naming the run rather than the branch — asserting run.head_sha == resolved_head and conclusion == success.
2. Test merge commit — the temporary join of your branch into the target — the refs/pull/N/merge ref that pull request runs actually check out.
3. Run attempt — which try you are looking at — the run_attempt counter, incremented by a re-run, with the latest shown by default.
4. Commit status — a small note pinned to a commit — a state plus a context, creatable by any token with write access.
5. continue-on-error — keep going after a failure — a step or job setting that turns a real failure into a reported success upstream.
6. Squash merge — many commits collapsed into one new commit — a merge strategy producing a target-branch commit that no check has ever seen.

41.8 Statuses, checks, and the two APIs behind them#

PLAIN41.8.1 in simple words#

1. There are two separate systems for pinning a result onto a commit. They were built years apart and both are still alive.
2. The older one is the commit status. It is a small note with four parts: a state, a short description, a link, and a name called the context.
3. Its states are pending, success, failure and error. That is all it can say.
4. The newer one is the check run, part of the checks system.
5. A check run says much more: whether it is queued, running or finished; what the outcome was; when it started and ended; a block of output text; and notes pinned to particular lines of particular files.
6. Check runs are collected into a check suite, which is roughly “everything one tool reported about one commit”.
7. Both systems attach their results to a commit hash. Both draw ticks and crosses in the same place on the page. That is why they get confused.
8. Branch protection lets you name some of these results as required.
9. A required result means: this pull request may not merge until a result with exactly that name reports success.
10. And here is the sharp edge. A required check that never arrives blocks the merge forever.
11. It blocks because “has not arrived” is not the same as “failed”. It sits in a third state that means “we are still waiting for this”.
12. Nothing times it out. Nothing gives up. It waits, and the merge button stays grey.

PLAIN41.8.2 a picture in your head#

1. Imagine a building site where a flat cannot be handed over until three inspection slips are pinned to the door.
2. Slip one is from the electrician, slip two from the plumber, slip three from the fire officer.
3. The old slips were postcards: a tick or a cross, a line of writing, nothing else.
4. The new slips are full forms: who inspected, when they started, when they finished, and photographs with arrows pointing at specific pipes.
5. The handover clerk does not judge quality. The clerk only checks that all three named slips are present and say “pass”.
6. Now suppose the fire officer was never told this flat exists.
7. There is no cross on the door. There is simply no third slip.
8. The clerk waits. Nobody is at fault, no alarm rings, and the flat is never handed over.

Where this comparison breaks: real inspectors eventually notice and telephone somebody. A required check has nobody to telephone. The rule is purely mechanical, and the only cure is a human editing the rule or arranging for the missing slip to appear.

PLAIN41.8.3 a worked example#

1. A commit status, created by hand, exactly as an old-style build tool would:

gh api -X POST repos/OWNER/REPO/statuses/$SHA \
  -f state=success \
  -f context='ci/unit-tests' \
  -f target_url="$BUILD_PAGE_URL" \
  -f description='412 tests, 0 failures'

2. A check run, as posted by an app, has far more structure:

{
  "name": "unit-tests",
  "head_sha": "3f1d02c9d5f0a1b2c3d4e5f6...",
  "status": "completed",
  "conclusion": "failure",
  "started_at": "2026-08-13T09:41:02Z",
  "completed_at": "2026-08-13T09:53:11Z",
  "output": {
    "title": "3 failing",
    "summary": "412 tests, 3 failures, 0 skipped",
    "annotations": [
      {
        "path": "src/cart/total.ts",
        "start_line": 44,
        "annotation_level": "failure",
        "message": "expected 1200, received 1199"
      }
    ]
  }
}

3. The annotation is the visible difference. It is what draws a red marker on line 44 of that file inside the pull request diff.
4. Reading both systems for one commit:

# newer: individual check runs
gh api "repos/OWNER/REPO/commits/$SHA/check-runs" \
  --jq '.check_runs[] | {name, status, conclusion}'

# older: the rolled-up commit status
gh api "repos/OWNER/REPO/commits/$SHA/status" \
  --jq '{state, statuses: [.statuses[].context]}'

5. The rollup rule for the older API is documented and simple: the combined state is failure if any context is error or failure; pending if there are no statuses at all or any context is pending; success only if every context’s latest status is success.
6. Read that middle clause again. Zero statuses rolls up to pending, not to success. Silence is treated as “not yet”, which is exactly the behaviour that makes a missing required check block forever.

PLAIN41.8.4 what is really happening inside#

1. When a run starts, the service creates a check suite for that commit and one check run per job.
2. Each check run begins as queued, becomes in progress, and finally becomes completed with a conclusion.
3. The conclusion is written once, when the job ends. Until then there is no conclusion at all, only a status.
4. That two-field design is the whole reason “pending” and “failed” are different things: status says whether it is over, conclusion says how it went, and the second is empty until the first says completed.
5. Branch protection compares a list of required names against the results present on the head commit of the pull request.
6. The comparison is by name, as text. It has no idea which tool produced the result and no idea what the tool did.
7. So renaming a job in your workflow file silently breaks a required check. The old name never arrives again, and every pull request blocks.
8. This is the single commonest cause of “the repository is stuck and nobody changed the rules”. Somebody did change something: the job’s name.
9. Path filters cause the same shape of failure from the other direction. If the workflow does not start, the check does not exist, and the required name never arrives.
10. The cure is a small job that always runs and always reports the required name, whose only work is to look at the other jobs and decide.

TECHNICAL41.8.5 the engineer’s version#

1. The commit status API is the older of the two and dates from the early years of the GitHub API. Create with POST /repos/{owner}/{repo}/statuses/{sha}. Fields: state (one of error, failure, pending, success), target_url, description, and context which defaults to the literal string default.
2. Documented limit: 1,000 statuses per SHA and context pair in a repository. Read them with GET /repos/{owner}/{repo}/commits/{ref}/statuses for the list, or .../status for the rollup.
3. In GraphQL the type is StatusState, with exactly five values: ERROR, EXPECTED, FAILURE, PENDING, SUCCESS. EXPECTED is the one you cannot post yourself; it is what a required-but-absent check looks like.
4. The checks API arrived with GitHub Apps in 2018 and is richer. A check suite is a collection of check runs created by one app for one commit. A check run is one individual test of that commit.
5. Only a GitHub App can create check runs. GITHUB_TOKEN inside Actions acts as an app installation token, which is why Actions can create them and your personal token generally cannot.
6. In GraphQL, CheckStatusState has six values: QUEUED, REQUESTED, WAITING, PENDING, IN_PROGRESS, COMPLETED. The REST spelling is lowercase.
7. CheckConclusionState has nine values. This table is the one to keep.

8. Notes on the awkward ones. neutral exists so a linter can report advice without blocking; it counts as not-failing. action_required is what an app returns when it needs you to authorize something, such as granting access.
9. stale can only be set by GitHub itself, and marks a result the platform has decided is no longer meaningful. startup_failure means the run could not begin, most often an invalid workflow file or an unavailable runner label.
10. timed_out on GitHub Actions arrives at timeout-minutes if you set one, and at the platform limit of 6 hours for a hosted job if you did not.
11. Branch protection matches required checks by name string against results on the head commit. Configure with PATCH /repos/{owner}/{repo}/branches/{branch}/protection or, in the newer model, with repository rulesets.
12. The permanent-block conditions, all documented, all real:

13. [skip ci], [ci skip], [no ci], [skip actions] and [actions skip] in a commit message, or a skip-checks: true trailer, suppress the run entirely. On a repository with required checks that is a self-inflicted permanent block.
14. The standard workaround is a gate job that always runs:

  ci-gate:
    if: always()
    needs: [lint, test, build]
    runs-on: ubuntu-latest
    steps:
      - name: Fail if any dependency did not succeed
        if: contains(needs.*.result, 'failure')
             || contains(needs.*.result, 'cancelled')
        run: exit 1

15. Require ci-gate and nothing else. It runs on every event that the workflow runs on, and its name is stable even when the jobs beneath it are renamed.
16. GitLab models the same idea differently: a job has allow_failure, a pipeline has a single status, and merge request approval rules reference the pipeline rather than individual named checks. There is no direct equivalent of a required check name, which removes this class of bug and removes the fine-grained control with it.
17. Observing tools: gh pr checks 42, gh pr checks 42 --watch, gh api repos/OWNER/REPO/commits/$SHA/check-runs, gh api repos/OWNER/REPO/branches/main/protection --jq .required_status_checks.

WORDS41.8.6 remember these#

1. Commit status — the old-style note on a commit — a state, context, description and target_url posted to the statuses endpoint.
2. Context — the name of an old-style status — the string branch protection matches against, defaulting to default.
3. Check run — the new-style detailed result — an app-created record with status, conclusion, timings, output and line annotations.
4. Check suite — all check runs from one app for one commit — the grouping the interface shows as a single reporting tool.
5. Annotation — a note pinned to a line of a file — a check run output entry with path, start_line and annotation_level.
6. Required status check — a name that must report success — a branch protection rule compared by text against results on the head commit.
7. Expected — waiting for a result that never came — StatusState.EXPECTED, the state that blocks a merge indefinitely.

41.9 Merge queues#

PLAIN41.9.1 in simple words#

1. Here is a problem that green ticks cannot solve on their own.
2. Two people each propose a change. Each change is tested alone, and each passes.
3. Both are merged within a minute of each other. The result is broken.
4. Nothing went wrong with either test. The tests answered the questions they were asked.
5. Neither test was ever asked the only question that matters: what happens when both of these land together?
6. Chapter 39 called this a semantic conflict: two changes that do not touch the same lines, and still contradict each other.
7. One person renames a function. The other person adds a new call to the old name. No line overlaps. The program does not build.
8. A merge queue exists to ask the missing question before the merge, not after.
9. It works like a queue at a counter. Changes wait in a line, and only one thing is joined at a time.
10. For each change in the line, the machine builds the result you would get if that change landed on top of everything ahead of it.
11. It tests that combined result.
12. Only if that passes does the change actually land.
13. So the thing that is tested is the thing that lands, in the order it lands.
14. That is the whole idea, and it is the only mechanical way to keep a mainline that always builds.

PLAIN41.9.2 a picture in your head#

1. Think of an aeroplane runway with one strip and a queue of aircraft.
2. Each pilot has already done a check on their own aircraft in the hangar. All passed.
3. But the tower does not clear them by hangar checks. The tower clears one aircraft at a time, in a known order, and re-checks the runway state each time.
4. Now add impatience. Waiting for each aircraft to finish before starting the next check is slow.
5. So the tower guesses. It assumes number one will be fine, and starts checking number two against a runway that already has number one on it.
6. It does the same for three and four. Four checks run at once, each assuming everything ahead of it succeeded.
7. If all guesses hold, everything lands in one runway’s worth of time instead of four.
8. If number two turns out to be faulty, the guesses made for three and four were built on a false assumption and must be thrown away and redone.
9. That guessing is called speculative execution, and it is the trick that makes queues fast enough to use.

Where this comparison breaks: aircraft are physical and independent; software changes interact. A faulty aircraft does not make the next one faulty, but a bad merge genuinely can invalidate every speculation behind it. Also, a runway handles one landing at a time by physics. A merge queue chooses to serialize, and could choose not to, at the cost of correctness.

PLAIN41.9.3 a worked example#

1. Three pull requests are ready. The test suite takes 12 minutes.
2. The old discipline, “rebase onto the latest mainline and re-run before merging”, plays out like this.

 0 min  A rebases onto main, starts CI
12 min  A green, A merges. main moves.
12 min  B rebases onto new main, starts CI
24 min  B green, B merges. main moves.
24 min  C rebases onto new main, starts CI
36 min  C green, C merges.

3. Total elapsed: 36 minutes, and three humans had to sit and watch.
4. The same three in a merge queue with speculation.

 0 min  A, B, C enter the queue in that order
 0 min  build main+A        (entry 1)
 0 min  build main+A+B      (entry 2, speculative)
 0 min  build main+A+B+C    (entry 3, speculative)
12 min  all three green
12 min  A, B, C merge in order. main moves once.

5. Total elapsed: about 12 minutes, and nobody watched anything.
6. Now the interesting case. Entry 2 fails, because B contradicts A.

12 min  entry 1 green, entry 3 green, entry 2 FAILED
12 min  A merges
12 min  B is removed from the queue and its author told
12 min  entry 3 is discarded: it assumed B would land
12 min  rebuild main+A+C
24 min  green, C merges

7. Entry 3’s result was correct about a world that never happened, so it is thrown away. That is the cost of speculation.
8. Costs compared, using 12-minute runs and a 5 percent chance that any given change fails when combined:

9. The wasted machine time under speculation is real: a failure at position two discards every build behind it. Teams accept that because machine minutes are cheaper than a broken mainline and cheaper than engineers waiting.

PLAIN41.9.4 what is really happening inside#

1. When a pull request is added to the queue, the service does not merge it. It creates a temporary branch.
2. That temporary branch contains the target branch, plus every queued change ahead of this one, plus this one.
3. That temporary branch is the thing your workflows run against. It is a real commit, with a real hash, and the run is filed against that hash.
4. This is why a merge queue needs its own trigger. A workflow that only listens for pull_request will never run for a queue entry.
5. When the run for an entry finishes green and everything ahead of it has merged, the entry is merged into the target branch.
6. If the run fails, that entry is removed from the queue, the author is told why, and the temporary branches for everything behind it are rebuilt without it.
7. Batching is the other lever. Instead of one entry per build, group five entries into one temporary branch and test the group.
8. A green batch merges five changes for the price of one build. A red batch tells you only that one of five is bad, and you must split it to find out which.
9. Splitting a failed batch in half repeatedly finds the culprit in about log2(n) further builds. That is the same bisection idea as git bisect from Chapter 40.
10. Every serious queue has the same three knobs: how many entries to build at once, how many to merge at once, and how long to wait before giving up on filling a batch.

TECHNICAL41.9.5 the engineer’s version#

1. The property a merge queue enforces has a name in the Rust community: the “not rocket science rule of software engineering”, a phrase attributed to Graydon Hoare, Rust’s original author. Stated plainly: automatically maintain a repository of code that always passes all the tests.
2. History. The Rust project’s bors bot implemented this from the early 2010s, later reimplemented as homu and bors-ng. OpenStack’s Zuul, in use from 2012, added speculative execution across multiple projects and called the mechanism a gate pipeline. Zuul v3 arrived in 2017.
3. Commercial and hosted implementations followed: Mergify, Aviator, Graphite, and GitLab’s merge trains, which appeared in GitLab 11.2 in 2018.
4. GitHub’s own merge queue entered public beta on 8 February 2023 and became generally available on 12 July 2023.
5. Mechanically, on GitHub: entries create temporary branches under a reserved prefix, gh-readonly-queue/, one per entry, named after the base branch and the pull request. You cannot push to them.
6. Workflows must listen for the merge_group event with activity type checks_requested, or they will not run for queue entries at all:

on:
  pull_request:
    branches: [main]
  merge_group:
    types: [checks_requested]

7. Forgetting merge_group is the classic first-day merge queue failure. The entry sits waiting for a required check that no workflow will ever produce, which is exactly the permanent-block condition from 41.8.
8. Documented settings, with their ranges as of August 2026:

9. Merge method is configurable as merge, rebase or squash, and applies to the whole queue rather than per pull request.
10. “Only merge non-failing pull requests” controls batch semantics. When on, every entry must pass. When off, a group may merge if the final entry passes for the combined changes, which is faster and admits a change that never passed on its own.
11. On failure, GitHub removes the entry from the queue, records the reason on the pull request timeline, recreates the temporary branch without it, and continues with the rest.
12. Cost model. With batch size b, per-run time T, and independent failure probability p per change, a batch is green with probability (1 - p)^b. Expected builds per merged change falls with b while all is well, and rises sharply once b * p approaches 1. Most teams land on batches of 2 to 8.
13. The older discipline is still available and still correct: branch protection offers “Require branches to be up to date before merging”, which forces each author to rebase or merge the latest target and re-run before the merge button unlocks.
14. That setting has two costs a queue does not: it serializes humans rather than machines, and it has a race. Between your last green run and your click, somebody else can merge. The window is small but not zero.
15. It also scales badly. On a busy repository, the mainline moves faster than your test suite runs, so you can never get to the front of the line. Teams hit this at roughly merges per hour x test minutes > 60.
16. Comparison of approaches:

17. Observing tools: gh pr merge --auto, gh api repos/OWNER/REPO/merge-queue for queue state, the merge_group payload inside the run, and git ls-remote origin 'refs/heads/gh-readonly-queue/*' to see live entries.

WORDS41.9.6 remember these#

1. Merge queue — a line of changes joined one at a time — a serializing gate that tests the prospective merged state before merging.
2. Semantic conflict — two changes that clash without touching the same lines — a textually clean merge that fails to build or fails tests.
3. Speculative execution — testing entries assuming the ones ahead succeed — parallel builds of stacked prospective states, discarded on any failure.
4. Batch — several entries tested as one — a group merged together, requiring bisection to locate a culprit when it fails.
5. merge_group — the event a queue entry raises — the GitHub Actions trigger with activity type checks_requested.
6. Merge train — GitLab’s name for the same idea — sequential pipelines on the projected result of merging each queued request.
7. Not rocket science rule — always keep the mainline passing — the invariant a gating queue mechanically enforces.

41.10 mergeStateStatus, value by value#

PLAIN41.10.1 in simple words#

1. When you ask GitHub about a pull request, one field tells you why the merge button is or is not available. It is called mergeStateStatus.
2. It has exactly eight possible values, and they are not all the same kind of thing, which is why people misread them.
3. Some values are about the code itself: can these two branches be joined at all?
4. Some are about the rules: the code is joinable, but a policy says not yet.
5. One is about the pull request’s own state: it is a draft.
6. And one is about GitHub itself: I have not worked it out yet, ask again.
7. The reader saw three of them in one session: CLEAN, BLOCKED and UNKNOWN.
8. Two of these are worth learning before the rest.
9. UNKNOWN almost never means something is wrong. It means the answer is still being computed. The correct response is to ask again in a second.
10. BLOCKED does not mean a merge conflict. It means the branches join fine and a protection rule is not satisfied.
11. Getting those two right removes most of the confusion this field causes.

PLAIN41.10.2 a picture in your head#

1. Think of a clerk at a counter deciding whether your form can be filed.
2. DIRTY is: your form physically will not fit in the folder. The paper is the wrong shape. Fix the paper.
3. BEHIND is: your form is on an old version of the template. Copy it onto the current template.
4. BLOCKED is: the form is perfect and the folder is ready, but a signature is missing. Nothing is wrong with your paper.
5. UNSTABLE is: everything is signed, but one optional report attached to it came back with a complaint. The clerk will still file it if you insist.
6. DRAFT is: you wrote “draft” across the top yourself. The clerk is obeying you.
7. CLEAN is: file it.
8. UNKNOWN is: the clerk has not finished reading it yet. Come back in a moment.

Where this comparison breaks: a clerk gives you one reason. mergeStateStatus gives you the first blocking reason it finds, in an order GitHub does not publish. A pull request can be behind and blocked and have failing checks all at once, and you will be shown a single word. Never treat that word as a complete list of what is wrong.

PLAIN41.10.3 a worked example#

1. Asking for the field, together with the things that explain it:

gh pr view 42 --json \
  number,mergeable,mergeStateStatus,isDraft,headRefOid,\
reviewDecision

2. A typical answer while GitHub is still thinking:

{
  "number": 42,
  "mergeable": "UNKNOWN",
  "mergeStateStatus": "UNKNOWN",
  "isDraft": false,
  "headRefOid": "3f1d02c9d5f0a1b2c3d4e5f60718293a4b5c6d7e",
  "reviewDecision": "APPROVED"
}

3. Wait one or two seconds and ask again. This is not a workaround. It is the documented way the field behaves.

{
  "mergeable": "MERGEABLE",
  "mergeStateStatus": "BLOCKED",
  "reviewDecision": "APPROVED"
}

4. Read those two lines together, because alone each is misleading.
5. mergeable: MERGEABLE says the branches join without conflict.
6. mergeStateStatus: BLOCKED says a rule is unsatisfied. The code is fine. The policy is not.
7. To find out which rule, ask the checks, not this field:

gh pr checks 42
gh api repos/OWNER/REPO/branches/main/protection \
  --jq .required_status_checks.contexts

8. Nine times out of ten the answer is a required check sitting in Expected, which is exactly the permanent block from 41.8.

PLAIN41.10.4 what is really happening inside#

1. GitHub does not keep a permanent answer to “can this merge?”.
2. It computes one on demand, in the background, by attempting a test merge of the head into the base.
3. That computation takes time, and the API answers immediately whether or not it has finished.
4. So the first response after any change to either branch is very often UNKNOWN, with the REST equivalent field mergeable set to null.
5. Asking again triggers or picks up the computation. This is documented, and it is why every serious script polls this field rather than reading it once.
6. Once the test merge succeeds, GitHub knows the branches join, and moves on to the rules.
7. It then reports the first obstacle it finds: draft status, protection rules, check results, or nothing at all.
8. This is why the values are a mixture of categories. The field is not a classification of the pull request. It is a report of the first thing stopping the button from lighting up.

TECHNICAL41.10.5 the engineer’s version#

1. mergeStateStatus is a GraphQL enum on the PullRequest type. GitHub’s documented values and descriptions, with the action each implies:

2. UNKNOWN, precisely. The mergeability computation is asynchronous. In REST, GET /repos/{owner}/{repo}/pulls/{number} returns mergeable: null until the background job completes, and the documentation instructs you to retry. In GraphQL the same condition surfaces as UNKNOWN.
3. Therefore: UNKNOWN is not an error, not a conflict, and not a policy failure. It is “not yet”. Poll with a short delay, a few times, before treating it as anything else. Concluding anything from a single UNKNOWN is the commonest mistake made with this field.
4. BLOCKED, precisely. It means the branches merge cleanly and a branch protection or ruleset requirement is unmet. It is not a merge conflict. DIRTY is the merge conflict value.
5. Common causes of BLOCKED, none of which are visible in the field itself: a required review missing, a requested change outstanding, a required status check absent or failing, a CODEOWNERS review missing, a required conversation unresolved, or a required deployment or signature rule.
6. UNSTABLE versus BLOCKED is the distinction people most often get backwards. UNSTABLE means a check is not passing but that check is not required, so a user with permission can still merge. BLOCKED means a rule forbids it outright.
7. BEHIND appears only when the repository has “Require branches to be up to date before merging” enabled. Without that setting, an out-of-date branch reports CLEAN or UNSTABLE instead, and merges happily.
8. HAS_HOOKS is specific to GitHub Enterprise Server, where an administrator can install pre-receive hooks. It means everything is fine but a server-side hook still gets the final word at push time, so a merge can still be rejected after you click.
9. DIRTY means Git cannot construct the merge commit without conflict resolution. Nothing in continuous integration will fix it. A human must rebase or merge and resolve.
10. Companion fields you should always fetch alongside it, because mergeStateStatus alone is not diagnostic:

11. On some GitHub Enterprise Server versions, mergeStateStatus sits behind a schema preview and requires the Accept header application/vnd.github.merge-info-preview+json. The gh CLI sets what it needs; a hand-written GraphQL client may not.
12. Practical polling pattern, which is what a release script should contain:

for i in 1 2 3 4 5; do
  S=$(gh pr view 42 --json mergeStateStatus \
        --jq .mergeStateStatus)
  [ "$S" != "UNKNOWN" ] && break
  sleep 2
done
echo "state: $S"
case "$S" in
  CLEAN|HAS_HOOKS) echo "safe to merge" ;;
  BLOCKED)  echo "a rule is unmet; run gh pr checks" ;;
  UNSTABLE) echo "a non-required check is failing" ;;
  BEHIND)   echo "update branch and re-run CI" ;;
  DIRTY)    echo "resolve conflicts by hand" ;;
  DRAFT)    echo "mark ready for review" ;;
  UNKNOWN)  echo "still computing; do not conclude" ;;
esac

13. GitHub’s own documentation notes that the REST field mergeable_state, the lowercase sibling of this enum, has historically been treated as unofficial and subject to change. Prefer the GraphQL enum, and prefer reading the underlying facts over branching on this one word.

WORDS41.10.6 remember these#

1. mergeStateStatus — why the merge button is or is not lit — a GraphQL enum on PullRequest with eight values.
2. mergeable — can the branches be joined at all — a separate tri-state field, null or UNKNOWN while still computing.
3. CLEAN — nothing is in the way — mergeable with passing commit status.
4. BLOCKED — a rule says no, not a conflict — branch protection or ruleset requirement unmet.
5. UNSTABLE — mergeable but something non-required is failing — merge is permitted, quality is not vouched for.
6. DIRTY — the branches genuinely conflict — Git cannot construct the merge commit.
7. BEHIND — your branch is on an old base — only reported when “require branches to be up to date” is enabled.
8. UNKNOWN — GitHub has not finished computing — re-query; it carries no information about the pull request.

41.11 What a good pipeline actually does, in order#

PLAIN41.11.1 in simple words#

1. A pipeline is a list of checks in a chosen order. The order is not decoration.
2. The rule is: cheapest and most likely to fail goes first.
3. A check that takes 8 seconds and catches a typo should never run after a check that takes 20 minutes.
4. So a sensible order looks like this.
5. Lint: read the code without running it, and complain about style and obvious mistakes. Seconds.
6. Build: turn source into something runnable. If it does not build, nothing after this matters.
7. Unit tests: test small pieces on their own, with nothing real attached. Fast, many, precise about where the fault is.
8. Integration tests: start real pieces together, a database, a queue, and check they talk correctly. Slower, fewer, vaguer about where the fault is.
9. Security scanning: look for known-bad dependencies, leaked keys, and dangerous code patterns.
10. Artifact build: package the thing you will actually ship, once, and keep it.
11. Deploy to staging: put that exact package on a copy of production.
12. Smoke tests: a handful of checks against the running copy. Does the home page load? Can one user log in?
13. Production: ship the same package that passed everything above.
14. Notice the package is built once and reused. Rebuilding before production would mean shipping something nobody tested.

PLAIN41.11.2 a picture in your head#

1. Think of security at an airport, in the order they actually use.
2. First a glance at your ticket at the door. One second, catches most people who are in the wrong building.
3. Then the document check. Thirty seconds.
4. Then the bag scanner. Two minutes.
5. Then, for a few people, a hand search. Ten minutes.
6. Nobody hand-searches everybody first. It would be correct and it would be useless, because the queue would never move.
7. Each stage is more expensive and catches fewer people than the one before.

Where this comparison breaks: airport stages are independent, and a pipeline’s stages are not. A build failure makes every later stage meaningless, so a pipeline stops, while an airport keeps processing the rest of the queue. Also, the airport cannot rerun you from a cache; a pipeline can and should.

PLAIN41.11.3 a worked example#

1. Real timings from a medium web project. Yours will differ; the shape holds.

2. Run in strict order, that is about 18 minutes. Too slow.
3. Run lint, unit tests and the security scan in parallel after the build, and the critical path drops to about 12 minutes.
4. Caching the dependency download saves roughly 60 to 90 seconds of that on every single run.
5. A worked cache and artifact pair in GitHub Actions:

      - uses: actions/cache@v4
        with:
          path: ~/.npm
          key: npm-${{ hashFiles('package-lock.json') }}
          restore-keys: npm-

      - run: npm ci && npm run build

      - uses: actions/upload-artifact@v4
        with:
          name: app-${{ github.sha }}
          path: dist/
          retention-days: 14

6. Note the cache key. It is a hash of the lock file, so a dependency change makes a new key automatically, and a stale cache can never be used.
7. Note the artifact name. It carries the commit SHA, so the package you deploy can be traced back to the exact commit that produced it.

PLAIN41.11.4 what is really happening inside#

1. Two words that sound alike and are not: cache and artifact.
2. A cache is a speed trick. It holds things you could recreate, like downloaded dependencies. Losing it costs time and nothing else.
3. An artifact is a product. It holds the thing you built and will ship. Losing it means you cannot deploy without rebuilding.
4. Never treat a cache as a source of truth, and never rebuild an artifact between test and deploy.
5. Parallelism splits work across machines. Four machines running a quarter of the tests each finish in roughly a quarter of the time, plus the fixed startup cost of each machine.
6. That fixed cost is why splitting a 40-second suite into eight parts makes it slower, not faster.
7. Matrix builds are parallelism over configurations rather than over test files: the same suite on three operating systems and two language versions.
8. Fail-fast decides what happens when one parallel branch fails. On, the siblings are cancelled, saving money. Off, they finish, giving you the full picture.
9. Use fail-fast on for pull requests, where speed matters, and off for nightly matrix runs, where the full picture matters.
10. A flaky test is a test that passes and fails on the same code, with no change in between.
11. Flaky tests do something worse than waste time. They teach people that red means “press the button again”.
12. Once a team believes that, red stops carrying information, and the whole pipeline becomes decoration. This is the way continuous integration dies.

TECHNICAL41.11.5 the engineer’s version#

1. The stage order above is the pipeline described in Continuous Delivery by Jez Humble and David Farley in 2010, where it is called a deployment pipeline, with the commit stage first and later stages progressively slower and more production-like.
2. The proportions come from the test pyramid, named by Mike Cohn in Succeeding with Agile in 2009: many fast unit tests, fewer integration tests, very few end-to-end tests.
3. The 10-minute build rule comes from Extreme Programming and is repeated in the Continuous Delivery literature. Past roughly 10 minutes, developers stop waiting for the result and context-switch, and the feedback loop that justifies the whole apparatus is gone.
4. GitHub Actions cache limits as documented in August 2026: 10 GB total per repository, with least-recently-used eviction, and entries removed after 7 days without access. Cache is keyed by an exact string, with restore-keys giving ordered prefix fallbacks.
5. Artifact retention on GitHub Actions defaults to 90 days, configurable from 1 to 90 days for public repositories and 1 to 400 days for private and enterprise repositories.
6. Caches are scoped by branch: a run on a branch can read caches from that branch and from the default branch, but not from unrelated branches. This is a deliberate isolation boundary, since a cache is a writable path shared between runs and therefore a cross-branch attack surface.
7. Security scanning splits into named categories, and mixing them up leads to thinking you are covered when you are not:

8. CodeQL came to GitHub through the acquisition of Semmle in September 2019. Dependabot came through the acquisition of Dependabot in May 2019. Both are now built into the platform.
9. Software bills of materials have two mainstream formats: SPDX, whose version 2.2.1 became ISO/IEC 5962:2021, and CycloneDX, standardized as ECMA-424 in
   2024. Producing one is now a common late pipeline stage.
10. Flakiness, with a real figure: Google’s testing blog reported in May 2016 that almost 16 percent of their roughly 4.2 million tests showed some level of flakiness, and that flaky failures consumed a substantial share of engineering attention. Treat the exact percentage as specific to Google, and the phenomenon as universal.
11. Standard flakiness controls, in increasing order of honesty: retry the test and hope; quarantine it into a non-blocking suite and file a bug; fix the root cause, which is nearly always shared state, real time, real network, or unordered iteration.
12. Retrying without quarantining is the trap. It hides the signal and lets the flake rate grow until the suite is useless.
13. Required versus optional checks, restated in terms of 41.8: a required check must appear and must report success. An optional check appears and is advisory, and produces UNSTABLE rather than BLOCKED when it fails.
14. A workable policy: require the gate job, the build, and the unit tests. Leave slow end-to-end suites, performance benchmarks and advisory linters optional, and review them on a schedule instead.
15. Test time budgets, as targets rather than laws:

16. The four DORA metrics from the 2018 book Accelerate by Nicole Forsgren, Jez Humble and Gene Kim are the standard way to judge whether the pipeline is actually helping: deployment frequency, lead time for changes, change failure rate, and time to restore service. Note that two of the four are about recovery, not prevention.
17. Observing tools: gh run view --log, the timing breakdown on a run page, gh api repos/OWNER/REPO/actions/cache/usage, and any test reporter that emits JUnit XML for trend analysis.

WORDS41.11.6 remember these#

1. Deployment pipeline — the ordered set of checks from commit to production — the staged model set out in Continuous Delivery, 2010.
2. Cache — a recreatable speed-up — keyed storage of dependencies, safe to lose, never a source of truth.
3. Artifact — the built product you will ship — a retained output passed between jobs and to deployment, built once.
4. Matrix build — the same job across configurations — a cross product of variables expanded into parallel jobs.
5. Fail-fast — cancel siblings on first failure — a matrix strategy trading diagnostic completeness for speed and cost.
6. Flaky test — passes and fails on identical code — a non-deterministic test, usually caused by shared state, timing or ordering.
7. SAST, SCA, DAST — code scan, dependency scan, running-system scan — static analysis, software composition analysis, dynamic testing.
8. Test pyramid — many small tests, few large ones — the proportional model named by Mike Cohn in 2009.

41.12 Deployment strategies#

PLAIN41.12.1 in simple words#

1. You have a package that passed everything. Now you must put it in front of real users. How you do that is the deployment strategy.
2. Recreate: stop the old version, start the new one. Simple, and there is a gap where nothing works.
3. Rolling: you have ten copies running. Replace them a few at a time. No gap, but for a while both versions are live at once.
4. Blue-green: run two complete environments. Blue is serving users. Deploy to green, test it, then switch all traffic to green in one moment. Blue stays there, ready to take traffic back.
5. Canary: send 1 percent of users to the new version. Watch the error rate. If it looks fine, go to 5 percent, then 25, then everyone.
6. Feature flags: ship the new code to everyone but keep it switched off, then switch it on for chosen users later, without deploying anything.
7. That last one is the important idea, so say it clearly.
8. Deploying is putting code on machines. Releasing is letting users reach it.
9. Feature flags separate those two things, so a deploy stops being a scary event and becomes routine.
10. And turning a feature off is instant, while undoing a deploy is not.

PLAIN41.12.2 a picture in your head#

1. Picture changing the menu at a restaurant with a hundred tables.
2. Recreate: close the restaurant, change everything, reopen. Everyone waits outside.
3. Rolling: change the menus table by table. Nobody waits, but for twenty minutes two menus exist and the kitchen must cook from both.
4. Blue-green: build an identical second dining room, set it up completely, check it, then walk everyone across in one go. Keep the old room furnished for a week in case.
5. Canary: give the new menu to two tables, watch what they send back, then widen it.
6. Feature flags: print every dish on every menu from the start, but mark the new ones “ask the waiter”. Turning them on is a word to the staff, not a reprint.

Where this comparison breaks: menus have no state. Software has databases, and a database cannot be walked across to a new room. That single difference is what makes rollback hard, and it is the subject of 41.12.4.

PLAIN41.12.3 a worked example#

1. A service with 12 identical copies, a 90-second start time, and a rolling update replacing 25 percent at a time.
2. 12 copies, 25 percent surge means 3 new copies start while at most 3 old ones go away.
3. Four waves of 3, each about 90 seconds plus a health check, is roughly 7 to 8 minutes to fully roll out.
4. During those 8 minutes, both versions serve real traffic. Both must be able to read the same data and answer the same requests.
5. Compare the same release under four strategies:

6. Blue-green and canary buy fast rollback with double the running cost during the change. That is the trade, stated honestly.
7. A Kubernetes rolling update is configured with two numbers:

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 25%
    maxUnavailable: 25%

8. maxSurge is how many extra copies may exist above the target. maxUnavailable is how many may be missing. Both default to 25 percent.
9. Setting maxUnavailable: 0 guarantees no reduction in capacity and requires spare room to surge into.

PLAIN41.12.4 what is really happening inside#

1. Rollback sounds symmetric. It is not.
2. Code rollback is easy: the old package still exists, so put it back.
3. Data rollback is often impossible, because the new version has already written data in a new shape.
4. Suppose the release renamed a column from phone to phone_number and copied the values across.
5. New code writes to phone_number. Users create records for twenty minutes.
6. Now roll the code back. The old code reads phone, which no longer receives writes. Those twenty minutes of phone numbers are invisible.
7. Nothing crashed. The data is simply wrong, which is worse.
8. The discipline that fixes this is called expand and contract, or parallel change, and it takes three releases instead of one.
9. Expand: add phone_number while keeping phone. Write to both. Read from phone. Deploy. This release is safe to roll back.
10. Migrate: switch reads to phone_number. Still writing both. Deploy. Still safe to roll back.
11. Contract: stop writing phone, then drop it, in a later release, once you are certain you will not roll back past this point.
12. The rule underneath: every deployment must be compatible with the version before it and the version after it, because during any rolling update both are running at once.
13. That rule is why database changes are the hard part of continuous deployment, and why teams that ignore it get exactly one bad night and then adopt it.

TECHNICAL41.12.5 the engineer’s version#

1. Blue-green deployment as a named practice was popularized by Dan North and Jez Humble and set out in Continuous Delivery in 2010. Canary releasing takes its name from the caged birds carried into coal mines to detect gas.
2. Feature toggles were described in detail in Pete Hodgson’s 2016 article Feature Toggles on Martin Fowler’s site, which classifies them into release toggles, experiment toggles, ops toggles and permission toggles, with very different lifetimes.
3. Flickr’s 2009 engineering post Flipping Out is the widely cited early description of running continuous deployment behind flags, with more than ten deploys a day.
4. Strategy comparison with the properties that actually decide the choice:

5. Canary requires observability that most teams underestimate: per-version metrics. If your error rate is not broken down by build, a canary tells you nothing you did not already know.
6. Typical canary schedule: 1 percent for 10 minutes, 5 percent for 20 minutes, 25 percent for 30 minutes, then 100 percent, with automatic rollback on any guard metric breach. Guard metrics are usually error rate, latency at the 99th percentile, and a business counter such as checkouts per minute.
7. Kubernetes supports Recreate and RollingUpdate natively; blue-green and canary are built on top with service selectors, ingress weights, or a service mesh. Argo Rollouts and Flagger are the common controllers for this.
8. Expand and contract is documented as the ParallelChange pattern by Danilo Sato in 2014. Its formal requirement is that every schema state supports both the previous and the next application version, sometimes called N-1 compatibility.
9. Migrations that are not reversible in practice: dropping a column, dropping a table, narrowing a type, adding a NOT NULL constraint without a default, and any data transformation that loses the original value. Plan these as separate releases, never bundled with a behaviour change.
10. GitHub Actions models targets as environments, which carry their own secrets and variables plus protection rules:

11. 43,200 minutes is 30 days, the documented maximum wait timer. A wait timer is a real control: it is the mechanical version of “let it bake in staging overnight”.
12. A job binds to an environment with environment: production, at which point the environment’s secrets become available and its protection rules apply before the job starts, not during it.
13. GitLab expresses the same idea with environment: plus when: manual, and protected environments with approval rules. Argo CD expresses it as a sync policy, manual or automated, per application.
14. Observing tools: gh api repos/OWNER/REPO/deployments, gh api repos/OWNER/REPO/environments, kubectl rollout status, kubectl rollout undo, and per-version dashboards in your metrics system.

WORDS41.12.6 remember these#

1. Recreate — stop the old, start the new — a strategy with a deliberate outage window and no version overlap.
2. Rolling update — replace instances a few at a time — controlled by surge and unavailability limits, with both versions live.
3. Blue-green — two full environments, one switch — instant cutover and instant rollback at double running cost.
4. Canary — a small slice of traffic first — progressive delivery gated on per-version guard metrics.
5. Feature flag — code shipped but switched off — runtime toggle decoupling deploy from release, reversible in milliseconds.
6. Expand and contract — change the schema in compatible steps — the ParallelChange pattern giving N-1 compatibility across releases.
7. Environment — a named deployment target with rules — a scope holding secrets, reviewers, wait timers and branch policies.

41.13 Secrets in CI#

PLAIN41.13.1 in simple words#

1. A pipeline needs passwords. It signs in to a registry, uploads to a server, talks to a cloud account.
2. Those values cannot live in the repository, because everyone who can read the code can read them, forever, in every clone.
3. So they are stored separately, in a box the service keeps locked, and handed to the machine only while the job runs.
4. Inside the job they appear as environment variables, which are just named values the running program can read.
5. When the job ends, the machine is destroyed and the values go with it.
6. Two rules matter more than everything else.
7. Rule one: never print a secret. Not to the log, not in an error message, not in a debug dump.
8. The service tries to help by blacking out anything matching a known secret, but that is text matching. Change the text even slightly and the blackout fails.
9. Encode a key in base64 and print it, and it goes to the log in full, because the blacked-out pattern no longer matches.
10. Rule two: a stranger’s pull request gets no secrets at all.
11. That is why a build from an outside contributor can test the code and cannot publish anything.
12. And the modern advice goes further: stop storing long-lived cloud passwords at all.
13. Instead the job proves who it is, and the cloud hands it a temporary key that expires in an hour.
14. Nothing long-lived exists to steal, which changes the shape of the whole problem.

PLAIN41.13.2 a picture in your head#

1. Think of a building that contractors visit.
2. The old way: give each contractor a copy of the master key. It works forever, it can be copied, and if one is lost nobody knows.
3. The new way: the contractor shows an identity card at reception. Reception checks it against a list, and issues a visitor badge valid for one hour, for one floor.
4. The badge cannot be reused tomorrow. Losing it costs almost nothing.
5. Reception never gives out the master key, because there is no need.
6. That is the difference between a stored cloud password and identity federation.

Where this comparison breaks: the contractor’s identity card is a physical thing they carry. In continuous integration the identity is issued fresh by the platform for each job, and its trustworthiness rests entirely on the platform not lying about which repository is asking. So you must pin the trust rule down to an exact repository and branch, or any repository on the platform can walk in with a valid card.

PLAIN41.13.3 a worked example#

1. The wrong way, and it is written like this in real repositories:

      - run: |
          echo "Token is ${{ secrets.API_TOKEN }}"
          curl -H "Authorization: Bearer $TOKEN" ...

2. Two faults. The value is interpolated into the script text, so it can appear in an error trace. And it is printed on purpose.
3. The right way passes secrets through the environment, never into the script body:

      - env:
          API_TOKEN: ${{ secrets.API_TOKEN }}
        run: |
          curl -sS -H "Authorization: Bearer $API_TOKEN" \
               "$API_BASE/v1/publish"

4. Now the shell receives a variable name. The value never becomes part of the command text that appears in logs and traces.
5. The identity-federation version needs no stored password at all:

    permissions:
      id-token: write
      contents: read
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::111122223333:role/ci
          aws-region: ap-south-1

6. There is no secrets.AWS_SECRET_ACCESS_KEY anywhere. The id-token: write permission lets the job ask GitHub for a signed identity document, and the cloud exchanges it for credentials lasting about an hour.
7. Pinning a third-party action to an exact commit, with the version in a comment so tools can still update it:

      - uses: actions/checkout@11bd719  # v4.2.2

8. That short form is for the page. In a real file, write the full 40-character hash, because a tag can be moved and a full hash cannot.

PLAIN41.13.4 what is really happening inside#

1. A secret is encrypted before it leaves your browser, stored encrypted, and decrypted only at the moment a job that is allowed to see it starts.
2. The service also registers each secret value with the log processor, which scans every line and replaces exact matches with asterisks.
3. That masking is genuinely useful and genuinely shallow. It matches strings.
4. It does not know that base64 -w0 key.pem produces a different string containing the same secret. It does not know that printing a JSON object containing the key is the same disclosure.
5. This is exactly how the March 2025 attack on a popular third-party action worked: the malicious code read secrets out of the runner process and printed them in an encoded form, and masking did not catch it.
6. Fork pull requests get no secrets because the code being tested is not yours. Section 41.5 covered why, and this is the other half of the same rule.
7. For identity federation the sequence is short and worth knowing.
8. The job asks GitHub for a signed token describing itself: which repository, which branch, which environment, which workflow.
9. The cloud provider has been configured in advance to trust tokens signed by GitHub, but only when the description matches a rule you wrote.
10. If it matches, the cloud issues short-lived credentials. If it does not, the request is refused and nothing is leaked.
11. The whole exchange takes under a second and leaves nothing behind.

TECHNICAL41.13.5 the engineer’s version#

1. GitHub encrypts secrets client-side with a libsodium sealed box before upload, stores them encrypted, and decrypts them only when injecting into a runner. Documented limits as of August 2026: 48 KB per secret, 1,000 organization secrets, 100 repository secrets, 100 environment secrets.
2. A value larger than 48 KB must be encrypted into a file committed to the repository, with only the passphrase stored as a secret. This is the documented workaround, not a hack.
3. Masking is implemented by registering values with the log service. You can register more at runtime with the workflow command ::add-mask::value, which is how you protect a secret derived inside the job.
4. Masking limitations, all real: it does not survive transformation such as base64 or URL encoding, it does not apply to values split across lines, and it does not protect artifacts, only logs.
5. Scoping: repository secrets, organization secrets with repository allow lists, and environment secrets which are only available to jobs bound to that environment and only after its protection rules pass.
6. GITHUB_TOKEN is an automatically issued installation token, valid for the duration of the job, invalidated at the end. Declare permissions: at workflow or job level and grant the minimum. For newly created repositories and organizations GitHub changed the default to read-only in 2023; older ones may still default to write, so check rather than assume.
7. OIDC federation, precisely. GitHub Actions gained support in a changelog entry on 27 October 2021 and was declared generally available in November
8. The job requests a JSON Web Token from the endpoint in ACTIONS_ID_TOKEN_REQUEST_URL, authenticating with ACTIONS_ID_TOKEN_REQUEST_TOKEN, both injected only when permissions: id-token: write is set.
9. The issuer is https://token.actions.githubusercontent.com. The claim that matters is sub, whose shape is one of:

repo:OWNER/REPO:ref:refs/heads/main
repo:OWNER/REPO:environment:production
repo:OWNER/REPO:pull_request

10. On AWS the exchange is sts:AssumeRoleWithWebIdentity against an IAM OIDC identity provider, producing credentials of at most one hour by default. On Azure it is a federated credential on an app registration. On Google Cloud it is Workload Identity Federation.
11. The critical misconfiguration is a trust policy that checks the issuer but not the sub claim, or checks it with a wildcard. That configuration lets any repository on GitHub assume your role. Always match the full sub, including the ref or environment.
12. Third-party action supply chain risk, with a real case. In March 2025 the action tj-actions/changed-files was compromised: its version tags were retargeted to a malicious commit that read secrets out of runner memory and printed them into the workflow log.

13. The mechanism deserves attention: nothing was pushed to your repository. A tag in someone else’s repository was moved. uses: owner/action@v4 resolves that tag at run time, on every run, with no record of what it pointed at yesterday.
14. Pinning to a full 40-character commit SHA removes that entire class of attack, because a SHA names immutable content. Moving a tag cannot change what a SHA resolves to.

15. Pinning has a cost: you stop receiving security fixes automatically. Dependabot understands SHA pins with a trailing version comment and raises pull requests to move them, which restores updates while keeping the review step.
16. Earlier incidents in the same family, worth knowing as precedent: the Codecov bash uploader compromise in April 2021, where a modified script exfiltrated environment variables from customers’ CI jobs, and the SolarWinds Orion build system compromise disclosed in December 2020.
17. Other controls that matter: run untrusted work on ephemeral, isolated runners; never use self-hosted runners on public repositories; set permissions: {} and add back only what a job needs; and enable secret scanning with push protection so a committed key is rejected at push time.
18. Observing tools: gh secret list, gh api repos/OWNER/REPO/actions/permissions, gh api repos/OWNER/REPO/actions/permissions/selected-actions, and a repository policy allowing only actions from verified creators or a pinned allow list.

WORDS41.13.6 remember these#

1. Secret — a value the pipeline needs and must not reveal — encrypted at rest, injected as an environment variable for one job.
2. Masking — blacking out secret values in logs — exact string replacement by the log processor, defeated by any encoding.
3. ::add-mask:: — register a value for masking at runtime — a workflow command for secrets derived inside a job.
4. OIDC federation — proving identity instead of holding a key — a signed JWT exchanged for short-lived cloud credentials.
5. sub claim — which repository, branch or environment is asking — the field the cloud trust policy must match exactly.
6. id-token: write — permission to request an identity token — required before the OIDC endpoint variables are injected.
7. SHA pinning — naming an action by immutable content — uses: owner/act@<40 hex>, immune to tag retargeting.
8. Supply chain attack — compromising something you depend on — as in the March 2025 tj-actions/changed-files incident, CVE-2025-30066.

41.14 Reading a failed run like an engineer#

PLAIN41.14.1 in simple words#

1. A run went red. Most people open it, read the last line, and guess.
2. The last line is almost never the fault. It is usually the summary of the fault, several steps downstream.
3. Do this instead, in order.
4. Find which step failed. Not which job, not which line. The step.
5. Read that step from its beginning, not from its end.
6. The real message is usually in the first few lines after the step starts, and everything below it is consequence.
7. Ask one question before anything else: does this fail on my machine too?
8. If it fails locally, this is a normal bug and continuous integration did its job. Fix it and stop reading.
9. If it passes locally and fails in the pipeline, you have a different problem, and guessing will waste an hour.
10. Then the difference between the two environments is the fault, and there is a short list of things that difference can be.
11. Work through that list in order rather than changing things at random.
12. And if you still cannot see it, re-run with debug output switched on, which prints what the machine was actually doing.

PLAIN41.14.2 a picture in your head#

1. A car will not start in your driveway but starts fine at the garage.
2. A bad mechanic replaces parts until it works.
3. A good mechanic writes down every difference between the driveway and the garage: colder, on a slope, different fuel, different battery charge.
4. Then tests the differences one at a time.
5. The car is the same. The environment is not. The fault lives in the gap.

Where this comparison breaks: you cannot drive a build to the garage, but you can bring the garage to the build. A pipeline runs a published machine image, so you can start the identical image locally, which is a luxury no mechanic has.

PLAIN41.14.3 a worked example#

1. Getting the logs without opening a browser:

# which runs failed recently
gh run list --status failure --limit 5

# the whole log, all jobs, all steps
gh run view 11947722301 --log

# only the failing steps, which is what you want first
gh run view 11947722301 --log-failed

# one job's log
gh run view --job 33417290011 --log

2. A real failure the reader would meet, coming from macOS locally and Ubuntu in the pipeline:

Run npm test
> jest
 FAIL  src/Cart/index.test.ts
  Cannot find module './cartTotal' from 'src/Cart/index.ts'

3. The file on disk is src/Cart/CartTotal.ts and the import says ./cartTotal.
4. On macOS this works, because the default file system is case-insensitive. On the Linux runner it does not, because ext4 is case-sensitive.
5. Nothing is wrong with the test, the runner, or the cache. The two machines disagree about whether two file names are the same name.
6. Confirming it without guessing:

git ls-files src/Cart | head
# src/Cart/CartTotal.ts     <- what git actually stores
grep -rn "cartTotal" src/Cart/index.ts

7. Git stores the name it was given. Your local file system hid the mismatch. The runner did not.

PLAIN41.14.4 what is really happening inside#

1. Debug output is not on by default because it is enormous and it can leak information.
2. Two switches turn it on, both set as repository or environment variables, or as secrets, with the value true.
3. ACTIONS_STEP_DEBUG makes each step print its internal decisions.
4. ACTIONS_RUNNER_DEBUG makes the runner agent itself print how it set up the job, which is what you need when the failure happens before your code runs.
5. With gh, gh run rerun 11947722301 --debug sets these for one re-run, which is safer than leaving them on.
6. Reproducing locally has three levels, in increasing fidelity.
7. Level one: run the same command with the same environment variable, since many tools change behaviour when CI=true is set.
8. Level two: run the same command inside the same container image the pipeline uses, so the operating system and installed versions match.
9. Level three: run the whole workflow locally with a tool such as act, which interprets the workflow file and runs the jobs in containers. It is an approximation, not the real service, and it will differ on some actions.
10. One more trick for the truly stuck: a step that opens an interactive session on the runner, so you can look around while the job is paused. Use it on private repositories only, and never on a job holding production secrets.

TECHNICAL41.14.5 the engineer’s version#

1. Log structure. A run contains jobs; a job contains steps; each step is a log group. The interface collapses groups, so the failing step may be closed and invisible until you expand it. --log-failed filters to failing steps and is the correct first command.
2. Read the step from the top. A stack trace prints the deepest frame first in some languages and last in others, and the actual message is at the boundary between your code and the library.
3. Exit code 1 is a generic failure. Some codes are informative: 127 is command not found, 126 is found but not executable, 137 is SIGKILL which on a runner almost always means the out-of-memory killer, and 143 is SIGTERM which usually means a cancellation or a timeout.
4. Ordered checklist for a job that fails only in CI. Work down it; do not skip.

5. Rows 5, 6, 9 and 12 are the four that catch most people, and all four have the same root cause: your working copy contains state that the repository does not, or your machine normalizes something the runner does not.
6. Case sensitivity in detail. macOS APFS is case-insensitive but case-preserving by default; Linux ext4 and xfs are case-sensitive; Windows NTFS is case-insensitive. Git stores the exact name. A rename that only changes case may not even be recorded unless you use git mv explicitly or set core.ignorecase false.
7. Time and locale. Hosted runners run in UTC with a minimal locale. A test that formats a date, sorts strings, or parses a decimal comma will behave differently from a laptop set to Asia/Kolkata. Pin TZ and LC_ALL explicitly in the workflow rather than hoping.
8. Diagnostic step worth adding permanently to a slow-to-debug pipeline:

      - name: Environment fingerprint
        if: always()
        run: |
          uname -a
          node -v || true
          python3 -V || true
          echo "TZ=$TZ LANG=$LANG"
          df -h /
          free -m || vm_stat || true
          git rev-parse HEAD

9. Always upload evidence on failure. A failing run that kept no test report, no screenshot and no server log costs a second run to learn anything:

      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: failure-evidence-${{ github.run_id }}
          path: |
            reports/
            logs/
            screenshots/

10. if: failure() and if: always() are the two conditions that matter here. Without one of them the upload step is skipped, because a failed previous step stops the job by default.
11. Retention and download: artifacts are downloadable with gh run download 11947722301, and logs with gh run view --log > run.txt for grepping.
12. Debug variables: ACTIONS_STEP_DEBUG and ACTIONS_RUNNER_DEBUG, both set to true, as repository variables or secrets. gh run rerun ID --debug enables them for one attempt. Debug logs are larger and are retained the same way as normal logs.
13. Local reproduction with act from the nektos/act project runs workflows in Docker containers. It approximates hosted runners; differences in preinstalled tooling, services and some actions are expected. Treat a green act run as encouraging, not as verification.
14. GitLab’s equivalents: the job log with CI_DEBUG_TRACE: "true" for shell tracing, artifacts with when: always, and gitlab-runner exec for local execution. Jenkins exposes the console output plus the pipeline steps view.
15. One caution on CI_DEBUG_TRACE and on set -x: shell tracing prints every expanded command, and every expanded command may contain a secret. Never enable shell tracing on a job that holds credentials.

WORDS41.14.6 remember these#

1. Step log — the output of one command in a job — a collapsible log group, filterable with gh run view --log-failed.
2. ACTIONS_STEP_DEBUG — extra detail about each step — a variable set to true, enabling verbose action and expression logging.
3. ACTIONS_RUNNER_DEBUG — extra detail about job setup — runner-agent level logging, for failures before your code runs.
4. Exit code 137 — the process was killed — SIGKILL, on a runner almost always the out-of-memory killer.
5. Case sensitivity — whether two names are the same name — differs between APFS, NTFS and ext4; Git stores the exact stored name.
6. if: always() — run this step even after a failure — the condition that makes failure evidence get collected.
7. act — run workflows on your own machine — a local interpreter of workflow files using Docker, approximate rather than exact.

41.15 The wider landscape#

PLAIN41.15.1 in simple words#

1. Every idea in this chapter exists in every tool. Only the words change.
2. GitHub Actions lives inside GitHub. You write YAML in your repository and the machines are supplied for you.
3. GitLab CI lives inside GitLab and works the same way, except GitLab is also the issue tracker, the registry and the deployment tool, all one product.
4. Jenkins is a program you install on your own server. It is older than the others, it does anything, and you maintain all of it yourself.
5. CircleCI is a service that only does pipelines. It is not your code host, so you connect it to one.
6. Buildkite splits the job in an unusual way: they run the web interface and the queue, you run the machines.
7. Argo CD does something different from all of the above. It does not build. It watches a repository describing what should be running, and makes the real system match it.
8. That last idea has a name: GitOps.
9. In GitOps, the repository is the statement of what should be true, and a program inside your system continuously fixes any difference.
10. Nobody deploys by pushing. Everybody deploys by changing a file and merging it.
11. The difference matters most for keys. If nothing pushes into your system, your pipeline never needs credentials to it.

PLAIN41.15.2 a picture in your head#

1. Two ways to keep a garden the way you want it.
2. Push: you walk in with tools whenever you decide, and change things. You need a key to the gate, and so does anyone helping you.
3. Pull: you pin a drawing of the intended garden on the gate. A gardener who already lives inside reads the drawing every few minutes and fixes anything that differs.
4. Nobody outside needs a key. The drawing is public and versioned. Every change is an edit to the drawing.
5. If a storm knocks a plant over, the gardener notices and rights it, without anyone asking.

Where this comparison breaks: a gardener can only fix what the drawing describes. GitOps only reconciles resources under its control, so anything created by hand outside that scope drifts unnoticed. And some changes, like data migrations, are not describable as a desired end state at all.

PLAIN41.15.3 a worked example#

1. The same idea, “run the tests”, in four configuration languages.

# GitHub Actions: .github/workflows/ci.yml
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm test

# GitLab CI: .gitlab-ci.yml
test:
  image: node:22
  script:
    - npm ci
    - npm test

// Jenkins: Jenkinsfile
pipeline {
  agent { docker { image 'node:22' } }
  stages {
    stage('test') {
      steps { sh 'npm ci && npm test' }
    }
  }
}

# CircleCI: .circleci/config.yml
version: 2.1
jobs:
  test:
    docker: [{ image: cimg/node:22.0 }]
    steps:
      - checkout
      - run: npm ci && npm test

2. Four files, one idea: check out the code, get a Node environment, run two commands.
3. The differences that actually matter are not syntax. They are who owns the machines, who owns the upgrades, and what happens when you need something unusual.

PLAIN41.15.4 what is really happening inside#

1. Push deployment and pull deployment are the real architectural fork, and every tool sits on one side of it.
2. In push deployment, the pipeline holds credentials to production and reaches in to change it.
3. That is simple, immediate, and means a compromised pipeline is a compromised production system.
4. In pull deployment, an agent runs inside production, reads a repository, and applies the difference.
5. Nothing outside holds production credentials. The pipeline’s only job is to update a file in a repository.
6. The agent also loops forever, so it repairs drift: someone changing something by hand is silently undone within minutes.
7. That loop is the part people underestimate. It converts deployment from an event into a continuously enforced statement.
8. The cost is indirection. You merge a change and then wait for a program you are not watching to notice it, which is harder to debug than a command that either worked or did not.

TECHNICAL41.15.5 the engineer’s version#

1. Comparison on the three axes that decide the choice.

2. Jenkins. Started as Hudson by Kohsuke Kawaguchi at Sun Microsystems in 2004, first released 2005, renamed Jenkins after a community vote concluded on 29 January 2011. Jenkins 2.0, released in April 2016, introduced Pipeline as code with the Jenkinsfile in a Groovy-based domain-specific language.
3. Jenkins’s strength and weakness are the same fact: well over a thousand plugins. Anything is possible, and the plugin set is your responsibility to patch, test and upgrade forever.
4. GitLab CI. Began in 2012 as a separate application and was merged into GitLab in version 8.0 in September 2015. Uses .gitlab-ci.yml with stages, and a separate gitlab-runner binary with pluggable executors.
5. CircleCI. Launched 2011, config in .circleci/config.yml, with reusable packages called orbs introduced in November 2018. CircleCI disclosed a security incident in January 2023 and instructed customers to rotate all secrets stored in the platform, which is a useful reminder that a hosted CI provider is part of your trust boundary.
6. Buildkite. From Melbourne, Australia, starting in 2013. The distinguishing model is hybrid: Buildkite hosts the control plane and interface, and you run the agents on your own machines, so source code and secrets stay on your infrastructure.
7. Argo CD. Created at Applatix, which Intuit acquired in January 2018. Argo became a CNCF project and graduated, together with Flux, in December 2022.
8. GitOps as a term was coined by Alexis Richardson of Weaveworks in 2017. The OpenGitOps project states four principles: the desired state is declarative; it is versioned and immutable; agents pull it automatically; and agents continuously reconcile actual state towards it.
9. Flux is the other main implementation, also CNCF-graduated. Weaveworks, the company that created Flux and coined the term, ceased trading in early 2024, and the project continued under the foundation. That separation of project from vendor is exactly why foundation governance matters.
10. Others worth recognizing by name: Travis CI, which popularized in-repository configuration from 2011; TeamCity from JetBrains, 2006; Bamboo from Atlassian; Azure Pipelines; AWS CodeBuild and CodePipeline; Google Cloud Build; Tekton, a Kubernetes-native pipeline engine under the CD Foundation; and Concourse, which models everything as resources and tasks.
11. The push versus pull distinction, in security terms:

12. Combining them is the common modern arrangement: continuous integration builds and tests and publishes an image, then updates an image tag in a configuration repository; a cluster agent notices and rolls it out. The pipeline never holds a cluster credential.
13. Portability, honestly stated: pipeline configuration is not portable between these tools, and every migration is a rewrite. What is portable is the shell script your steps call. Teams that keep the real work in scripts and use the YAML only as an orchestrator migrate in days rather than months.
14. Where experts disagree: self-hosted Jenkins for control, or a hosted service for speed. Control genuinely matters in regulated environments and on unusual hardware. Against it, maintaining Jenkins is a permanent part-time job nobody plans for. The deciding factor is whether you have staff to own it.
15. Observing tools across all of them: gh run, glab ci, the Jenkins api/json endpoint per build, circleci config validate, buildkite-agent logs, and argocd app get plus argocd app diff for reconciliation state.

WORDS41.15.6 remember these#

1. GitOps — the repository states what should be running — declarative desired state, pulled and continuously reconciled by an in-cluster agent.
2. Reconciliation loop — keep fixing the difference forever — the controller pattern comparing desired state to actual state on a timer.
3. Drift — reality quietly diverging from the file — manual changes to a system under declarative control, reverted on the next reconcile.
4. Push deployment — the pipeline reaches into production — outbound deploys requiring production credentials in the CI system.
5. Pull deployment — production reaches out for its own config — an agent inside the boundary, so no inbound credentials exist.
6. Orb — CircleCI’s reusable configuration package — a versioned bundle of jobs, commands and executors, introduced in 2018.
7. Jenkinsfile — Jenkins pipeline as code — a Groovy-based declarative or scripted pipeline definition stored in the repository.

41.98 Common wrong ideas#

1. Wrong: continuous integration is a tool you install. Right: it is a habit of merging to the mainline at least daily; the tool only enforces the habit, and a team merging quarterly with Jenkins running is not doing it.
2. Wrong: the green tick means this code is good. Right: it means the checks somebody wrote passed on some commit. It says nothing about the tests that were never written, and nothing about the code you are looking at now.
3. Wrong: a passing branch means the head commit passed. Right: a run is bound to one head_sha. A branch is a moving label, so a branch-level tick may belong to a commit that is no longer the head.
4. Wrong: CD means continuous deployment. Right: it usually means continuous delivery, which is “always releasable, released by a human decision”. Continuous deployment removes the human. The two are different practices with the same initials.
5. Wrong: pull_request_target is just pull_request with more permissions. Right: it runs your trusted workflow file with your secrets, so checking out the pull request head and executing anything from it hands a stranger your credentials.
6. Wrong: mergeStateStatus: BLOCKED means there is a merge conflict. Right: it means the branches merge cleanly and a protection rule is unmet. The conflict value is DIRTY.
7. Wrong: UNKNOWN means something is wrong. Right: it means GitHub has not finished computing mergeability. Re-query after a short delay and conclude nothing from the first answer.
8. Wrong: a skipped check is the same as a passed check. Right: a job skipped by an if: condition reports success, while a workflow skipped by a path filter never reports at all and blocks the merge indefinitely.
9. Wrong: pinning an action to @v4 is safe because it is a version. Right: a tag is a movable pointer in someone else’s repository. Only a full commit SHA is immutable, as the March 2025 tj-actions/changed-files compromise showed.
10. Wrong: re-running until it goes green is a fix. Right: it converts a flaky test into a permanent lie, and once people stop believing red, the entire pipeline stops carrying information.

41.99 Chapter summary in 20 lines#

1. Continuous integration is a practice, not a product: merge to the mainline at least daily, verify every merge automatically, and fix a broken mainline before starting new work.
2. The term came from Grady Booch in 1991, became an Extreme Programming practice in 1999, and got its working definition from Fowler and Foemmel in
3. Continuous delivery means every change is always releasable; continuous deployment means every change that passes goes live without a human. Same initials, different practices.
4. A runner is a machine that claims a job, checks out one exact commit, runs your steps, and reports an exit code. Hosted runners are destroyed after each job, which is why “it works on my laptop” fails there.
5. The workflow file lives in the repository, so the pipeline is versioned, reviewable and bisectable like any other code, and a token needs the workflow scope to change it.
6. Triggers decide what starts a run: push, pull_request, schedule, workflow_dispatch, repository_dispatch, tags and releases, narrowed by branch and path filters.
7. pull_request runs a fork’s code with no secrets and a read-only token. pull_request_target runs your file with full secrets, so executing the pull request’s code under it is the pwn request vulnerability.
8. The central idea of the chapter: a run is bound to one commit SHA, never to a branch. A branch is a moving pointer, so “CI passed on main” is a statement about whichever commit main pointed at then.
9. The correct verification is four steps: resolve the head SHA from the server, list runs filtered by that SHA, read the specific run by ID, and confirm the run’s head_sha matches.
10. A branch-level tick can lie in at least six ways: older commit, merge result rather than head, skipped rather than passed, posted by another app, overwritten by a re-run, or attached to a merge commit that no longer exists.
11. Two APIs report results on a commit: the older commit status API with four states, and the newer checks API with check suites, check runs, timings and line annotations.
12. Conclusions are success, failure, neutral, cancelled, skipped, timed_out, action_required, stale and startup_failure, and a required check that never arrives stays Expected and blocks forever.
13. Merge queues solve the semantic conflict: two changes that each pass alone and break together. The queue builds the prospective merged state, tests it, and only then merges, serializing the landing order.
14. Speculative execution tests several queued entries at once assuming those ahead succeed, which is fast when all pass and wasteful when one fails. GitHub’s merge queue became generally available on 12 July 2023.
15. mergeStateStatus has eight values. CLEAN and HAS_HOOKS mean go, BEHIND means update, DIRTY means real conflict, DRAFT means you marked it, UNSTABLE means a non-required check failed.
16. BLOCKED means a protection rule is unmet, not a conflict; UNKNOWN means GitHub is still computing mergeability, so re-query and conclude nothing.
17. A good pipeline runs cheapest first: lint, build, unit tests, integration tests, security scanning, artifact build, staging deploy, smoke tests, production, with the artifact built once and reused.
18. Deployment strategies trade downtime against cost: recreate, rolling, blue-green, canary, and feature flags, which separate deploying code from releasing it. Database migrations make rollback hard, so expand and contract.
19. Secrets are injected per job and masked by string matching only; forks get none; OIDC federation replaces long-lived cloud keys with one-hour credentials; and third-party actions must be pinned to a full commit SHA.
20. Read a failure by finding the failing step, reading it from the top, reproducing locally, and working an ordered checklist of environment differences before changing anything at random.