A Check You Have Only Ever Seen Pass Is Not a Check

Two of my CI guards had never run. Not once, not on any commit, not since the day they were written.

They existed. They were in the pipeline file. They had a name, a purpose, and a comment explaining what they protected. Every push produced a green tick on the commit list.

The step they lived in died before executing a single line.

What they were protecting

Some background, because it explains why this mattered more than a broken build normally does.

The site serves a demo page per customer, and each demo needs an image file to exist at a known path. A missing file means the top of the page is broken. Those demo links go out in cold outreach emails, so the first time a prospect ever sees my work is also the exact moment the breakage would show.

That is the failure I wrote the guards for. Check every expected file is present, fail the build if one is not.

Good instinct. Zero execution.

Two characters of YAML

The steps were written as folded scalars. In YAML, >- means "fold this block," and folding turns every newline into a space:

- >-
  python3 - <<'PY'
  import os, sys
  ...
  PY

Fold that and the whole heredoc collapses onto one line. Bash never finds a PY alone on a line, so it reads to the end of input looking for one and gives up:

here-document at line 190 delimited by end-of-file (wanted `PY')

Literal style, |, preserves the newlines and the same script runs fine. The neighboring steps in the file were folded too, deliberately, because they were semicolon-separated shell one-liners that genuinely do not care about newlines.

So the distinction is not "which style looks nicer." It is whether newlines carry meaning in the thing you are embedding. A heredoc terminator carries meaning. So does Python indentation, which folding would have destroyed even if the terminator had survived.

The error cited a line that does not exist

Worth a paragraph on its own, because it cost me time.

The error says line 190. The pipeline file is nowhere near 190 lines long.

That number refers to the script the runner generated and executed, not to the YAML I wrote. CI systems assemble your steps into a shell script with their own preamble, variable exports and traps, then run that. Error messages come from the assembled artifact.

When a CI error quotes a line number that cannot exist in the file in front of you, stop reading the source. Go and look at what the runner actually produced. Most systems will show you, and it turns a confusing message into an obvious one.

The part that is actually about guards

Here is what makes this worse than an ordinary broken build.

A broken feature announces itself. Someone clicks the thing, the thing does not work, you hear about it. The feedback loop is built into the fact that people use it.

A broken check announces nothing. It fails in the direction of silence. And while it is failing, it is doing something worse than nothing: it is upgrading an assumption into a certainty. I was not merely unprotected, I believed I was protected, and I made decisions on that basis. I sent demo links without spot-checking them, because a guard existed for that.

A broken feature costs you the feature. A broken check costs you the thing the check was watching, plus your confidence that it was safe.

Every push since the guards were added had been red on the real pipeline. Nobody noticed, because the site deployed correctly every single time and the commit list showed a green tick from the deploy integration sitting right next to the red one. The signal that something was wrong arrived as email, and email about a repository that visibly works gets filed as noise.

Watch your check fail, once

The fix is not better YAML review. It is a thirty-second ritual I now apply to every new check.

Before you trust a guard, break the thing it guards and watch it go red.

Delete one of the files. Push. Confirm the pipeline fails, and confirm it fails with the message you wrote. Then put the file back and watch it go green. Now you have seen both states, which is the only way to know the check discriminates between them at all.

A check you have only ever seen pass is indistinguishable from a check that always passes, and you cannot tell which one you have by reading it.

I did the same thing later with a database schema check, deliberately running it against a known bug first to watch it catch it. Same reasoning, and it is the cheapest quality step I know.

Two smaller rules that came out of this

Make a scanner fail when it finds nothing. My file check now refuses to pass if it inspects fewer files than it expects. A scanner whose pattern stops matching reports "all good" forever, which is the same failure in a subtler outfit.

Know which pipeline you are looking at. If anything else posts commit status to your repository, a deploy bot for example, then your CI result is the other one. Most eyes read "is there a green tick," not "are all the ticks green." That is a reporting hazard, not a curiosity, and it is what hid this for weeks.

The guards work now. I know they work, because I have watched them fail.