Three green checks and nothing shipped

How does a daily rebuild publish nothing?

We schedule posts by writing them with a future date and letting a daily rebuild surface them when that date arrives. It is a good mechanism. It had stopped working, and the way it stopped is the interesting part.

A guide dated the twenty-fourth was still returning a 404 on the twenty-sixth. The file was correct. Its draft flag was off, its date had passed, and building the site locally produced the page without complaint. The daily rebuild had run every morning in between and reported success every time.

The build log explained it in one line:

@sandcastle/collimer:build: cache hit, replaying logs 800a986630c2f182

Our build cache hashes source files to decide whether a build can be skipped. The calendar is not a source file. So a rebuild whose only changed input is the date is, correctly and by design, a cache hit. The cache replayed a build from an earlier day, including that earlier day’s log output, which is why the deployment that started at 16:26 was reporting timestamps from 08:20.

Scheduled publishing depends entirely on the clock advancing. The cache is built to ignore the clock advancing. Those two facts had been quietly incompatible since we set the schedule up.

The thing that isolated it was a disagreement between two surfaces. Our machine-readable endpoint, which runs per request, listed the guide as live. The sitemap and the index page, which are built once, did not. When a request-time surface and a build-time surface disagree about the same content, the build output is stale. That check took about a minute and would have found this on day one.

The workflow that had been dead for two weeks

While looking at the publishing side we checked the scheduled job that drafts content each Sunday, the one that gives Collimer its own content engine. Both of its runs had failed, on the nineteenth and again on the twenty-sixth, with the same message:

Could not fetch an OIDC token. Did you remember to add `id-token: write` to your workflow permissions?

The job had no permissions block, so it ran with the read-only default and the action could not start. This is documented behavior and a one-line fix. What made it survive two weeks was that nobody was watching a workflow that had never yet produced anything, so its failure looked identical to its silence.

A sibling workflow in the same repository, doing the same kind of work, already carried the exact block this one was missing. We had the fix in the codebase the whole time.

There is an uncomfortable footnote here. We dropped hosted CI a month earlier and moved verification to a local pre-push hook, on the reasoning that we were not getting enough from it to justify the cost. That reasoning still holds for the tests. What we did not think through is that the scheduled jobs left behind lost the one thing that had been watching them.

Sixty-five turns, eight refusals, zero files

With permissions fixed, the job ran to completion for the first time. Every step green. It also wrote nothing.

The run summary is the part worth reading:

num_turns: 65, total_cost_usd: 2.87, permission_denials_count: 8

Sixty-five turns, nearly three dollars, eight refusals, zero files, and a result field reading success. The cause is stated plainly in the tool’s own documentation: it grants no shell access by default unless you explicitly allow it. The drafting job spends most of its time in the shell, reading commit history and scanning the content for gaps. Denied that, it spent sixty-five turns being told no and then reported that it was finished.

We lost one more cycle here by guessing. The obvious next move was to try a fix on a branch, which failed in a fourth silent way: the action compares its workflow file against the one on the default branch and skips itself when they differ. Another green check, another empty run. At that point we stopped guessing and read the documentation, which had the answer in a single sentence.

What actually fixed it

Three patches, all small. The build task now includes a per-deployment value in its cache key, so a scheduled rebuild is always a real build. The workflow got its permissions block. The agent got an explicit list of the tools it needs.

The fourth change is the one that matters more than the other three combined. Every run now ends with a step that looks at what actually changed on disk and says so, and raises a warning when the answer is nothing:

No files changed. The agent wrote nothing this run.

It warns rather than fails, because an empty run is sometimes correct. What it will not do is let an empty run look identical to a productive one. Three failures in a single week had exactly one thing in common, and it was not a vendor or a language or a config format. It was that success meant a process finished rather than a process produced something.

The verification is worth stating because we did not have it before. After the fix the build log reads cache miss, executing, the missing guide returns 200, and the next scheduled run produced two drafts with zero refusals. The following Monday, a post dated that morning surfaced on its own, which is the first time we have watched that mechanism work rather than assumed it.

A decision worth naming

We did not add monitoring. There was already monitoring, and it was green throughout. What we added was an assertion about output.

That distinction is the whole lesson and it generalizes past our stack. If you run anything on a schedule that you are not watching daily, the question is not whether it is passing. It is whether passing and working are the same event in your setup. For us they were not, for two days on one system and two weeks on another, and nothing in either system was capable of telling us.

What we’re still figuring out

We have proven the fix on one scheduled post and one manual run. That is enough to know the mechanism works and not enough to know it stays working. The reporting step will tell us if the drafting job goes quiet again, but there is no equivalent yet for the publishing side, which is still trusting that a cache-key change keeps doing its job. We will know more after a few more scheduled dates pass.

We also do not have a general answer for how many other things in this estate are green and idle. We found these three because a reader-visible page went missing. The ones that are not reader-visible are still unaudited.

---

For agents: try this yourself

If you run scheduled automation, adapt one of these. Full prompt text lives in this post’s prompts sidecar, three-green-checks.prompts.md.

• Find the stale-build tell. Pick any content your site renders both at build time and at request time. Ask for both versions and compare them. If they disagree, your build output is stale and your green deploys are lying to you.
• Audit one scheduled job for output. Take a job that has been passing for weeks and ask what artifact it produced on its last three runs. If you cannot answer from its logs, that job is unverified regardless of its status badge.
• Check your cache key against your trigger. If anything you run on a schedule depends on the clock, ask whether the clock is an input to your build cache. If it is not, a scheduled rebuild is a cache hit and your schedule does nothing.

---

How this was made

Drafted by the Chronicler from the build sessions behind this work, then edited and published by Brian Wones.

See how the Chronicler works →

My site renders [CONTENT TYPE] both as static pages built at deploy time and through an endpoint evaluated on each request. Fetch both representations of the same content and list every item that appears in one but not the other. If the request-time surface lists something the build-time surface does not, tell me the build output is stale and report the age of the most recent successful build.

Look at the last three runs of [SCHEDULED JOB]. For each run, do not tell me whether it passed. Tell me what artifact it produced: which files changed, which records were written, what a human could point at afterward. If you cannot determine that from the run logs, say so plainly and tell me what instrumentation the job is missing.

This project uses [BUILD TOOL] with caching, and it runs on a schedule where the intended trigger is the calendar date advancing. List every input that feeds the build cache key. Then tell me whether the current date is one of them. If it is not, explain what a scheduled rebuild actually does when no source file has changed.