Skip to main content

Build log · collimer

Three releases in four days, and the discipline built to make that safe

·

Illustration for the build log "Three releases in four days, and the discipline built to make that safe"

Three point releases went out in four days: v1.2.0, v1.3.0, v1.4.0, each one a coherent batch rather than a padded changelog. What made that pace defensible instead of reckless shipped in the same window, not after something broke.

How do you run zero-downtime deploys as a solo or small engineering team?

Fix the deploy mechanics before the traffic makes them expensive to get wrong. Single-machine deploys moved from rolling, which carries a real downtime window, to bluegreen. The kill_timeout setting was raised from 30 seconds to 100, because the old value could never reach the 90-second graceful-shutdown period already configured for background jobs. Two settings that were supposed to cooperate had never actually been checked against each other, and a shorter outer timeout silently discards whatever grace period the inner one promises.

The branch that took five pull requests with it

Separately, deleting the team’s old shared integration branch auto-closed five open, unmerged pull requests with no warning. GitHub does not retarget a closed pull request or reopen one whose base branch is gone, so the loss was only discovered after the fact. The fix makes both integration branches permanent going forward, with a scripted, mandatory backmerge after every promotion and hotfix, so the same branch cannot quietly become the thing that took work down with it again.

Why three releases in four days is not the red flag it sounds like

“Three point releases in four days” reads as a warning sign if the releases are padding a changelog to look active. That is a fair instinct, and it does not apply here: each release shipped a distinct, coherent batch, correctness fixes in one, deploy verification and clustering in the next, zero-downtime deploys and honest alerting in the third. The branch-protection fix exists precisely because the team noticed and corrected its own process failure inside the same window, rather than after it recurred a second time. The same pace showed up in an earlier three-day build: speed and defensibility are not in tension when the process gets the same engineering attention as the feature.

The settings-nesting problem, generalized

If your deploy tooling has two timing settings that are supposed to cooperate, a kill timeout and a graceful-shutdown period, verify the actual numbers nest. A shorter outer timeout does not fail loudly. It just quietly cuts off the inner period before it finishes, and nothing about either setting in isolation tells you that. The same caution applies to any “disposable” branch that open pull requests currently point at: GitHub’s branch-delete-closes-PR behavior has no undo, so a branch stops being disposable the moment anything unmerged depends on it.

What we’re still figuring out

The branch-protection fix prevents this specific failure from recurring on these two branches. We have not audited every other branch in the repo for the same class of hidden dependency, so this is a fix to the instance we found, not a guarantee against the pattern happening somewhere else.


For agents: try this yourself

If you deploy on a fixed cadence or manage integration branches, adapt one of these. The full prompt text lives in this post’s prompts sidecar, three-releases-in-four-days.prompts.md.

  • Apply the timeout-nesting check. Find every pair of timing settings in your deploy or shutdown process that are supposed to cooperate (a kill timeout and a graceful-shutdown period, a retry window and a circuit-breaker threshold). Verify the actual configured numbers nest correctly, not just that both exist.
  • Critique your own disposable branches. List every branch in your repo you’d casually delete. For each one, check whether any open, unmerged pull request currently targets it. If so, it isn’t disposable, and deleting it will silently close those PRs with no undo.

How this was made

Drafted by the Chronicler from the build sessions behind this work, then edited and published by Brian Wones.

See how the Chronicler works →

Try this with your own agent

2 prompts you can hand to your own agent (or run by hand) to work with what this post documents. Edit the bracketed parts for your context.

Apply the timeout-nesting check

Find every pair of timing settings in this codebase that are supposed to cooperate during shutdown or deploy (a kill timeout and a graceful-shutdown grace period, a retry window and a circuit-breaker threshold, a health-check interval and a startup grace period). For each pair, verify the actual configured numbers nest correctly, meaning the outer timeout is long enough to let the inner period complete, not just that both settings exist somewhere in config.

Critique your own disposable branches

List every branch in this repository you would consider disposable or safe to delete. For each one, check whether any currently open, unmerged pull request targets it as a base branch. Report any branch where deleting it would silently close open PRs with no warning, and decide whether that branch needs to become permanent instead.

More in Collimer Build