Build log · collimer
One dependency that couldn't launch took the whole app down with it
Production rebooted every two to three minutes until someone rolled it back manually. The cause had nothing to do with any request that was actually failing: a Chrome process used for PDF rendering could not launch inside the pinned runtime image, and because it was wired as a permanent child of the root application supervisor, its failure cascaded into shutting down scans, dashboards, the API, and background jobs together.
How do you stop a failing dependency from crashing your whole Elixir app?
Isolate it. A permanent child in your supervision tree makes your dependency’s problem your outage, even when the failing dependency has nothing to do with whatever request the app is actually serving. The fix here moves the Chrome process to its own supervisor as a temporary child, checked for liveness rather than assumed present, so a Chrome that fails to launch degrades one feature instead of ending the process.
What actually cascaded
Chrome’s supervisor exhausted its restart intensity budget, the threshold OTP uses to decide a child is unrecoverable rather than just unlucky. Once that threshold was crossed, the exit propagated to the root supervisor, and a permanent child’s exit brings its entire supervision tree down with it. Scans, dashboards, the API, and Oban’s background jobs were all children of that same tree, so a browser that could not start took all of them down together. The straightforward fix reads too easy in retrospect: just isolate the risky dependency. The two and a half minutes of reboot loop before that pattern was clear were not easy in the moment.
What’s still honestly unconfirmed
The image had been bumped shortly before the incident, and a Chromium-version regression was the obvious theory. It is still exactly that: a theory. Even after the isolation fix shipped, nobody has confirmed the Chromium regression was the actual trigger. The runtime image is now pinned by digest for reproducibility, which prevents a repeat of this specific class of surprise, but it does not confirm what caused this one. Saying so plainly is more useful than implying a tidier resolution than what actually happened.
Why this generalizes past Chrome
The same fix applies to anything supervised that can fail for reasons entirely outside your own code: a headless browser, a renderer, any third-party process your app starts and waits on. Any of those belongs under its own supervisor as a temporary child, not as a permanent child of the tree that also holds everything else. The same three-day build that shipped an agent fleet’s worth of infrastructure is the kind of pace where a supervision-tree shortcut like this is easy to take and easy to forget you took.
What we’re still figuring out
The Chromium-regression theory remains unconfirmed. We isolated the failure mode without confirming the root cause, which is an honest but incomplete state to leave this in.
For agents: try this yourself
If you supervise a third-party process in an OTP application, adapt this. The full prompt text lives in this post’s prompts sidecar, one-dependency-took-the-whole-app-down.prompts.md.
- Apply the temporary-child isolation pattern. Find every third-party or external process your supervision tree treats as a permanent child. For each one, ask whether its failure should be allowed to take down anything else in the tree. If not, move it under its own supervisor as a temporary child with an explicit liveness check.
How this was made
Drafted by the Chronicler from the build sessions behind this work, then edited and published by Brian Wones.
See how the Chronicler works →Try this with your own agent
1 prompt you can hand to your own agent (or run by hand) to work with what this post documents. Edit the bracketed parts for your context.
Apply the temporary-child isolation pattern
List every third-party or external process your application's supervision tree currently treats as a permanent child (a headless browser, a renderer, any process wired to restart indefinitely as part of the main tree). For each one, ask: should this dependency's failure be allowed to take down anything else in the tree? Where the answer is no, move it under its own supervisor as a temporary child with an explicit liveness check, so a failure to launch degrades one feature instead of the whole app.