← All writing

Guide

How do I catch an automation that stops working without an error?

Aug 14, 2026

An automation that dies without an error looks exactly like one that's working. This is the small check that watches the silence instead, and the reproducible benchmark showing where it works and where it fails.

Watch the gap, not the error log. If an automation's output has been silent for longer than a multiple of its expected cadence, treat it as dead and say so.

Most monitoring watches for errors: an exception, a non-zero exit, a 500. But the worst failures don't error. The process just goes quiet. The logger stops writing. The nightly job silently skips. The agent loop exits clean. Nothing fires, because nothing failed — it just stopped, and you find out days later.

In my own stack, "days later" has a number. A logging pipeline once went silent for 23 days before I caught it. No exception, no failed run, no alert; it simply stopped, and nothing in the system was pointed at the silence. I'll keep that incident to its recorded shape, because the pattern matters more than the war story, and the pattern is everywhere.

Why doesn't normal monitoring catch this?

Because error monitoring subscribes to events, and a silent stop emits none. An alert rule can only react to something that happens. A stream that stops producing output generates exactly nothing to react to, so the dashboard doesn't turn red. It just stops updating, which looks a lot like a calm week.

The fix is to invert the question. Don't ask "did anything fail?" Ask "when did this last succeed, and is that gap normal?"

What does the detector actually do?

silent-failure-detector is that inverted question as code. Give it the timestamps of everything a job has emitted and the cadence you expect (how often output is supposed to arrive), and it tells you whether the stream has stopped. One tuning knob, multiplier (default 3×), trades how fast it catches a real death against how often it false-alarms on a bursty stream. It is deterministic — same inputs, same answer, no AI judgment in the loop — and read-only, no network, about 40 lines of Python with nothing outside the standard library.

For a stream that should emit daily, the default flags once the gap passes three days. That is the ceiling I wanted after my incident: silence measured in days, not weeks.

That part is easy, and I want to be blunt: anyone can write this check in an afternoon. A staleness check is not the product. The promise isn't "I wrote a check." It's "here's how I know the check works."

How do I know the check itself works?

Because I benchmarked it and published the harness. I built a 105-case labeled test set: 50 streams still alive, 55 that genuinely stopped. A deterministic grader scores the detector against those labels, with no LLM judge anywhere (no AI model grading results), so anyone re-running gets identical numbers. And I deliberately seeded a grey zone of cases right at the 2–4× threshold boundary, so the eval would show the detector's real limits instead of a flattering 100%.

The measured result: 83.8% accuracy (88 of 105). It catches 81.8% of real silent failures, with 86.0% specificity (it doesn't cry wolf on streams that are merely slow). Every one of the 17 errors sits in the threshold grey zone; on clearly-alive, clearly-dead, and empty streams it is 100% correct.

The two failure modes are exactly the inherent tradeoff of staleness detection, deciding liveness from the age of the last output. One is a false alarm on an alive-but-currently-slow stream; the other is a one-window lag on a stream that just died. Lower the multiplier to catch deaths faster; raise it to tolerate bursty cadence. No setting gives you both, and a tool that claims otherwise is lying to you.

It's not 100%, and claiming 100% would be the exact slop this whole project exists to refute. 83.8% with every error explained and bounded is the honest, useful number, and you can re-run the eval and check it yourself.

What does 83.8% not tell you?

Three limits, stated plainly. The gold set is synthetic: generated streams, not captures of live production systems, so real-world accuracy on non-synthetic streams is explicitly not claimed. The labels are honest: each case's ground truth comes from the generator's intent, set independently of the detector's rule, under a fixed seed (20260625) so the set rebuilds identically. And 105 cases is a small sample, so the benchmark reports 95% Wilson confidence intervals (an error-bar method sized for small samples) instead of bare percentages.

Everything is public: the skill and its eval harness, and the BENCHMARK.md receipt with the full confusion matrix, the complete table of hits, misses, and false alarms. Reproduce it with python3 eval/make_goldset.py && python3 eval/run_eval.py; the numbers above come from an actual run confirmed on 2026-06-25.

This detector is one small organ of a larger discipline: cannot fail silently beats cannot fail. The check doesn't stop an automation from dying. It guarantees the death can't stay invisible.

So audit your own stack for a minute. Mine hid a dead pipeline for 23 days. Which of your automations could stop tonight without a single error, and how many days would pass before anything told you?

I'm Ali — I run real life-and-work admin on AI agents, then check their work in the open. More at /about.

Published under my standards. Found an error? Tell me — corrections go in the corrections log, dated, never silent.

@online{ali2026silentfailure,
  author = {Ali},
  title  = {How do I catch an automation that stops working without an error?},
  date   = {2026-08-14},
  url    = {https://alidoes.ai/silent-failure-detector/}
}

Caught something I got wrong? Send it directly. Confirmed corrections go in the corrections log.