Audit of silent no-ops in analytics pipelines
An audit that found silent no-ops and fail-plausible reports on exit 0, then installed a fail-loud evidence gate so a missing artifact cannot look like a healthy run.
What changed. Mean time to detect a dead pipeline fell from fourteen days of manual reconciliation to about thirty seconds at run time.
Stack. fail-loud contracts · artifact evidence gates · schema versioning · external falsifier
Context
The worst incident in a distributed analytics stack is often not a 500. It is a job that exits 0, paints a green dashboard, and does no useful work. Extractors return empty lists or a disk cache from two months ago; downstream jobs treat the payload as fresh. Language-model aggregators, given a timeout or a missing field, write a fluent report anyway.
A financial-analytics team had bolted on collectors and model-written summaries. For weeks the board looked healthy. Analysts then found by hand that a key stress path was being skipped because a schema had changed and nobody had bumped the version. The studio was asked to audit the automations, name the failure classes, and put a preventive gate on the data path.
This is an engineering case about pipelines that lie. It is not a claim about portfolio returns.
Task
Read the codebase and the run logs. Classify silent failures. Install a contour that refuses to call an empty or stale artifact a success. No worker, and no model, may certify its own output.
What we built
The audit recorded eight critical holes and split them into two classes.
Class C — classical silence: try/except: pass, a returned [], a stuck cache. One producer dropped a stress-test field without raising schema_version; a validator accepted the report and labelled the book “PASS, zero risk.”
Class D — fail-plausible: a model or parser given partial data emits a coherent, well-formed fiction. A linter will not catch it.
The fix is a four-layer Fail-Loud and Traceable Evidence rule. Empty defaults are banned; a source failure must exit non-zero with context on stderr. Success is an evidence check, not a return code: artifact byte size above a floor, mtime inside a freshness window, a real diff in the store — re-confirming yesterday’s rows is not a pass. Schema changes require a version increment; soft-parsing unknown or missing fields is forbidden on critical paths. PASS is issued only by an isolated gatekeeper that checks hash and structure of the final artifact. Each step is written to a WAL keyed to source hashes.
What changed
Detection moved from a two-week manual reconciliation to a failure at the moment the job runs — on the order of thirty seconds, not fourteen days. In the following six months of this contour, unnoticed degradations of the empty-cache kind did not recur. A field-signature mismatch now aborts the run instead of minting a fictional risk score. The audit trail is a log, not a recollection.
What we would do differently
Ship the evidence gate — size, mtime, schema version — before the taxonomy lecture. Operators needed the alarm on night one. The eight-hole catalogue was useful for the report; it did not stop the next silent skip. Classification is a memo. An empty file that cannot be called green is the control.
Related service: AI agent audit