Audit of silent no-ops in analytics pipelines
Review pipeline success against source freshness, schema and final artifact evidence, with legitimate unchanged results kept distinct from failure.
What changed. The acceptance path checks source and artifact evidence; detection-time savings and a recurrence rate are not established here.
Stack. fail-loud contracts · artifact evidence gates · schema versioning · external falsifier
Context
An analytics job can exit successfully while a required field, source update or downstream artifact is missing. A model-written summary can make incomplete data look coherent. The audit focuses on the evidence behind the reported result rather than the process colour alone.
Task
Trace source collection, schema handling and final reporting. Distinguish an accepted empty or unchanged result from a skipped operation. Require a visible failed or unknown state when the expected evidence is unavailable.
What we built
The review separates ordinary silent failure, such as a swallowed exception, from a plausible-looking report generated from incomplete input. A missing-field fixture tests the boundary where a removed stress field could otherwise become a default zero and a passing score.
The evidence gate checks the source observation time, expected coverage, schema compatibility and final artifact content. File size, modification time and a changed hash can help diagnose freshness, but none alone proves useful work. A source can legitimately contain no new rows, while rewriting an old result can produce a new timestamp without fresh data.
Record an unchanged result as valid only when the source was actually checked and the task permits that outcome. Preserve source and artifact identifiers with the run result. Check that a deliberate source failure reaches the final status and is not replaced by a fluent summary or an empty success default.
What changed
The report distinguishes source failure, schema incompatibility, valid no-change and unverified output. This case does not establish a detection-time improvement or a period without recurrence; those claims need a dated sample and observation window.
What we would do differently
Start with a known passing fixture, a missing-field fixture and a valid unchanged-source fixture. Verify the final reporting decision as well as the detector. A classification of failure modes is useful only when the acceptance path consumes the evidence.
Related service: AI agent audit