Case studies · Retail

The troubleshooting is done before anyone logs in

How a load failure became a review rather than a firefight.

The nightly load fails at quarter past two. At six, an agent starts, reads the logs, finds the real cause and determines where the fault belongs. At eight, a developer arrives to either a finished fix to review or a report pointing to the right owner. If all went well, nothing has happened.

Client: International retail company, anonymised
Industry: Retail
Engagement: AI-driven automation in data platform maintenance
Scope: Daily run before the working day starts, deterministic failure detection
Roles: Senior data engineer with an AI focus
Tech stack: BigQuery, dbt, git-based repo
Delivery: Agent for troubleshooting nightly loads, running daily
Method: RAID

The challenge

The real price is someone being pulled away from their work

Ask a development team what a broken nightly run costs and they will answer in hours of troubleshooting, and that is an underestimate. The real price is someone being pulled away from what they were doing. The work that was planned is paused and therefore delayed.

On top of that, the task depends on individuals. Reading a dbt log and understanding what broke requires knowledge of the code, the platform and the business domain. Rarely more than a couple of people have it, and they are the same people needed elsewhere. Bounded, recurring and unpopular, which makes it a good problem to automate.

The solution

An ordinary script decides whether something failed

The agent runs every morning, after the nightly run and before the working day. It does not decide for itself whether the load failed; an ordinary Python script does that by querying the execution environment and returning a structured answer.

The difference is decisive, because the most common reason automation of this kind does not work is a language model deciding something looks broken. Six outcomes are defined and five of them are failures. Only a genuinely successful run gives an OK; the rest trigger troubleshooting.

Delivery

Guardrails that stop the agent from cheating

Most of the work went into stopping the agent from cheating, because the easiest route to a green test is almost never to fix the fault but to hide it.

No code change is therefore proposed without the fault first being reproduced against real data and the fix proven green, and a fix may never make unexpected rows invisible to the test that caught them. Beyond that, the agent has read access, no automatic merge and no access to test or production.

Results

A review rather than a firefight

The diagnosis is done before the working day begins, and you do not have to be the person who knows the platform best to move forward. The time-consuming part, hunting for the cause of the fault, is automated.

The lessons that are reusable are also written back to the domain knowledge base that the documentation agent keeps current, so that the next person does not make the same mistake. A load failure has become a review rather than a firefight.

In brief

01
The diagnosis is ready before the working day begins, every day
02
Six defined outcomes, decided by an ordinary script rather than by a language model
03
Reusable lessons are written back to the shared domain knowledge base

Facing a similar challenge?