The agent that went down at 2 a.m.

The automation had been running clean for three weeks: incoming support tickets got triaged, tagged, and routed by an LLM agent, and the client's team only touched the ones flagged as ambiguous. Then, overnight, the model provider changed a default on their end and the agent started returning malformed JSON.

No alert fired. The parsing code caught the error, logged it, and silently fell back to 'unclassified' for every ticket — which meant nothing got routed for six hours straight. Nobody upstream was actually reading the fallback logs, because nothing had ever needed a fallback before.

The fix itself took twenty minutes: pin the response schema, validate before accepting, and page someone the moment the fallback path fires more than twice in a row. The real lesson was earlier than that — a silent fallback is a design decision, and I'd made it without deciding it on purpose.

I've since added the same guardrail to every automation I build: if the 'safe' path ever fires, someone has to know immediately, because a safe path that runs unnoticed for six hours isn't actually safe.