EXPERIMENT RECORD · DEMONSTRATED
AI agents fail in production for boring reasons: a rate limit, a quota ceiling, a provider hiccup. The interesting question is what happens next. In this demonstrated scenario — part of the Agent Rescue line of work, built around the WebMCP Challenge — an AI job failed at 73% completion due to an HTTP 429 condition, and the lab walked it back to 100%.
The scenario
STEP 01
The job fails at 73%
HTTP 429
An AI workload running through its task lifecycle halts at 73% completion, blocked by an HTTP 429 rate-limit response from its provider.
STEP 02
Failure identified
DIAGNOSIS
The failure condition is identified and recorded. A 429 is a recoverable quota event with a clear signature — not a bug, not a crash — and the response to it should be different.
STEP 03
Alternate provider selected
HUMAN APPROVAL
An alternate provider is selected to continue the job. The switch is approved by a human operator: the authority boundary stays explicit, and no failover happens silently.
STEP 04
Resumed from preserved progress
CHECKPOINT
The completed 73% is not discarded. Work state was preserved, so the job resumes from the checkpoint instead of restarting from zero — the difference between an inconvenience and a loss.
STEP 05
Recovery completed
100%
The workload completes. The loop — fail, identify, decide, preserve, resume, finish — is demonstrated end to end.
What the run demonstrates
- Failure detection — the system recognized a quota condition and recorded it rather than retrying blindly.
- Human-in-the-loop provider selection — the switch to an alternate provider required human approval, keeping authority explicit.
- Progress preservation — checkpointed state meant resumption, not restart.
- Resumption to completion — the recovery finished the job, which is the only outcome that counts.
Why this matters
Most agent demonstrations show the happy path: a task starts, runs, and finishes. Production is not the happy path. Real deployments hit rate limits, provider outages, degraded modes, and quota ceilings — often at the least convenient moment, deep into a long run. An agent that loses 73% of its work because a 429 arrived is not a system; it is a liability with a chat interface.
The Agent Rescue approach is to make failure boring: deterministic failure-and-recovery testing, where the failure is reproduced on purpose and the recovery path is demonstrated, recorded, and repeatable. Reliability stops being a claim and becomes something you can watch happen.
LIMITATIONS
This article documents the demonstrated scenario at the level cleared for publication. Detailed run records, timings, and configuration are released through the lab as they are cleared. A single demonstrated recovery is evidence of a working path — not a certification across all providers and conditions.
Related work: the Agent Rescue project page carries the full scenario and context — Agent Rescue / WebMCP — and the methodology article explains why we break things on purpose.