PROJECT / SAMPLE
The demo worked. Production didn't.
An unstable AI prototype was rebuilt around bounded model behavior, explicit state, evaluations, and observable deterministic code.
- Date
- 2026-06
- Type
- automation / software
- Status
- fictional fixture
- Published
- prototype only
Problem
A useful prototype began failing as real traffic and edge cases arrived. Model calls were uncontrolled, state lived inside prompt history, and expensive retries obscured the actual failure modes.
Constraints
- Customer-facing behavior had to remain available.
- The product's useful model capability could not be removed.
- The team needed evidence before changing prompts or providers.
Observed
- A single request could trigger an unbounded chain of model calls.
- Prompt changes had no repeatable evaluation gate.
- Retries mixed transient transport failures with permanent task failures.
Work
- Moved durable state out of prompt history.
- Bounded model calls by task and failure category.
- Created a representative evaluation set and release gate.
- Added traces across model, tool, and deterministic steps.
- Replaced repeated inference with deterministic validation where possible.
Decisions
- Use models for judgment, not for state management.
- Evaluate behavior on real task shapes before optimizing prompts.
- Make uncertainty explicit at the product boundary.
Result / SAMPLE
The critical path became faster, cheaper, and measurable enough for the team to change it safely.
- p95 latency: 18.4s → 3.1s
- inference spend: −71%
- production task success: 83% → 98.7%
What we did not do
We did not optimize the prototype prompt indefinitely. The main defects were software boundaries, not phrasing.
Technical index
LLM APIs / evaluation harnesses / distributed tracing / state machines