PROJECT / SAMPLE
Stop asking AI to click buttons it doesn't need to think about.
A screenshot-driven operations agent became a deterministic browser workflow, with model reasoning retained only where ambiguity required it.
- Date
- 2026-08
- Type
- automation / software
- Status
- fictional fixture
- Published
- prototype only
Problem
A business-critical workflow asked an LLM to inspect screenshots, infer interface state, and guess click targets on every step. It was slow, expensive, and unreliable in ways that were difficult to reproduce.
Constraints
- The existing workflow had to remain available during the changeover.
- Some decisions were genuinely ambiguous and still required model judgment.
- Failures needed to be replayable without repeating external side effects.
Observed
- Most model calls were spent rediscovering deterministic interface state.
- Screenshot coordinates coupled the automation to layout rather than behavior.
- The workflow had no durable action trace for diagnosis or replay.
Work
- Instrumented the existing workflow before changing behavior.
- Separated deterministic actions from reasoning decisions.
- Replaced screenshot navigation with DOM and browser APIs.
- Added replayable action traces and bounded fallback recovery.
- Kept model reasoning only for genuinely ambiguous steps.
Decisions
- Treat browser state as structured data, not pixels.
- Make every external side effect idempotent or explicitly non-replayable.
- Escalate ambiguity instead of hiding it inside repeated model calls.
Result / SAMPLE
The rebuilt path completed the same task with lower latency, far fewer tokens, and failures that could be replayed rather than guessed at.
- 61.7s → 4.8s median execution
- 92% fewer model tokens
- 99.4% completion across 1,000 replayed tasks
What we did not do
We did not replace weak navigation with a larger model. The deterministic steps did not need more intelligence; they needed less uncertainty.
Technical index
browser APIs / DOM automation / action traces / LLM routing