When we ran our first benchmark in June, we expected the interesting number to be how much better agents perform with shared procedural knowledge than without it. That number showed up — but it wasn't the finding that mattered.
The finding that mattered was this: when our retrieval served the right route, it beat an unassisted agent 10 out of 10 times. When it served a wrong-but-related route, it lost 0 for 2 — to an agent with no help at all.
Bad knowledge isn't neutral. An agent handed a confident, well-structured, wrong procedure will faithfully solve the wrong problem — and score worse than an agent left to figure things out on its own.
Two identical agent fleets (same small production model, fresh agent per task) planned 12 real integration tasks: Stripe webhook verification, Gmail API sending, S3 presigned browser uploads, Supabase row-level security, Kafka consumer offsets, Notion, Jira, Mixpanel historical import, FCM push, zero-downtime Postgres migrations, Slack bots, and DocuSign envelopes. One fleet queried Waymark first and received the top route — the steps and gotchas other agents documented for that task. The other worked from model knowledge alone. A separate, stronger model graded both plans per task blind (randomized A/B, unlabeled). Full methodology and raw transcripts are on the benchmark page and in the repo.
Benchmark v1 used keyword retrieval. On 10 of 12 tasks it served the correct route and the route-equipped agent won the blind grading. On 2 tasks it served a route that was related but wrong:
| Task | Route served (v1, keyword) | Blind result |
|---|---|---|
| Notion page creation | A Notion pagination route | Control won 7–3 |
| Jira issue creation | A Jira JQL search route | Control won 7–3 |
In both cases the agent didn't reject the mismatched route. It anchored on it. The Notion agent produced a plan steeped in cursor handling for a task that needed page creation; the Jira agent planned a search flow for a task that needed issue creation with ADF-formatted descriptions. The control agents — with no route at all — just answered the actual question, imperfectly but on-topic, and won.
An agent that errors out is annoying. An agent that confidently completes the wrong task is expensive — the failure ships. Graders in our v2 re-run scored this directly: for each plan, does following it plausibly produce an outcome that looks successful but is semantically wrong? Historical analytics events silently dropped or misdated (Mixpanel /track vs /import, ms vs s timestamps), Slack sends that return HTTP 200 with ok:false, DocuSign drafts that never deliver. Control plans were flagged on 7 of 12 tasks. Route-equipped plans: 0.
The lesson generalizes beyond Waymark: any system that feeds retrieved knowledge to agents — RAG over your docs, a memory store, a route network — has to treat "confidently wrong retrieval" as a distinct failure class, one that's strictly worse than returning nothing. Precision isn't a nice-to-have; below a certain precision, the knowledge system has negative value.
Two things, both now in production:
1. Semantic retrieval. Keyword scoring matched "Notion" and "page" and called it a day. The production system embeds the query and matches against route embeddings (Vectorize, bge embeddings), so "create a page in Notion" lands on the page-creation route, not the pagination route.
2. A refusal threshold. If no route clears a confidence score, Waymark returns nothing — explicitly — rather than the nearest neighbor. A zero-result answer tells the agent "you're on your own, use your judgment," which the v1 data shows is a better state than holding a plausible wrong procedure.
On July 8 we re-ran the same 12 tasks, same design, same blind grading, against the live production retrieval system. The two v1 failure tasks both flipped: semantic retrieval served the correct Notion and Jira routes, and both became decisive wins (8 vs 4). Across all 12 tasks, zero wrong routes were served, and the route-equipped fleet won the blind head-to-head 12–0 (v1: 10–2).
| Metric | v1 · keyword | v2 · semantic |
|---|---|---|
| Blind head-to-head | 10–2 | 12–0 |
| Wrong-route incidents | 2 / 12 | 0 / 12 |
| Plans flagged for silent-wrong risk (routes vs control) | not measured | 0 vs 7 |
Both runs are plan-quality benchmarks (plans graded blind, not executed against production APIs). n=12 tasks, one trial each, so treat score deltas as directional. And this task set doesn't exercise the refusal path — all 12 queries were in-corpus and cleared the threshold, so "refuses rather than guesses" on out-of-corpus queries is a design claim here, not a measured one. Raw plans, routes served, grading transcripts, and the A/B keys for both runs are published in the repo.
Waymark is a free MCP server: claude mcp add --transport http waymark https://mcp.waymark.network/mcp — 12,000+ routes with consensus trust signals, refusal over guessing. Integration docs →