Tools were table stakes
The eval wasn't about the model.
Every agent could use the tools. The useful question was whether it kept Archibus's boring rules intact after the visible tests ran out and hidden checks started behaving more like review.
I built a small benchmark to answer one question: could a general coding agent contend with ArchiBot on real Archibus-style work? The first thing it exposed wasn't a model. It was how little the eval could prove.
This article records the 18-task campaign refreshed on May 29, 2026. Later campaigns changed the task set and harness, so their scores should not be read as direct updates to this table.
The first version was too easy. Agents edited files, ran checks, and passed the public tests. That felt useful for about a day. When every serious lane scored 8/8, the score mostly said the runner worked and the examples were easy. It said almost nothing about the rules that usually bite later.
Chat output wasn't enough.
This started with Cursor Composer 2.5. Could it replace, or at least pressure, the managed ArchiBot path for Archibus coding tasks? Comparing chat answers wouldn't tell me. The agent had to write code in a clean task copy and face a verifier that didn't care how confident its final message sounded.
The harness made a fresh task copy for each run, used deterministic verifiers, and kept the fixtures sanitized. None of the tasks came from customer systems. They were small, public-safe versions of work I already deal with: comparing Archibus metadata to SQL table definitions, AXVW view bindings, WebCentral runtime profiles, SQL safety checks, and WebCentral startup remediation plans.
The inventory behind the bench mapped 71 Archibus skills into 14 task families. The scored sweep was smaller: 18 sanitized tasks. It did not test every skill or every Archibus workflow. I wanted to give the agent context, tools, and a plausible path, then see whether it still preserved the rule a support person needed.
A coding agent with tools is like a junior engineer with shell access. The shell access matters, but it does not mean they understand your production rules.
The visible tests lied.
The visible tests still had a job. They checked that the agent wrote files in the right place and returned the right shape. They didn't show whether the answer held up after the easy examples ran out.
I added hidden verifier overlays. After the agent finished, the harness copied in harder checks and ran them. That worked more like review. The agent kept its normal tools for the visible task, but it couldn't tune every branch to cases it couldn't see.
The failures weren't dramatic. They were enterprise-software boring: quoted identifiers, inline primary keys, JavaScript handlers written as quoted object properties, XML namespaces, uppercase database names, and comments that named a scoped column while the SQL itself stayed unscoped.
The 5/5 that was really a 0/5.
Cursor Composer 2.5 first looked fixable. Copied skills moved it a little. Mandatory Cursor rules moved it a lot. A deterministic preflight tool closed the last gap on the tuned hidden set. The lane went from 1/5 to 4/5 to 5/5. That climb looked like learning.
Rules plus a local checker are still useful. That 5/5 didn't prove they generalized.
I froze the setup and made a new holdout with different edge cases. Composer with copied skills scored 0/5. Frozen rules scored 0/5. Frozen rules plus the old preflight tool also scored 0/5. The harness hadn't lied about the old 5/5. I had tuned the support around the last exam and nearly called that capability.
The 0/5 kept the old 5/5 from becoming the product story.
What the larger sweep showed.
I don't use the old 5-task table as the main result. It caught overfitting, but it was too small. For the May 29, 2026 snapshot, I used an 18-task medium-effort baseline. It was still a small Archibus-style benchmark, not coverage of Archibus as a whole.
ArchiBot medium was the strongest lane I reran cleanly. It passed 17 of 18 tasks with the hidden checks held back until verification. Cursor GPT-5.5 medium-fast passed 12 of 18 after I stripped down the bench copy and cleared Cursor's project cache between tasks. Claude Sonnet medium passed 9 of 18 in Claude Code.
| Lane | Passed | Avg sec | What I take from it |
|---|---|---|---|
| ArchiBot medium | 17/18 | 110 | Best managed ArchiBot baseline. Missed the exact skill-router task. |
| Cursor GPT-5.5 medium-fast | 12/18 | 63 | Stricter rerun. Still strong, but missed several hidden domain rules. |
| Claude Sonnet medium | 9/18 | 99 | Used native tools, but missed more hidden Archibus rules. |
This isn't a clean model leaderboard. Cursor and Claude Code are agent harnesses, not bare models. They search, edit, run tests, and use their normal local tools. For the stricter Cursor row, I removed copied benchmark context, prior reports, docs, hidden verifier directories, and eval-specific Cursor project cache before every task. ArchiBot has its own endpoint context. The supported claim is narrower: at medium effort, with useful tools, the managed ArchiBot path kept more Archibus-specific rules.
I keep the higher-reasoning runs out of the medium baseline. ArchiBot high finished the full 18-task set at 18/18. Claude Opus xhigh finished 12/18. Cursor GPT-5.3 Codex high-fast finished 13/18. Those are ceiling checks, not the default product tradeoff.
Composer needs its own asterisk. A later run showed 16/18 in the Cursor agent harness, but the isolation was weaker than the label suggested. It disabled copied add-on context. It didn't remove normal Cursor tools, shell search, or ambient workspace state. After checking the sessions, I count that 16/18 as a Cursor harness event, not proof that the base Composer model learned Archibus.
Where the agents actually broke.
The misses sat in the boring parts of enterprise software. SQL table definitions had to
normalize quoted names, inline primary keys, NVARCHAR, comma-bearing default
values, and equivalent types. AXVW files used handler shapes the easy tests missed.
Runtime selection had to accept uppercase database names and numeric Java/Tomcat inputs.
A demo can skip those details. A usable answer can't.
The SQL safety task made this plain. A bind placeholder doesn't create tenant scope by
itself. The scoped column must appear in the SQL. A comment that names it doesn't count.
Neither does an alias such as token_masked. The agent had to preserve the
safety rule, not nearby text that looked right.
The remediation task had the same problem. A WebCentral startup log can contain several ugly clues, and the answer must put them in order. A database connection failure outranks a later login failure. A license-cache file may be part of the fix. The output has to guide an operator, not merely satisfy a parser.
This is not the cost post yet.
The money question matters, but the billing basis is not the same across lanes.
The lanes used different products and billing surfaces. I did not have one comparable dollar measure. A cost comparison would require a dated, instrumented run that counted failed attempts and retries. This post is about the eval shape.
What I'd tell anyone building an agent eval.
Keep the visible tests, but don't stop there. If the agent can see every case that matters, the score mostly measures how well it fits examples. Preserving a rule under review pressure is a different test.
Freeze the rules and tools, then run a fresh holdout with new edge cases. If the score collapses, you built a better answer key for the last exam.
Keep setup failures separate from quality failures too. Some Cursor Claude Opus runs stopped before task execution with capacity or setup errors reported by the harness. I excluded those runs from quality comparisons. The direct Claude Code run completed and still missed hidden Archibus edges.
The claim stays narrow. In this harness, refreshed May 29, 2026, hidden verifiers changed the result across this Archibus-style task family. Without them, I would have trusted a neat score that mostly proved the runner worked.
An agent benchmark without hidden verifiers mostly measures how well the agent can satisfy the examples it can already see.