Evaluating an Agentic System: Three Axes, and What Auditing the Measurements Changed

What makes the system agentic

A fixed pipeline would do the same thing every time: take the question, search once, paste the top results into the prompt, answer. This system gives the model tools and lets it decide. It writes its own search query, looks at the results, decides whether to open a page, may search again in different words, and finally either answers with citations or says the documentation does not cover the question.

That freedom is the point. A reader’s words are often not the documentation’s words, and the model rewrites them: asked why a job was killed, it searches for the memory limit. It can also notice that a search missed and try again, which a fixed pipeline cannot.

Figure 1: Exchanges with the model per question, over the documentation questions answered through the tool loop. One exchange means the model answered without using a tool at all. Three is the common shape: search, read, answer. Above three means it went back for more, usually in different words.

That freedom is also what makes evaluation hard. There is no single expected output to compare against, because two correct answers can take different routes, cite different pages and use different words. What can be checked is narrower and more useful: did it search, did it open a page, did it cite something that exists, did it stay inside what the documents say, and did it decline when it should have.

What the evaluation framework is, and why it helps

The framework has four parts. Labelled question sets say what a good answer looks like. Deterministic checks inspect each answer with about twenty rules, and no second model judges anything. A harness runs the real application question by question and records everything observable about the turn. A scorecard collects the results, and its headline is the worst cell rather than an average.

Figure 2: The question sets, each written for a failure the others cannot see. The counterparts matter as much as the main cases: an identifier trap is answerable but carries an unfamiliar word, and refusing it is as much a fault as answering an impossible question.

Three questions are kept apart, because they move for unrelated reasons. Is the application robust to a bad model? Deterministic, and gated on every push. Is this particular model good enough? Stochastic, and never gated, because the system runs on free model endpoints whose lineup changes without notice. Can the documents answer the question at all? No prompt can fix a missing page, and the documents are the cheaper thing to repair.

The framework earns its keep in three ways. It turns model choice into a measurement: the default model had been chosen from a latency remark in a code comment, and is now chosen from a table. It makes a class of regression unshippable, because an invented command flag fails a build rather than reaching a reader. And when something goes wrong, it says which of the three questions the fault belongs to, which is usually most of the work.

One detail matters more than it sounds. The harness imports and drives the real application rather than reimplementing its loop. A short reimplementation would drift the first time the real loop changed, and would then measure a fiction that never fails.

The checks were wrong before the application was

A rule with a mistake in it produces exactly what a correct rule produces: a number. So the first audit pass re-scored every answer the project had recorded, 514 of them, and read a sample of each rule’s output rather than the totals.

Figure 3: Findings from four rules over the same 514 recorded answers, before and after reading a sample of each rule’s output. Two rules were reporting nothing but their own bugs. footer-survived mattered most because it is a defect rather than a warning, so it can fail a build, and it was firing on correct refusals.

None of those rules was wrong in an interesting way. Each had been written against a real defect and then met prose it had not anticipated. The lesson is procedural: a check whose output nobody has read is not a measurement, and the more authoritative its number looks, the longer it survives.

An instruction is a request; a mechanism is a guarantee

The clearest application fault came from a reader rather than a test. Asked mid-conversation whether its answers could be trusted, the system replied that everything is “pulled from the official documentation (via the search_docs and read_doc tools)”. The substance is true and worth saying. The two function names are not: the reader cannot call them, cannot check them, and had asked about trust rather than about architecture.

The obvious fix is an instruction. It is worth knowing how far an instruction gets.

Figure 4: Sixteen questions about the system itself, asked four ways, run through the real application. Held means the delivered answer named no internal identifier. The two middle bars are the same instruction measured twice, which is what a sixteen-case set looks like when one case is six points.

An instruction cannot finish this job, for a structural reason. The tool names reach the model twice: once in the tool definitions the provider interface requires, and again in the lines of the prompt that tell the model what to call. A rule three paragraphs later is arguing with the rest of its own context.

So the second half of the fix does not depend on the model. Each tool carries a reader-facing name beside the identifier the interface needs, and the identifier is replaced on the way out. A substitution rather than a deletion, because deleting names from a list of them leaves a broken sentence. Two properties made it safe to ship: over every section of the corpus the substitution changes nothing, which a test asserts rather than claims, and whatever it removes is recorded, because a fix that hides the behaviour it corrects would score every model as discreet.

Figure 5: Six free models, the same sixteen questions each. Held counts the answer as delivered; alone removes the substitution and counts what the model managed by itself. The gap between each pair is the size of what the mechanism is catching.

Two findings withdrawn, because the metric was wrong

The system shows a strip of sources under each answer, built from the pages the model opened. A benchmark column labelled “cited a gold page” checked that strip. But a model can also cite a page from a search result without opening it, and the link it writes is real and clickable. Those turns scored as having cited nothing.

Figure 6: The same recorded answers under two definitions of citing a page. Single turns are unaffected, because a first turn searches and opens a page. A follow-up is exactly the turn that answers from what it already has, which is why the published figure for follow-ups was an artefact.

The framework had reported that models “go 100 % on the first turn and 78 to 89 % on the follow-up”, and had concluded that a follow-up loses the thread. That claim is now withdrawn. The withdrawal has a sting. Months earlier a line had been added to the prompt to make the model carry the earlier subject into its search query, then reverted because the apparent gain was within noise. The caution was right for the wrong reason: there was nothing to fix, and had it shipped, the project would now carry permanent guidance addressing a gap in a measurement.

What the audit found in the application

The application faults were fewer than the measurement faults, and were mostly text reaching a reader that nobody had decided to send. The sharpest was on the second answering path. Models that cannot use tools get one search up front and a single chance to answer. Eight of fourteen answers on that path named a tool, and three were nothing but the call written out as text. The entire answer to “how do I submit a batch job” was search("submit batch job sbatch RCC Midway"), under a strip of six real sources.

The model was not misbehaving. The instructions describe a search-and-read loop, because most turns have one, and this path’s own instruction never said otherwise.

Figure 7: The same fourteen questions down the single-pass path, before and after one sentence was added to that path’s instruction. Retrieval is unchanged because this path searches once with the reader’s own words, which is not something a sentence can address.

Three more, each found by probing rather than reported:

• A filename could carry an instruction. Uploaded files are quoted to the model between markers naming the file, so a file named with an embedded end-marker closed the quotation early and put a directive outside it. Repairing the structure was not enough: both promises about uploads concern a file’s content, while its name arrived in the application’s own voice. The name is now quoted as the user’s text. Two models, six cases each, nothing obeyed.
• A model’s own reasoning shipped as an answer. One turn in 554 returned 34,645 characters of a model reasoning about its instructions and quoting them clause by clause, cut off by the token ceiling without ever answering.
• A pasted web address defeated the refusal mechanism. The system declines when an unfamiliar word looks like the name of something: capitalised away from a sentence start, or introduced by a naming preposition. Inside a hostname both signals are blind, because the labels are lower case and no preposition introduces them. A question naming another centre’s machine inside a URL was answered confidently from these documents.

What ten passes taught me

Measure the measurement first. Every pass that began by reading what a rule reported found something. Passes that began by looking for an application bug mostly found the application working.

A check that cannot fail reads as a pass. This is the costly failure, because it is invisible by construction. The remedies are cheap: assert that every finding type is exercised by a test, assert that every element a layout rule depends on matched something at least once in a run, and make a new guard fail on purpose before trusting it.

Prefer the mechanism whose failure mode is visible. An instruction is a request that a weak model grants about half the time. Something the application does to the text is a guarantee, and when it is wrong the wrongness is on screen rather than in a distribution.

Do not let the fix blind the instrument. The substitution that removes tool names would have made every model look discreet. Recording what it removes keeps the behaviour measurable while the reader is protected.

Keep both readings when a metric has two honest definitions. “The page the model opened” and “the page the reader can click” are different questions, and reporting one under the other’s name produced a false finding about model behaviour.

Probe the shapes the labelled set omits. The pasted address was in no question set, and the sets had been carefully stratified. Fifteen deliberately awkward questions took an afternoon and found the one case where two independent signals were blind at once. A labelled set encodes the failures somebody already imagined.

The system ended these passes better than it started. The more useful outcome is that its numbers are now about the system rather than about themselves, and two things the project believed about its own models were retracted on evidence. A framework that can retract its own findings is worth more than one that has never had to.