Quick answer: you evaluate an AI agent at three levels. First, task success: did the agent accomplish the goal, judged only on the final outcome. Second, tool-call accuracy: did each individual tool call use the right tool with the right arguments. Third, trajectory scoring: was the overall path sensible, or did the agent wander, loop, and burn tokens before stumbling onto the answer. A useful eval suite measures all three, because an agent can pass any one of them while failing the other two.
Classic LLM evals assume one prompt in, one answer out. You compare the answer to a reference, compute a score, done. An agent breaks that assumption. Its output is a trajectory: a sequence of thoughts, tool calls, tool results, and a final response, and the sequence matters as much as the ending. Two agents can both return the correct answer, where one made two well-chosen searches and the other made fourteen calls, fetched the same page three times, and got lucky. Single-answer evals score those identically. In production they are not remotely the same system.
This post walks through the three levels using a web-search agent as the running example: an agent with web_search and fetch_url tools that answers questions from live web data, like the one in our agent internet access guide.
Level 1: task success
Task success is the outcome-only view. You give the agent a task with a verifiable end state, let it run the whole loop, and check the result. You do not look at how it got there.
The key word is verifiable. "Summarize this topic well" is not checkable. "What is the current stable version of Node.js" is, because you can resolve the ground truth at grading time. For a web-search agent, good task formats include:
- Closed factual lookups. "What year did X acquire Y?" Grade with exact or normalized string match.
- Fresh-data lookups. "What is the top story on this site right now?" Grade by fetching the ground truth yourself at eval time, since a static answer key goes stale.
- Multi-hop questions. "Who is the CEO of the company that makes X?" These force at least two dependent steps, which is where agents actually differ.
- Unanswerable questions. Include tasks where the correct behavior is "I could not verify this." An agent that never says so will hallucinate under pressure, a failure mode we cover in how to make AI agents cite sources.
Report task success as pass rate, but run each task several times. Agents are far less deterministic than single completions, because randomness compounds across every step of the loop. A task that passes 9 of 10 runs and a task that passes 5 of 10 both show up as "sometimes works" in casual testing. pass@k and its stricter cousin pass^k (passes all k runs) separate them.
Level 2: tool-call accuracy
Task success tells you whether the agent works. Tool-call accuracy tells you where it breaks. Here you score individual steps against what a competent agent should have done at that point:
- Tool selection. Given the state, did it pick the right tool? Searching when it already had the URL in context is a selection error, and so is fetching a page when it first needed to find one.
- Argument quality. Was the search query well-formed and specific? Did the fetch use the exact URL from the search results, or did the model mutate it? URL hallucination in
fetch_urlarguments is one of the most common web-agent bugs, and outcome-level evals hide it whenever a retry happens to save the run. - Result handling. After the tool returned, did the agent use what came back, or did it ignore the fetched content and answer from memory anyway? Compare the final answer's claims against the tool outputs in the transcript.
The cheap way to build step-level evals is to mine your own transcripts. Take 50 real trajectories, mark each tool call as correct or incorrect, and you have a labeled dataset plus an honest error taxonomy. Most teams find their errors concentrate in two or three categories, such as vague search queries or ignoring a 429 response and retrying instantly, which turns a vague "the agent is flaky" into a concrete fix list.
Tool-call accuracy is also where deterministic checks shine. You do not need an LLM judge to verify that a fetched URL appeared in prior search results, that arguments validate against the tool schema, or that the agent never called the same tool with identical arguments twice in a row. Assert those in code.
Level 3: trajectory scoring
Trajectory scoring judges the path as a whole. The questions it answers: was the plan coherent, was the step count reasonable, did the agent recover from bad tool results, and did it stop when it had enough information.
Two practical approaches:
Reference trajectories. For each task, write the tool sequence a strong agent would take, such as web_search then fetch_url then answer. Score observed trajectories on precision and recall against the reference: did the agent make the necessary calls, and how many unnecessary ones did it add. This is strict but fully automatic, and it catches the fourteen-call lucky run that task success misses.
LLM-as-judge over the transcript. Feed the full trajectory to a judge model with a rubric: efficiency, groundedness, recovery from errors, stopping behavior, each scored 1 to 5 with a required justification. Judges are noisy, so anchor them. Include two or three hand-scored example transcripts in the judge prompt, and spot-check 10 percent of judgments by hand until you trust the agreement rate.
Alongside the scores, track hard trajectory metrics that need no judge at all: steps per task, tokens per task, wall-clock time, tool error rate, and loop detection (the same call repeated with the same arguments). These are your early-warning system. A prompt change that nudges task success from 82 to 84 percent while doubling average steps is usually a regression wearing a good outfit.
Making it a habit
None of this matters as a one-time audit. The value shows up when the suite runs on every change: every prompt tweak, tool description edit, and model upgrade. Keep the harness boring: a directory of task definitions, a runner that executes each task k times, graders per level, and a table comparing the current run to the last accepted baseline.
One operational note for web-search agents specifically: your eval results are only as stable as your tools. If fetch_url randomly fails on JavaScript-heavy or bot-protected pages, your evals measure your scraping infrastructure, not your agent. That is one reason we route eval and production traffic through the same link.sc endpoints the agent uses, so a failed fetch is an agent problem, not an infrastructure mystery. If you are still building the agent itself, start with our step-by-step web scraping agent guide and add the eval harness once the loop works.
Start small. Twenty verifiable tasks, run five times each, with task success, one or two deterministic tool-call checks, and a step-count budget will catch the large majority of regressions. You can add judge-scored trajectories once the basics are green.
Ready to give your agent reliable search and fetch tools worth evaluating? Create a free link.sc account and get API access in under a minute.