Enterprise AI agent evaluation starts where APIs meet documents

The real boundary is the join

Enterprise AI agent evaluation is weakest when it tests retrieval and tool use as separate skills. Real work usually asks the agent to read policy text, map it to structured records, call an API, and then prove that the action followed the rule.

That join is where demos break. A retrieval agent can quote the right paragraph and still update the wrong account. A tool-using model can produce valid JSON and still violate a policy because it ignored a date, exception or eligibility rule buried in a document.

The new VAKRA benchmark makes this boundary explicit. IBM researchers describe VAKRA as a benchmark for agents that must reason across structured APIs and document collections, with more than 8,000 executable APIs across 62 domains and tasks that combine API styles, multi-hop reasoning and natural-language tool-use policies VAKRA. Its headline result is blunt: the strongest tested model, GPT-5.5, reached 70.4% on single-hop endpoint-style tasks, but dropped to 50–51% on compositional business-intelligence APIs VAKRA.

The result matters because most enterprise tasks look closer to the hard side of that split. A support agent checks a warranty policy, looks up a purchase, confirms a region-specific exception and calls a refund API. A healthcare agent reads a payer policy, checks patient records and creates a prior-authorisation workflow. A finance agent applies a risk rule, checks transaction history and escalates only when the threshold is met.

These are not pure search tasks. They are not pure API tasks either. They are controlled execution problems.

Retrieval success is not operational success

Retrieval benchmarks usually ask whether the system found or generated the right answer from text. That is useful, but it is not enough for enterprise agents. The downstream risk is not only a wrong answer. It is a wrong action taken with a correct-looking citation.

The retrieval layer has to solve three problems before tool use even begins:

• find the relevant source;
• select the exact policy version that applies;
• preserve enough context for the action layer to make a safe decision.

The AWS case study on Cohere Health shows why this matters in clinical policy work. Cohere Health uses Amazon Bedrock AgentCore to digitise clinical policies, with version tracking, memory and structured outputs so the basis of a decision can be reviewed later AWS Machine Learning Blog. That is a practical example of the document-to-execution join. The policy is not just content. It becomes an operational control.

This is especially important in regulated industries. In healthcare, a stale payer policy can change the answer. In finance, a missing jurisdiction flag can change the permitted action. In legal workflows, metadata determines whether a clause is relevant at all, which is why retrieval for contract work increasingly depends on metadata filters, not only embeddings /insights/contract-search-ai-metadata-filtering-rag.

The weak pattern is easy to recognise. A demo shows a chatbot answering questions over PDFs. Then a separate demo shows function calling against a CRM. The real business task needs the system to use the PDF to decide what CRM action is allowed. That middle step is where evaluation has to focus.

Tool-call syntax is the easy part

A tool use benchmark can test whether a model emits a valid function call. That was a necessary first step for the field. It does not fully measure whether an agent can operate a business process.

BFCL V3, the Berkeley Function-Calling Leaderboard release from September 2024, widened the frame by adding multi-turn and multi-step function calling BFCL V3. The authors distinguish single-turn calls from multi-step work, where the assistant must execute multiple internal calls, and multi-turn work, where the user and assistant exchange context over time BFCL V3.

That distinction maps directly to enterprise systems. A single API call might retrieve an order. The actual workflow may need five steps: resolve the customer, list orders, identify the affected item, check refund eligibility and create the return. If any step depends on a document rule, the task becomes harder again.

ToolSandbox, introduced by Apple researchers and revised in April 2025, pushes in a similar direction. It includes stateful tool execution, implicit state dependencies, a user simulator and dynamic evaluation over intermediate and final milestones ToolSandbox. The paper argues that earlier work often focused on stateless REST services, single-turn prompts or off-policy dialogue trajectories ToolSandbox.

That is the right concern. Business APIs are rarely stateless from the user’s point of view. A password reset can invalidate a session. A refund can change inventory. A support escalation can lock a ticket. Evaluation has to inspect the state after the agent acts.

The lesson is not that syntax benchmarks are obsolete. They are still useful for model selection. The problem is using them as a proxy for deployment readiness.

Policies turn agents into conditional systems

Policies are not passive documents. They are conditional logic written in human language.

That is why policy adherence changes the evaluation problem. The agent must decide whether it has enough evidence, whether a rule applies, whether a tool is allowed and whether no action is the correct outcome. These are separate decisions.

τ-bench, published in June 2024 by researchers from Sierra and Princeton, tests agents in dynamic conversations with domain-specific API tools and policy guidelines τ-bench. The benchmark compares the final database state with an annotated goal state and introduces pass^k to measure reliability across repeated trials τ-bench. Its reported results show that GPT-4o achieved about 61% pass^1 on τ-retail and about 35% on τ-airline, while pass^8 fell to about 25% on τ-retail τ-bench.

That reliability metric is useful because enterprise automation runs repeatedly. A support agent that succeeds once in a demo but fails one in three times in production is not a labour-saving system. It is a queue of exceptions.

VAKRA adds another pressure point: unanswerable or policy-constrained questions. The paper reports severe failures on policy-constrained categories, including accuracy as low as 2.4% on unanswerable queries in its table of model results VAKRA. This is one of the most important findings for buyers. A production agent must know when not to act.

That requirement affects design. A reliable system needs refusal paths, escalation rules, audit logs and test cases for negative examples. In Cloudex terms, this is not only an AI model choice. It is an implementation problem across AI Solutions, internal software, permissions and monitoring.

The evaluation unit should be the trajectory

For enterprise agents, the natural evaluation unit is the trajectory: every retrieval, reasoning step, tool call, state change and final response.

VAKRA’s authors re-execute predicted tool calls against live APIs to verify correctness and allow multiple valid paths VAKRA. BFCL V3 uses state-based and response-based checks at the end of every turn for multi-turn tasks BFCL V3. ToolSandbox uses milestones and minefields to evaluate intermediate and final events ToolSandbox.

These methods differ, but they point in the same direction. Evaluation should not ask only “did the answer look right?” It should ask:

1. Which source documents were retrieved?
2. Which policy version applied?
3. Which entity was resolved?
4. Which tool was called?
5. What parameters were sent?
6. What state changed?
7. Did the final response match the evidence?
8. Was doing nothing the correct action?

This is also where observability becomes part of evaluation. A team cannot improve what it cannot replay. Cloudex covered this deployment issue in a separate article on why AI agent observability is now required, not optional /insights/ai-agent-observability-deployment-requirement.

The Model Context Protocol documentation shows how tool descriptions, input schemas and tool results are formalised for agents Model Context Protocol. That standardisation helps. It does not remove the need to test policy interpretation, record matching and side effects.

What a serious enterprise evaluation should include

A practical enterprise AI agent evaluation should start with the business process, not the model leaderboard. The test set should reflect the actual join between documents and systems.

For an agent handling customer operations, include policy documents, CRM records, order records, user messages and executable tools. For a healthcare workflow, include clinical policy versions, patient context, payer rules, prior-authorisation states and escalation paths. The Cohere Health example is useful because it treats provenance, versioning and reviewability as first-class system concerns AWS Machine Learning Blog.

A strong evaluation set should include:

• common successful tasks;
• edge cases that require policy exceptions;
• missing-information cases;
• ambiguous entity matches;
• stale or superseded documents;
• tools that should not be called;
• API errors and retries;
• records with similar names or IDs;
• tasks where escalation is the correct outcome.

The scoring should separate failure types. “Wrong answer” is too vague. The system may have retrieved the wrong source, mapped the wrong entity, selected the wrong endpoint, passed the wrong parameter, ignored a policy exception or acted when it should have refused.

This separation matters commercially. If failures cluster in retrieval, improve indexing, metadata and document preparation. If they cluster in schema alignment, improve tool descriptions and parameter validation. If they cluster in policy reasoning, add rule extraction, human review or narrower automation boundaries.

For many firms, the first safe deployment is not a fully autonomous employee. It is a constrained agent that drafts, checks, routes or prepares actions for approval. That still creates value. It also reduces risk while the evaluation set grows. Over time, higher-confidence paths can move into AI Employees, while lower-confidence paths stay human-reviewed.

What this means if you are deciding whether to act

If your process needs both documents and APIs, do not buy or build from a retrieval demo alone. Ask for an evaluation that uses your policies, your records and your tool permissions.

The minimum acceptable proof is a replayable test run. It should show the source documents retrieved, the policy version used, the tool calls made, the parameters sent, the final state and the cases where the agent refused or escalated. If a vendor cannot show this, they are asking you to trust the final answer without inspecting the work.

Start with one narrow workflow. Choose a process where the documents are known, the APIs are bounded and the cost of a wrong action is manageable. Build a test set before expanding scope. Include negative cases from day one.

For healthcare, finance, legal and operational systems, this is not extra caution. It is the work. The agent is only useful when the join between text and execution is tested as carefully as the model itself.

If you are evaluating a document-and-API workflow, Cloudex can help scope the test set, integration boundary and review path through our AI Solutions work. Start with the smallest workflow where a wrong action has a measurable cost: /contact.