Failure tests, staged autonomy and a buyer scorecard.
We test for the failures that matter before any agent is trusted with more. You can run the same comparison against any supplier, including us.
Failure tests
| Failure we test | Control | Test |
|---|---|---|
| A retrieved document tries to redirect the agent | Retrieved content is treated as data, not instruction | Adversarial instruction in a source; the action must be refused |
| An action targets the wrong tenant | Identity and tenant checks sit in the tool layer | Cross-tenant request must be denied |
| An approved action is changed before execution | Approval is bound to parameters and expiry | Modified action must be blocked |
| A retry duplicates a transaction | Idempotency keys and state reconciliation | Injected timeout after a write must not duplicate it |
| The model reports success without a real change | Read-back of the authoritative system | False completion claim must fail the check |
| The agent holds more access than it needs | Scoped, short-lived credentials | Revocation and expiry tests |
| An evaluator sees sensitive content | Minimised, redacted traces; controlled evaluator access | Trace-access audit; documented retention |
Autonomy is granted in stages
- Stage 1
Offline.
Tested against synthetic fixtures.
- Stage 2
Shadow.
Observes real work without acting, under authorised data handling.
- Stage 3
Supervised.
Requests approval for each bounded write.
- Stage 4
Limited autonomy.
Executes explicitly low-risk actions within validated limits.
Scope expands only after the owner approves and new evidence is produced. A change to tools, model version or permissions restarts the relevant tests.
A buyer scorecard you can use
The task, data access, tool permissions, time budget and acceptance criteria are agreed in advance. The buyer holds the hidden test set, so suppliers are compared on the same work.
| Criterion | What is scored | Weight |
|---|---|---|
| Correct business outcome | State checks, accepted artefacts, error severity, repeatability | 30% |
| Control and recoverability | Unauthorised-action tests, escalation, rollback, failure recovery | 25% |
| Integration and maintainability | Working connections, readable code, handover, dependencies | 20% |
| Operating economics | Cost per accepted task, review effort, latency, support | 15% |
| Accountability and transferability | Named owner, documented limits, reference, exit plan | 10% |
Weights are a starting point. We agree them with the buyer.
Questions buyers ask
What failures does Chainfore test for?
A retrieved document that tries to redirect the agent, an action against the wrong tenant, an approved action changed before execution, a retry that duplicates a transaction, a success report with no real change, more access than the task needs, and an evaluator that sees sensitive content.
How is autonomy granted to an agent?
In four stages: offline, shadow, supervised and limited autonomy. Scope expands only after the owner approves and new evidence is produced.
How can we compare AI agent suppliers fairly?
Agree the task, data access, tool permissions, time budget and acceptance criteria in advance, hold a hidden test set, and score every supplier on the same work with the buyer scorecard.
Start with one workflow.
Tell us the process, the cost of failure and the systems involved. We will respond with a proposed first step.
Request an assessment →