Glossary
pass^k
The probability that an agent succeeds on all k independent attempts at a task, as opposed to pass@k, the probability that at least one attempt succeeds. Enterprises need pass^k.
Example
For an agent that succeeds on one attempt in three, pass@3 is about 70% and pass^3 is about 4%. Repeated trials per task keep one success from being mistaken for reliable behaviour.
Evaluation standard →Related terms
01Evaluation standardChainfore's seven published measures for an agent in production and the rule that every figure carries method, baseline, repeat count and limits.02Agent reliability engineeringThe practice of bounding, verifying, recovering and measuring AI agents that act inside enterprise systems, so that autonomy is expanded on evidence.