Adaptive statistical evaluation for LLM agents. Three techniques — Adaptive Sampling, ROPE-NHST, and Bayesian Adaptive Inference — deliver statistically sound benchmarks without fixed-N waste. 83% cost reduction over fixed-N approaches measured in arXiv:2603.02601. AGPL v3.
Running N=100 samples for every evaluation task charges you the same cost whether the signal resolves after 5 samples or never. AgentAssay stops as soon as it has enough evidence — and tells you when more samples would not change the conclusion.
Pick one technique or combine all three in a pipeline. Each is independently sound; together they cover a wider range of agent behavior profiles.
Samples agent performance until statistical confidence bounds tighten to within a configured threshold. Stops early when the signal is clear; runs longer when the agent is noisy. Eliminates the guesswork of choosing N upfront.
Defines a Region of Practical Equivalence around the null value. Evaluation stops when the posterior is entirely inside ROPE (practically equivalent) or entirely outside (meaningfully different). Interpretable, principled, and efficient.
Maintains a Bayesian posterior over agent performance. Each sample updates the prior. The framework stops when the posterior's credible interval is narrow enough to drive a decision. Naturally accounts for heterogeneous agent behavior.
Published in the Journal of AI Research. Preprint on arXiv; data archive on Zenodo.
AgentAssay is one of three Qualixar tools in the AI reliability stack.