evalstats

Narrative intelligence for evalstats: 1 tracked articles, claims, and spin patterns across AI and technology coverage.

How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstats

A new open-source Python package called evalstats provides statistically calibrated methods for evaluating LLM judges in small-sample AI research, correcting for inflated false positives and judge bias that undermine reliability of significance claims.

Spin 25% Verified AI Risk Moderate

arXiv Computation and Language

Sep 30, 2026

Related Claims

01 Running statistics over raw LLM judge scores leads to inflated false positives, especially when inter-rater agreement appears 'almost perfect'.

Markdown (.md) · JSON-LD schema (.json) · Machine-readable for AI & GEO