Prompt Framing Distorts Count-Based Evaluation of LLM Error Detection: Evidence from Numeric Anchoring

Summary

arXiv:2607.01240v1 Announce Type: new Abstract: Count-based F1 is widely used as a proxy for LLM error-detection quality, but this paper shows that it can rise dramatically without a corresponding improvement in span localization, a gap termed F1 Inflation. The paper introduces ErrorBench, a controlled stress-test protocol for prompt-induced count distortion. ErrorBench evaluates six contemporary LLMs under five prompt conditions over 4,290 responses from 143 CoNLL-2014 passages. Under CoNLL-201

SpinGraph analysis pending — check back after processing.

• Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale
• Parameter Golf: What Really Works?
• From Monolingual to Multilingual: Evaluating Mamba for ASR in South African Languages
• Comparing Architectures for Supervised Political Scaling
• Grounded Optimization: A Layered Engineering Framework for Reducing LLM Hallucination in Automated Personal Document Rewriting
• FaithMed: Training LLMs For Faithful Evidence-Based Medical Reasoning