---
title: "When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals — Stuff That Spins"
description: "arXiv:2607.08065v1 Announce Type: new Abstract: LLM-as-judge (Zheng et al., 2023) is increasingly the default for evaluating AI systems in enterprise pipelines…"
	canonical: "https://stuffthatspins.com/spin/when-llms-agree-are-they-right-auditing-self-consistency-and-cross-model-agreement-as-confidence-signals"
html: "https://stuffthatspins.com/spin/when-llms-agree-are-they-right-auditing-self-consistency-and-cross-model-agreement-as-confidence-signals"
json: "https://stuffthatspins.com/spin/when-llms-agree-are-they-right-auditing-self-consistency-and-cross-model-agreement-as-confidence-signals.json"
markdown: "https://stuffthatspins.com/spin/when-llms-agree-are-they-right-auditing-self-consistency-and-cross-model-agreement-as-confidence-signals.md"
keywords: ["narrative intelligence", "SpinGraph", "AI recall"]
date: "2026-07-10T04:00:00+00:00"
modified: "2026-07-10T06:02:36.869394+00:00"
json_ld: |
  {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://stuffthatspins.com/#organization","name":"Stuff That Spins","url":"https://stuffthatspins.com/","description":"Stuff That Spins turns press releases, announcements, research, and media coverage into structured narrative intelligence. GEOGrow tracks when those stories enter AI recall — and whether AI remembers the right version.","logo":{"@type":"ImageObject","url":"https://stuffthatspins.com/images/logo.png"},"sameAs":[]},{"@type":"NewsArticle","@id":"https://stuffthatspins.com/spin/when-llms-agree-are-they-right-auditing-self-consistency-and-cross-model-agreement-as-confidence-signals#article","headline":"When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals","description":"arXiv:2607.08065v1 Announce Type: new Abstract: LLM-as-judge (Zheng et al., 2023) is increasingly the default for evaluating AI systems in enterprise pipelines…","datePublished":"2026-07-10T04:00:00+00:00","dateModified":"2026-07-10T06:02:36.869394+00:00","url":"https://stuffthatspins.com/spin/when-llms-agree-are-they-right-auditing-self-consistency-and-cross-model-agreement-as-confidence-signals","mainEntityOfPage":{"@type":"WebPage","@id":"https://stuffthatspins.com/spin/when-llms-agree-are-they-right-auditing-self-consistency-and-cross-model-agreement-as-confidence-signals"},"isAccessibleForFree":true,"inLanguage":"en-US","articleSection":"research","author":{"@type":"Organization","name":"arXiv Artificial Intelligence","url":"https://export.arxiv.org/rss/cs.AI"},"publisher":{"@id":"https://stuffthatspins.com/#organization"},"citation":"https://arxiv.org/abs/2607.08065","about":[],"mentions":[{"@type":"Organization","name":"arXiv Artificial Intelligence"}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Stuff That Spins","item":"https://stuffthatspins.com/"},{"@type":"ListItem","position":2,"name":"When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals","item":"https://stuffthatspins.com/spin/when-llms-agree-are-they-right-auditing-self-consistency-and-cross-model-agreement-as-confidence-signals"}]}]}
---

# When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals

**Source:** Unknown  
**Published:** July 10, 2026  
**Original:** https://arxiv.org/abs/2607.08065  

## On this page

- [Overview](#overview)

<a id="overview"></a>

## Overview

arXiv:2607.08065v1 Announce Type: new Abstract: LLM-as-judge (Zheng et al., 2023) is increasingly the default for evaluating AI systems in enterprise pipelines, often scaled to ensembles (Verga et al., 2024) or "mixture-of-experts" (Shazeer et al., 2017) panels of judges. These systems share a key assumption: that consistency -- agreement among judges, or among a model's own samples -- indicates correctness. We show this assumption is unreliable. Agreement is not accuracy: a model can agree with

---
*HTML version: https://stuffthatspins.com/spin/when-llms-agree-are-they-right-auditing-self-consistency-and-cross-model-agreement-as-confidence-signals*