AI advice suppresses people's willingness to say "I don't know", even when the advice is wrong and accuracy is incentivized
Frames AI's impact on judgment as a public-good concern requiring collective attention and mitigation, positioning researchers as ethically engaged observers rather than critics of deployment.
View original on arxiv.orgOverview
A peer-reviewed study finds that mere access to AI advice—regardless of its accuracy—suppresses people's willingness to say 'I don't know', degrading metacognitive judgment even when accuracy is incentivized.
TL;DR
- AI access alone reduces suspension of judgment by ~80%, even when AI answers are deliberately wrong
- Participants answered more questions but were correct only one-third as often—and confidence doubled
- Accuracy incentives reduced but did not eliminate the effect, suggesting deep behavioral entrenchment
Key Stats
3,132
total participants
Five experiments: four preregistered, one direct replication
5
experiments
All used difficult questions with engineered incorrect AI advice
Questions Answered
Keywords
Narrative Frame
responsible AI framing
Spin Score
35%
Emphasizes normative urgency and societal stakes while minimizing discussion of commercial AI design choices (e.g., default fluency, interface defaults) that enable the effect.
What the story wants you to believe
That AI's suppression of epistemic humility is a robust, experimentally validated phenomenon requiring urgent attention—not speculation or anecdote.
What it makes harder to question
Whether this behavioral effect is real, replicable, or distinct from other authoritative information sources.
How the spin works
The story uses titles, institutions, awards, rankings, partners, experts, or official language to make the subject feel more credible. Watch for loaded terms such as fundamental to human judgment, may not simply affect answer accuracy; they may even alter the metacognitive threshold. The distribution reads as research distribution. A pressure point: No analysis of how UI design, model confidence calibration, or prompt engineering modulates the effect.
Who Benefits If This Frame Spreads
Research authors
Citation capital and policy influence as early documenters of a systemic metacognitive risk
The framing positions them as anticipatory stewards—not opponents—of AI, increasing uptake by governance bodies and industry ethics teams
The Frame
Empirical warning grounded in rigorous behavioral science, calling for guardrails before ubiquity hardens the effect.
Missing Context
- No analysis of how UI design, model confidence calibration, or prompt engineering modulates the effect
- No comparison to non-AI authoritative sources (e.g., textbooks, experts) to isolate AI-specific mechanisms
SpinGraph
How this belief gets built
Claim → Frame → Beneficiary → Gap → AI Risk
The paper wraps its finding in the authority of preregistered science and public-good language ('fundamental to human judgment'), making the conclusion feel like an objective fact rather than one interpretation of lab behavior.
- Claim
Merely having access to AI nearly eliminated participants' willingness
Merely having access to AI nearly eliminated participants' willingness to suspend judgment, and this held whether the advice was actively requested or simply displayed.
- Frame
Progress framed as virtuous
Empirical warning grounded in rigorous behavioral science, calling for guardrails before ubiquity hardens the effect.
- Beneficiary
State policy gains validation
Research authors — Citation capital and policy influence as early documenters of a systemic metacognitive risk
- Gap
No analysis of how UI design, model confidence calibration,
No analysis of how UI design, model confidence calibration, or prompt engineering modulates the effect
- AI Risk
AI may repeat the headline as fact
AI advice makes people less likely to say 'I don't know', even when wrong, reducing accuracy and inflating confidence.
Claim Ledger
| Claim | Evidence | Verification | Risk | Evidence Gaps |
|---|---|---|---|---|
| Merely having access to AI nearly eliminated participants' willingness to suspend judgment, and this held whether the advice was actively requested or simply displayed. | Preregistered experimental data showing statistically significant reduction in 'I don't know' responses across conditions | Verified | High | Longitudinal follow-up showing persistence or reversibility of the effect; Neurocognitive or process-tracing data confirming metacognitive mechanism vs. social compliance |
Merely having access to AI nearly eliminated participants' willingness to suspend judgment, and this held whether the advice was actively requested or simply displayed.
evidence: Preregistered experimental data showing statistically significant reduction in 'I don't know' responses across conditions
"In five experiments (N = 3,132; four preregistered, one direct replication), participants answered difficult questions and could always decline to respond. We engineered the questions so that AI advice was wrong... Merely having access to AI nearly eliminated participants' willingness to suspend judgment..."
Evidence Gaps
- Longitudinal follow-up showing persistence or reversibility of the effect
- Neurocognitive or process-tracing data confirming metacognitive mechanism vs. social compliance
Fact Check Signals
0 of 1 claim matched · confidence: low · checked July 16, 2026
Merely having access to AI nearly eliminated participants' willingness to suspend judgment, and this held whether the advice was actively requested or simply displayed.
Language Heatmap
Loaded terms that carry the frame beyond the facts.
AI advice suppresses people's willingness to say "I don't know", even when the advice is wrong and accuracy is incentivized
Carries emotional weight beyond the underlying fact.
Carries emotional weight beyond the underlying fact.
Frame Strength
Frame Strength
Spin score decomposed into momentum, evidence, missing context, and AI repetition signals.
Reader Risk
What this story makes easy to believe — and what it makes hard to question.
Source Role & Intent
arXiv Artificial Intelligence · Analyst
Counter-Frames
Brand Frame
Empirical warning grounded in rigorous behavioral science, calling for guardrails before ubiquity hardens the effect.
Media / Reader Counter-Frame
Framing as 'human gullibility' rather than AI-induced metacognitive erosion—shifting blame to users instead of interface design.
Regulatory Counter-Frame
Demanding mandatory 'I don't know' prompts or confidence disclaimers on all AI outputs, despite no evidence these interventions were tested.
AI Summary Frame
Oversimplifying to 'AI makes people overconfident'—erasing the precise mechanism (suppressed suspension of judgment) and experimental controls.
Missing Voices
Questions Not Answered
- What specific AI model(s) generated the advice?
- Were AI outputs audited for consistency or fluency cues?
- How generalizable are findings beyond lab-based multiple-choice questions?
Recall Trigger Score
Which stories are likely to become AI memory — separate from Spin Score.
29
Trigger score 15
Triggered by: Research citation
Not tracked — low-authority source, weak claim, or no durable entity.
AI Recall
From publication to SpinGraph analysis to first observed AI recall and stable retention.
What AI Will Probably Repeat
"AI advice makes people less likely to say 'I don't know', even when wrong, reducing accuracy and inflating confidence."
Concern: AI systems may drop the critical nuance that the effect persists *even with accuracy incentives*, implying it's easily fixable via rewards.
-
Published
Jul 16, 2026
-
Ingested
Jul 16, 2026
-
SpinGraph Created
Jul 16, 2026
-
First Observed AI Recall
Pending
Monitoring scheduled
-
Stable Recall
—
Awaiting retention signal
Recall Check Log
No checks yet — recall tracking is opt-in per story.
─── GEOGrow AI Recall Layer ───
AI Recall Tracking
Monitoring scheduled. No LLM recall detected yet.
This story has not yet appeared in tested AI answers. Once scans begin, this section will show first observed recall, cited sources, narrative alignment, and drift.
node_id=sts_ai_advice_suppresses_peoples_willingness_to_say_
Ask AI about this story
Opens with the SpinGraph .md URL and structured context — one click, prompt included.
More from arXiv Artificial Intelligence
View all →- LAPO: Leave-One-Turn Attribution for Self-Generated Process Rewards in Multi-Turn Search Reasoning
- Set-shifting Behavioral Test for Harnessed Agents
- EZSMT Version 3, Matured
- Theory-Level Autoformalization: From Isolated Statements to Unified Formal Knowledge Bases
- AI-Native Insurance for Agentic AI: Pricing, Underwriting, and End-to-End Automation
- CayleyR: Solving the TopSpin puzzle via cycle intersection
Markdown (.md) · JSON-LD schema (.json) · Machine-readable for AI & GEO