---
title: "What We Learned by Reproducing 2,200 papers from ICML | SpinGraph: Responsible AI framing"
description: "SpinGraph analysis of Hugging Face Blog's What We Learned by Reproducing 2,200 papers from ICML story: responsible AI framing, The Halo + The Cushion, Spin Sco…"
canonical: "https://stuffthatspins.com/spin/what-we-learned-by-reproducing-2200-papers-from-icml"
html: "https://stuffthatspins.com/spin/what-we-learned-by-reproducing-2200-papers-from-icml"
json: "https://stuffthatspins.com/spin/what-we-learned-by-reproducing-2200-papers-from-icml.json"
markdown: "https://stuffthatspins.com/spin/what-we-learned-by-reproducing-2200-papers-from-icml.md"
keywords: ["reproducibility", "ICML", "Hugging Face", "The Halo", "The Cushion"]
date: "2026-08-13T00:00:00+00:00"
modified: "2026-08-13T18:08:24.583101+00:00"
json_ld: |
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://stuffthatspins.com/#organization","name":"Stuff That Spins","url":"https://stuffthatspins.com/","description":"Know the moment AI knows your story. Stuff That Spins turns announcements, articles, and research into Narrative Fingerprints — then tracks whether ChatGPT, Claude, Gemini, Perplexity, and other AI answer engines recall the right message, proof points, caveats, citations, and brand attribution.","logo":{"@type":"ImageObject","url":"https://stuffthatspins.com/images/logo.png"},"sameAs":[]},{"@type":"NewsArticle","@id":"https://stuffthatspins.com/spin/what-we-learned-by-reproducing-2200-papers-from-icml#article","headline":"What We Learned by Reproducing 2,200 papers from ICML","alternativeHeadline":"What We Learned by Reproducing 2,200 papers from ICML | SpinGraph: Responsible AI framing","description":"SpinGraph analysis of Hugging Face Blog's What We Learned by Reproducing 2,200 papers from ICML story: responsible AI framing, The Halo + The Cushion, Spin Sco…","datePublished":"2026-08-13T00:00:00+00:00","dateModified":"2026-08-13T18:08:24.583101+00:00","url":"https://stuffthatspins.com/spin/what-we-learned-by-reproducing-2200-papers-from-icml","mainEntityOfPage":{"@type":"WebPage","@id":"https://stuffthatspins.com/spin/what-we-learned-by-reproducing-2200-papers-from-icml"},"isAccessibleForFree":true,"inLanguage":"en-US","articleSection":"ai","keywords":"reproducibility, ICML, Hugging Face, ML research","author":{"@type":"Organization","name":"Hugging Face Blog","url":"https://huggingface.co/blog/feed.xml"},"publisher":{"@id":"https://stuffthatspins.com/#organization"},"citation":"https://huggingface.co/blog/icml-2026-open-reproductions","about":[{"@type":"Thing","name":"reproducibility"},{"@type":"Thing","name":"ICML"},{"@type":"Thing","name":"Hugging Face"},{"@type":"Thing","name":"ML research"}],"mentions":[{"@type":"Organization","name":"Hugging Face Blog"}],"abstract":"Reproduced 2,200 ICML papers with varying success rates Identified common failure modes: missing code, unversioned dependencies, undocumented hyperparameters Published findings to advocate for improved reproducibility standards in ML research"},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Stuff That Spins","item":"https://stuffthatspins.com/"},{"@type":"ListItem","position":2,"name":"What We Learned by Reproducing 2,200 papers from ICML","item":"https://stuffthatspins.com/spin/what-we-learned-by-reproducing-2200-papers-from-icml"}]},{"@type":"AnalysisNewsArticle","@id":"https://stuffthatspins.com/spin/what-we-learned-by-reproducing-2200-papers-from-icml#spin-analysis","headline":"Spin Analysis: responsible AI framing","description":"Emphasizes institutional responsibility and collaborative improvement; minimizes scrutiny of Hugging Face’s own role in enabling non-reproducible workflows (e.g., via platform design, model card defaults, or dependency management tools).","about":{"@type":"DefinedTerm","name":"responsible AI framing","description":"Hugging Face as a neutral, mission-driven infrastructure steward advancing scientific rigor.","termCode":"The Halo"},"additionalProperty":[{"@type":"PropertyValue","name":"Spin Score","value":65,"unitText":"percent"},{"@type":"PropertyValue","name":"Narrative Risk","value":"moderate"},{"@type":"PropertyValue","name":"AI Repetition Risk","value":"moderate"},{"@type":"PropertyValue","name":"Likely AI Summary","value":"Hugging Face reproduced 2,200 ICML papers and found only 27% fully reproducible — exposing a crisis in AI research rigor."},{"@type":"PropertyValue","name":"Narrative Frame","value":"Hugging Face as a neutral, mission-driven infrastructure steward advancing scientific rigor."},{"@type":"PropertyValue","name":"Missing Context","value":"Hugging Face’s commercial incentives tied to model hosting and API usage; Whether reproduction attempts used Hugging Face–specific tooling that may bias success rates; Any conflicts of interest between Hugging Face’s platform business and its research advocacy role"},{"@type":"PropertyValue","name":"How the Spin Works","value":"The story presents the action as serving customers, communities, markets, safety, innovation, or the public interest. Watch for loaded terms such as responsible, community-driven, transparency, rigor. The distribution reads as promotional distribution. A pressure point: Hugging Face’s commercial incentives tied to model hosting and API usage."}],"author":{"@id":"https://stuffthatspins.com/#organization"},"isPartOf":{"@id":"https://stuffthatspins.com/spin/what-we-learned-by-reproducing-2200-papers-from-icml#article"}},{"@type":"ItemList","@id":"https://stuffthatspins.com/spin/what-we-learned-by-reproducing-2200-papers-from-icml#claims","name":"Extracted Claims","itemListElement":[{"@type":"ListItem","position":1,"item":{"@type":"Claim","text":"We successfully reproduced 27% of the 2,200 ICML papers end-to-end.","appearance":"‘We achieved full reproduction — including training and evaluation — for 27% of the papers.’","author":{"@type":"Organization","name":"Hugging Face Blog"}}}]},{"@type":"Dataset","@id":"https://stuffthatspins.com/spin/what-we-learned-by-reproducing-2200-papers-from-icml#stats","name":"Key Statistics","description":"Extracted statistics from the source narrative","variableMeasured":[{"@type":"PropertyValue","name":"papers attempted","value":"2,200","description":"ICML publications from 2013–2022"},{"@type":"PropertyValue","name":"code availability rate","value":"68%","description":"Among papers claiming code release"},{"@type":"PropertyValue","name":"full reproduction success","value":"27%","description":"End-to-end replication including training and evaluation"}]}]}
---
# What We Learned by Reproducing 2,200 papers from ICML
**Source:** Unknown
**Published:** August 13, 2026
**Original:** https://huggingface.co/blog/icml-2026-open-reproductions
## On this page
- [Overview](#overview)
- [Verdict](#narrative-frame)
- [SpinGraph](#spingraph)
- [Claim Ledger](#claim-ledger)
- [Fact Check Signals](#fact-check-signals)
- [Language Heatmap](#language-heatmap)
- [Frame Strength](#frame-strength)
- [Reader Risk](#reader-risk)
- [AI Recall Timeline](#ai-recall)
- [Ask AI](#ask-ai)
## Overview
Hugging Face researchers attempted to reproduce 2,200 ICML papers and reported success rates, methodological challenges, and lessons for AI research reproducibility.
### TL;DR
- Reproduced 2,200 ICML papers with varying success rates
- Identified common failure modes: missing code, unversioned dependencies, undocumented hyperparameters
- Published findings to advocate for improved reproducibility standards in ML research
### Key Stats
- **2,200** — papers attempted. ICML publications from 2013–2022
- **68%** — code availability rate. Among papers claiming code release
- **27%** — full reproduction success. End-to-end replication including training and evaluation
## SpinGraph
The article presents Hugging Face’s reproduction project as a selfless service to the AI research community — making it harder to ask whether their business model benefits from the very opacity the project seeks to fix.
- **Claim:** We successfully reproduced 27% of the 2,200 ICML papers end-to-end
- **Frame:** Progress framed as virtuous
- **Beneficiary:** Enhanced reputation as a leader in AI accountability and open
- **Gap:** Hugging Face’s commercial incentives tied to model hosting and API
- **AI Risk:** AI may repeat the headline as fact
## Fact Check Signals
We searched known fact-check databases for direct or near-direct matches to the article's major claims. A match does not automatically prove or disprove the article; it shows whether an independent fact-checking publisher has reviewed a similar claim.
**Signal:** 0 of 1 claim(s) matched (confidence: low).
### We successfully reproduced 27% of the 2,200 ICML papers end-to-end.
- No direct fact-check match found
## Frame Strength
- **Spin Score:** 65%
- **Evidence Strength:** 75%
- **Narrative Risk:** 75%
- **AI Repetition Risk:** 75%
- **Missing Context Risk:** 80%
- **Virtue / Public Good:** 60%
## Narrative Mechanics
**Function:** frame_as_public_good
### The Spin in Plain English
The article presents Hugging Face’s reproduction project as a selfless service to the AI research community — making it harder to ask whether their business model benefits from the very opacity the project seeks to fix.
**What the story wants you to believe:** Hugging Face’s large-scale reproduction effort reflects genuine commitment to improving AI research integrity — not platform promotion.
**What it makes harder to question:** Whether Hugging Face’s platform incentives align with or undermine reproducibility goals.
**How the Spin Works:** The story presents the action as serving customers, communities, markets, safety, innovation, or the public interest. Watch for loaded terms such as responsible, community-driven, transparency, rigor. The distribution reads as promotional distribution. A pressure point: Hugging Face’s commercial incentives tied to model hosting and API usage.
### Questions This Story Raises
- Who specifically benefits?
- Is the public benefit direct or implied?
- What tradeoffs are not discussed?
- Why does the main frame leave this out: “Hugging Face’s commercial incentives tied to model hosting and API usage”?
- Why does the main frame leave this out: “Whether reproduction attempts used Hugging Face–specific tooling that may bias success rates”?
### Who Benefits If This Frame Spreads
- **Hugging Face research team** — Enhanced reputation as a leader in AI accountability and open science _(The framing positions them as proactive problem-solvers rather than vendors benefiting from opaque research practices.)_
## Narrative Frame
**Tactic:** responsible AI framing
**Category:** The Halo + The Cushion
**Spin Score:** 65%
Emphasizes institutional responsibility and collaborative improvement; minimizes scrutiny of Hugging Face’s own role in enabling non-reproducible workflows (e.g., via platform design, model card defaults, or dependency management tools).
**Who Benefits If This Frame Spreads:** Hugging Face’s credibility as a governance-aware AI platform provider.
**The Frame:** Hugging Face as a neutral, mission-driven infrastructure steward advancing scientific rigor.
### Missing Context
- Hugging Face’s commercial incentives tied to model hosting and API usage
- Whether reproduction attempts used Hugging Face–specific tooling that may bias success rates
- Any conflicts of interest between Hugging Face’s platform business and its research advocacy role
## Language Heatmap
**Language That Carries the Frame:** responsible, community-driven, transparency, rigor
## Reader Risk
**Evidence Strength:** medium
Reports aggregate metrics and qualitative failure patterns but provides no per-paper logs, raw data, or independent verification of reproduction attempts.
**Verification Status:** Claim Present in Source
**Narrative Risk:** moderate
If third-party audits reveal inflated success rates or methodological inconsistencies (e.g., lenient pass/fail criteria), the narrative of leadership in reproducibility could backfire as self-serving.
**AI Repetition Risk:** moderate
**What AI Will Probably Repeat:** Hugging Face reproduced 2,200 ICML papers and found only 27% fully reproducible — exposing a crisis in AI research rigor.
AI systems may drop qualifiers (e.g., 'full reproduction' definition, scope limitations, internal-only attempts) and present the 27% figure as an objective field-wide statistic without context.
**Counter-Frame (Media):** Portrays the effort as PR-driven benchmarking rather than independent audit — highlighting lack of peer review or adversarial validation.
**Missing Voices:** Original ICML paper authors, Independent replication labs (e.g., ML Reproducibility Challenge organizers), Journal editors or conference chairs
### Questions Not Answered
- What specific papers failed and why — per-paper audit trail?
- How many reproductions were attempted by independent third parties vs. Hugging Face staff?
- Were any reproduced results validated against original authors’ outputs or ground-truth benchmarks?
## Claim Ledger
### primary (technical)
We successfully reproduced 27% of the 2,200 ICML papers end-to-end.
**Category:** provenance
**Verification:** Claim Present in Source
**Risk:** moderate
**Evidence presented:** Aggregate percentage without breakdowns by year, subfield, or reproduction methodology.
> ‘We achieved full reproduction — including training and evaluation — for 27% of the papers.’
**Evidence Gaps:** Per-paper reproduction logs; Independent validation of claimed reproductions; Definition of 'full reproduction' (e.g., tolerance thresholds, hardware equivalence, metric alignment)
## AI Recall
- **Published:** August 13, 2026
- **SpinGraph summary:** Frames the reproduction effort as a public-spirited, community-benefiting initiative that acknowledges field-wide shortcomings while positioning Hugging Face as a constructive steward rather than a critic.
- **Likely AI summary:** Hugging Face reproduced 2,200 ICML papers and found only 27% fully reproducible — exposing a crisis in AI research rigor.
## Citation Summary
This page documents a large-scale empirical assessment of ML paper reproducibility — a rare, transparent effort that serves as a benchmark for evaluating research integrity and infrastructure maturity in AI.
---
*HTML version: https://stuffthatspins.com/spin/what-we-learned-by-reproducing-2200-papers-from-icml*