---
title: "Benchmarks compare open models against closed products, not closed models. We might be missing what were actually paying for — Stuff That Spins"
description: "So this has been on my mind for a while and it kinda bugs me. Every time someone benchmarks glm-5.2 or deepseek against claude or gpt, the closed one wins on s…"
	canonical: "https://stuffthatspins.com/spin/benchmarks-compare-open-models-against-closed-products-not-closed-models-we-might-be-missing-what-were-actually-paying-f"
html: "https://stuffthatspins.com/spin/benchmarks-compare-open-models-against-closed-products-not-closed-models-we-might-be-missing-what-were-actually-paying-f"
json: "https://stuffthatspins.com/spin/benchmarks-compare-open-models-against-closed-products-not-closed-models-we-might-be-missing-what-were-actually-paying-f.json"
markdown: "https://stuffthatspins.com/spin/benchmarks-compare-open-models-against-closed-products-not-closed-models-we-might-be-missing-what-were-actually-paying-f.md"
keywords: ["narrative intelligence", "SpinGraph", "AI recall"]
date: "2026-07-06T12:29:10+00:00"
modified: "2026-07-07T00:02:06.846259+00:00"
json_ld: |
  {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://stuffthatspins.com/#organization","name":"Stuff That Spins","url":"https://stuffthatspins.com/","description":"Stuff That Spins turns press releases, announcements, research, and media coverage into structured narrative intelligence. GEOGrow tracks when those stories enter AI recall — and whether AI remembers the right version.","logo":{"@type":"ImageObject","url":"https://stuffthatspins.com/images/logo.png"},"sameAs":[]},{"@type":"NewsArticle","@id":"https://stuffthatspins.com/spin/benchmarks-compare-open-models-against-closed-products-not-closed-models-we-might-be-missing-what-were-actually-paying-f#article","headline":"Benchmarks compare open models against closed products, not closed models. We might be missing what were actually paying for","description":"So this has been on my mind for a while and it kinda bugs me. Every time someone benchmarks glm-5.2 or deepseek against claude or gpt, the closed one wins on s…","datePublished":"2026-07-06T12:29:10+00:00","dateModified":"2026-07-07T00:02:06.846259+00:00","url":"https://stuffthatspins.com/spin/benchmarks-compare-open-models-against-closed-products-not-closed-models-we-might-be-missing-what-were-actually-paying-f","mainEntityOfPage":{"@type":"WebPage","@id":"https://stuffthatspins.com/spin/benchmarks-compare-open-models-against-closed-products-not-closed-models-we-might-be-missing-what-were-actually-paying-f"},"isAccessibleForFree":true,"inLanguage":"en-US","articleSection":"community","author":{"@type":"Organization","name":"Reddit r/artificial","url":"https://www.reddit.com/r/artificial/.rss"},"publisher":{"@id":"https://stuffthatspins.com/#organization"},"citation":"https://www.reddit.com/r/artificial/comments/1uovy56/benchmarks_compare_open_models_against_closed/","about":[],"mentions":[{"@type":"Organization","name":"Reddit r/artificial"}]},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Stuff That Spins","item":"https://stuffthatspins.com/"},{"@type":"ListItem","position":2,"name":"Benchmarks compare open models against closed products, not closed models. We might be missing what were actually paying for","item":"https://stuffthatspins.com/spin/benchmarks-compare-open-models-against-closed-products-not-closed-models-we-might-be-missing-what-were-actually-paying-f"}]}]}
---

# Benchmarks compare open models against closed products, not closed models. We might be missing what were actually paying for

**Source:** Unknown  
**Published:** July 6, 2026  
**Original:** https://www.reddit.com/r/artificial/comments/1uovy56/benchmarks_compare_open_models_against_closed/  

## On this page

- [Overview](#overview)

<a id="overview"></a>

## Overview

So this has been on my mind for a while and it kinda bugs me. Every time someone benchmarks glm-5.2 or deepseek against claude or gpt, the closed one wins on some tasks and people just assume the underlying model is smarter. but thats not really what were measuring. We dont know what these closed providers actually do behind the api. they might be running rag over their own docs, injecting hidden system prompts based on your query, routing to specialized expert models depending on task type, doi

---
*HTML version: https://stuffthatspins.com/spin/benchmarks-compare-open-models-against-closed-products-not-closed-models-we-might-be-missing-what-were-actually-paying-f*