Benchmarks Are Not Validation: A System-Level View of Financial LLM Applications

Every story gets a Spin Verdict: a primary spin type (and secondary when the framing blends), a specific tactic name, and a score for how strongly the narrative is steered. Examples beneath each type are tactics, not separate categories.

Spin Score measures how strongly the framing steers the narrative (0–100%). Higher scores mean more deliberate spin tactics — loaded language, selective emphasis, or omitted context. Many stories blend two types (e.g. Halo + Hype).

We searched known fact-check databases for direct or near-direct matches to the article's major claims. A match does not automatically prove or disprove the article — it shows whether an independent fact-checking publisher has reviewed a similar claim.

• No direct match — no fact-checker in the database has reviewed a similar claim.
• Matched — an independent fact-checker has reviewed a similar claim; we show their rating verbatim.
• Conflicting coverage — fact-checkers disagree on a similar claim.

This is evidence discovery, not an automated truth score. Ratings and wording come directly from the publishing fact-checker.

Frame Strength decomposes the overall spin into individual signals. Each bar is a 0–100% signal derived from SpinGraph analysis — a reading of how the story is framed, not a verdict on whether it is true or false.

Spin Score

How strongly the story pushes a particular narrative frame — the combined weight of loaded language, selective emphasis, and omitted context. 0% reads as neutral reporting; higher means more deliberate spin.

• 0–33% Low — Largely neutral reporting; little detectable framing.
• 34–66% Moderate — Noticeable slant — the story leans a particular way.
• 67–100% High — Heavily framed; the angle drives the piece.

Evidence Strength

How well the story’s claims are backed by verifiable, independent evidence rather than assertion or promotion. Higher is stronger. Low scores flag claims that rest on the source’s own word.

• 0–33% Weak — Claims rest mostly on assertion or a single interested source.
• 34–66% Mixed — Some verifiable backing, but key claims are thinly sourced.
• 67–100% Strong — Well supported by independent, checkable evidence.

Narrative Risk

The chance the framing shapes reader perception faster than the underlying facts justify — how misleading the overall story could be even when individual facts are accurate.

• 0–33% Low — Framing stays close to what the facts support.
• 34–66% Moderate — Framing outruns the facts in places — read with care.
• 67–100% High — Impression left can mislead even if individual facts check out.

AI Repetition Risk

How likely AI answer engines (search, chatbots) are to absorb and repeat this story’s framing as fact when summarizing the topic later.

• 0–33% Low — Framing is unlikely to propagate through AI summaries.
• 34–66% Moderate — Some risk the slant gets echoed as fact.
• 67–100% High — Framing is sticky and likely to be repeated as fact.

Missing Context Risk

How much important context the story leaves out, based on the omitted-context signals SpinGraph detected.

• 0–33% Low — Little material context appears to be omitted.
• 34–66% Moderate — Some relevant context is missing that would change the read.
• 67–100% High — Key context is left out, skewing the takeaway.

Momentum / Inevitability · Virtue / Public Good

Framing-tactic intensities that appear only when the story leans on those specific spin patterns (e.g. “the future is already here” or “this is for the public good”).

• 0–33% Low — The tactic is barely present.
• 34–66% Moderate — The tactic shapes part of the framing.
• 67–100% High — The tactic is a dominant part of the pitch.

Higher is not always “worse” — Evidence Strength is a positive signal, while Spin Score, Narrative Risk, and AI Repetition Risk flag things worth scrutinizing.