How Cadle ranks

How ranking goes live

1. Workers pull upstreams — BenchLM, Artificial Analysis, LMArena, OpenRouter / HF prices, Hugging Face cards, and the unified Model Matrix. Epoch AI Benchmarking Hub scores are used under CC-BY. Soft sources can skip without wiping the catalog.
2. Snapshot write — ranked models land in a cached worker snapshot with Cadle Cap ranks, Matrix ranks, matrix cells, and methodology strings. Desk APIs read that snapshot only.
3. HTTP + client cache — list views use slim projections and short TTL / stale-while-revalidate so warm loads stay in the 1–2s band.
4. Cadle 50 boards — Models recompute from the latest Cap ranking. Apps / Tools / Companies use fixed editorial review scores (no artificial drift). Boards refresh from the newest worker snapshot on a 6-hour UTC cadence.
5. User reweighting — on /models you can change capability weights live; that reorders the Cadle Cap board in the browser without rewriting Matrix Overall.

Live ≠ continuous streaming scores. Live means each board recomputes from the newest worker snapshot (models / matrix / pricing) or the newest 6-hour UTC key (apps / tools / companies). Cards always show which board you are looking at. Cadle Cap never invents empty Matrix cells.

How we rank models (Cadle Cap)

Cadle Cap is a peer rank built from Matrix cells. It answers: how elite is this model inside today's comparable pool?

1. Read the same cells as Model Matrix: Intelligence, Coding, Agentic, Math, Knowledge, Reasoning, Human preference.
2. Need at least six filled expert cells. Pre-release / xhigh SKUs cannot own Cap.
3. Convert each factor to a percentile inside the pool that also has that factor. Intelligence 97 ≈ top 3% — not BenchLM's absolute 82.8.
4. Thin coverage shrinks toward the board median. A model with only GPQA + Math cannot beat a fully measured flagship. Default blend is Balanced.
5. Confidence = 50% category coverage + 25% source breadth + 25% cross-source agreement. Confidence never raises or lowers capability rank.

Price, OpenRouter popularity, followers, news mentions, and star ratings never enter Cadle Cap.

How we rank Model Matrix

Model Matrix is the datasheet, not a prettier Cap. Neither board is “better” — Cap compares peers, Matrix shows what was measured. Cards use the same cells as Benchmarks → Matrix:

• Overall · AA IQ — headline composites only. Overall = BenchLM overall; AA IQ = Artificial Analysis Intelligence Index. They stay visible so you can see a model that wins an index but loses expert boards. They are not part of AVG.
• Coding · Agentic · MMLU · GPQA · Math · LiveCodeBench · HumanEval · GSM8K · Truth · Reason · ARC-AGI — expert cells. Each cell is a quality-tiered median: primary labs (BenchLM / AA / LiveBench / Epoch / SWE / BigCode / BFCL) outrank secondary indexes and community catalogs. A dash means no measurement — we do not invent Accuracy, Reliability, Safety, Cost, or Reason-from-Elo.
• Coverage AVG — importance-weighted mean over filled expert cells only (blanks skipped). A model needs at least six expert cells to earn a Matrix #.

Active matrix families: BenchLM, Artificial Analysis, LLM Stats, Epoch AI (CC-BY), LMArena, LiveBench, Vectara, BFCL, Open LLM Leaderboard, SWE-bench, and BigCode. Analysis graphs and Benchmark systems read the same merge. Cadle Cap and Matrix AVG can disagree — that is intentional. Epoch AI data is reproduced under CC-BY; see epoch.ai/benchmarks/use-this-data.

How family boards measure

Language is not the only Cap. Each family ranks only on axes it actually measures. A model can sit on the catalog with scores and still be unranked on Cap if it fails that board's fill rule.

• Language — Intelligence, Coding, Agentic, Math, Knowledge, Reasoning, Human preference. Six expert cells.
• Vision — MMMU, MathVista, ChartQA, OCRBench, AI2D, RealWorldQA, MMBench, MMVet, Hallusion, TextVQA, Human preference.
• Image — Human preference, Realism (AA category), Aesthetics (AA category), Creativity (AA category), Prompt fidelity (AA category).
• Video — Realism, Motion, Temporal consistency, Creativity, Prompt fidelity, Human preference.
• Audio — Transcription (short-form), Transcription (long-form), Multilingual ASR, Human preference.
• Agents — Tool calling, Multi-turn, Web research, Memory, Software eng., Agent tasks, Human preference.

How we rank companies

Company ranks are an editorial Cadle 50 board, not a market-cap ticker and not a model-score average.

1. Editorial desk seeds a base score (≈55–99) from platform leverage: compute / cloud distribution, frontier labs, open-weight impact, enterprise reach, and full-stack control (chip → model → product).
2. Scores are fixed editorial reviews (no artificial drift). Trends stay flat until the desk revises a score. Boards refresh from the worker snapshot on a 6-hour UTC cadence.
3. Company rank is not the mean of that lab's model Caps. A lab can ship #1 models and still sit below a compute platform on the company board.

See Feed → Cadle 50 → Companies. Next refresh: next 6-hour UTC bucket.

How we rank tools

Tools = developer platforms and infra (frameworks, hubs, SDKs, vector DBs, eval / observability). Same Cadle 50 editorial mechanics as companies, with a different rubric:

• Developer adoption and default-stack status (e.g. LangChain, Hugging Face Hub).
• Production readiness — streaming DX, RAG / agent primitives, deploy path.
• Open vs proprietary leverage and ecosystem gravity.
• Fixed editorial scores; snapshot refresh every 6 hours UTC — no artificial score noise.

Studio / directory pages may also show trending or visit signals — those are engagement rails, not the Cadle 50 Tools score.

How we rank apps

Apps = products people open daily (ChatGPT, Cursor, Midjourney, Copilot, …). Editorial Cadle 50 board:

• Product quality and workflow fit for the stated job.
• Distribution and retention (who actually uses it as a default).
• Category leadership (coding IDE, answer engine, creative video, voice, …).
• Same fixed editorial scores and 6-hour snapshot refresh as Tools and Companies.

An app can rank above its underlying model provider — product packaging is scored separately from model Cap.

How we rank pricing

Cadle Value Rank is a 0–100 value score on gateway rows. It is not Cadle Cap. List price never enters a capability board.

1. Dollars — OpenRouter, LiteLLM, Together / DeepInfra / Fireworks / Groq, catalog DB, then BenchLM pricing.json for published $/1M.
2. Intelligence — Artificial Analysis index first. If AA is missing, BenchLM overall fills the intel overlay so pts/$ is not blank.
3. Latency / speed — AA tokens/s and TTFT when present, else BenchLM speed.json.
4. Pts/$ — intelligence / max($0.05, in+out). The $0.05 floor stops free rows from looking infinitely cheap. Displayed pts/$ is capped at 120 so free / near-free rows do not dominate the column; Rank then blends value, speed, latency, and source trust.

Model and provider portfolios show the same $ and pts/$ strip. DeepSeek API SKUs (chat / reasoner) join V3 / R1 catalog names; compact slugs without hyphens are not used as profile ids.

How we rank headlines & briefs

News quality score blends source tier trust, high-signal vs low-signal language, summary substance, title quality (anti-listicle), freshness buckets, and per-source caps. Headlines take the top band with max 2 per source; briefs are the next tier with denser substance requirements.

What never enters a capability score

• User star ratings (shown separately on cards / profiles — never Cap).
• Followers, likes, or feed engagement.
• News mention counts (used for momentum rails only).
• List price alone (pricing has its own board).
• Editorial Cadle 50 scores for apps / tools / companies do not rewrite model Caps.

Changelog

• pulse-capability-v11 · 2026-09-09

  Model Matrix quality merge: primary labs (BenchLM / AA / LiveBench / Epoch / SWE / BigCode / BFCL) outrank HF/PwC/other for each cell. Overall = BenchLM overall only; AA IQ = AA intelligence only. Stop inventing Reason from LMSYS Elo. Truth column no longer mixes Vectara/Winogrande into HellaSwag.

• pulse-capability-v10.1 · 2026-09-04

  Language Cap needs six filled expert cells; empty Matrix cells are skipped in Cap (never invent zeros). Cap is a peer rank, Matrix is the datasheet. Unranked catalog rows keep scores and stay searchable. Pricing Value Rank may overlay BenchLM overall + speed when AA is missing; price never enters Cap.

• pulse-capability-v10 · 2026-08-20

  Split family Cap boards (language, vision, image, video, audio, agents). Stopped inventing Accuracy, Reliability, Safety, or Cost cells. LiveCodeBench is official generation Pass@1 only.

• pulse-capability-v9.2 · 2026-08-14

  Persisted per-family matrix sources on catalog records; cards distinguish single-source figures from multi-source medians. Methodology no longer lists silent connectors as active. Worker fails the job on thin-write, CI local-only upsert, or required-source timeout.

• pulse-capability-v9.1 · 2026-08-14

  Disclosed proxy-estimated matrix cells and thinner evidence on curated specialty boards.

• pulse-capability-v8.0 · 2026-08-01

  Separated capability from confidence, switched factors to within-pool percentiles, removed composite double-counting, and made weights user-adjustable.

• pulse-capability-v7 · 2026-08-01

  Anchored quality to BenchLM/AA published composites and shrunk thin evidence toward the field median.