The open results

Agreement with the owner's fit observation

15 questions. 13 fit-agreement prompts. 2 confidence variants, kept separate. Every historical attempt remains visible.

Batch karlby-98-2026-10-19-first-pass · published 2026-10-19T21:32:42.981Z · 4 source runs

models in the published roster: 30
selected completed final answers: 402 / 450
selected answers with published review: 402
models with all 13 fit prompts reviewed: 19

A batch at a glance

Fit agreement, model by model.

Only complete, reviewed fit coverage earns an official rank. Partial coverage is visible, not converted into wrong answers.

Jump to a model archive · all 30

YES · 1 point
NO · 0 points
Uncertain · 0 points
Missing / unreviewed · not scored

1
Google: Gemini 3.5 Flash Lite
6 / 13 fit agreement · rank 1 · 13/13 reviewed
YES 6 · NO 7 · uncertain 0 · missing or unreviewed 0
2
StepFun: Step 5 Preview
5 / 13 fit agreement · rank 2 · 13/13 reviewed
YES 5 · NO 5 · uncertain 3 · missing or unreviewed 0
3
DeepSeek: DeepSeek V4.1 Flash 1017
4 / 13 fit agreement · rank 3 · 13/13 reviewed
YES 4 · NO 7 · uncertain 2 · missing or unreviewed 0
3
Meta: Muse Wonder
4 / 13 fit agreement · rank 3 · 13/13 reviewed
YES 4 · NO 7 · uncertain 2 · missing or unreviewed 0
5
Anthropic: Claude Sonnet 5.5
3 / 13 fit agreement · rank 5 · 13/13 reviewed
YES 3 · NO 5 · uncertain 5 · missing or unreviewed 0
6
Meta: Muse Glimmer 30B
2 / 13 fit agreement · rank 6 · 13/13 reviewed
YES 2 · NO 11 · uncertain 0 · missing or unreviewed 0
6
OpenAI: GPT-6 Luna
2 / 13 fit agreement · rank 6 · 13/13 reviewed
YES 2 · NO 6 · uncertain 5 · missing or unreviewed 0
6
SpaceXAI: Grok 4.7
2 / 13 fit agreement · rank 6 · 13/13 reviewed
YES 2 · NO 9 · uncertain 2 · missing or unreviewed 0
9
Anthropic: Claude Haiku 5.5
1 / 13 fit agreement · rank 9 · 13/13 reviewed
YES 1 · NO 9 · uncertain 3 · missing or unreviewed 0
10
Anthropic: Claude Opus 5.5
0 / 13 fit agreement · rank 10 · 13/13 reviewed
YES 0 · NO 12 · uncertain 1 · missing or unreviewed 0
10
Google: Gemini 3.8 Flash
0 / 13 fit agreement · rank 10 · 13/13 reviewed
YES 0 · NO 13 · uncertain 0 · missing or unreviewed 0
10
Google: Gemini 4 Argon
0 / 13 fit agreement · rank 10 · 13/13 reviewed
YES 0 · NO 13 · uncertain 0 · missing or unreviewed 0
10
IBM: Granite 4.2 8B
0 / 13 fit agreement · rank 10 · 13/13 reviewed
YES 0 · NO 12 · uncertain 1 · missing or unreviewed 0
10
Meta: Muse Spark Open
0 / 13 fit agreement · rank 10 · 13/13 reviewed
YES 0 · NO 13 · uncertain 0 · missing or unreviewed 0
10
Mistral: Ministral 3 3B 2512
0 / 13 fit agreement · rank 10 · 13/13 reviewed
YES 0 · NO 13 · uncertain 0 · missing or unreviewed 0
10
NVIDIA: Nemotron 3.5 Lightning
0 / 13 fit agreement · rank 10 · 13/13 reviewed
YES 0 · NO 11 · uncertain 2 · missing or unreviewed 0
10
NVIDIA: Nemotron 3.5 Super VL
0 / 13 fit agreement · rank 10 · 13/13 reviewed
YES 0 · NO 11 · uncertain 2 · missing or unreviewed 0
10
OpenAI: GPT-6.1 Sol
0 / 13 fit agreement · rank 10 · 13/13 reviewed
YES 0 · NO 3 · uncertain 10 · missing or unreviewed 0
10
Upstage: Solar Mini 4
0 / 13 fit agreement · rank 10 · 13/13 reviewed
YES 0 · NO 12 · uncertain 1 · missing or unreviewed 0
Unranked
Cohere: Command A+
Unranked · observed YES 0/12 reviewed · fit coverage 12/13
YES 0 · NO 12 · uncertain 0 · missing or unreviewed 1
Unranked
DeepSeek: DeepSeek V4 Pro 0813
Unranked · observed YES 5/9 reviewed · fit coverage 9/13
YES 5 · NO 4 · uncertain 0 · missing or unreviewed 4
Unranked
inclusionAI: Ling 3.1 Flash
Unranked · observed YES 4/12 reviewed · fit coverage 12/13
YES 4 · NO 6 · uncertain 2 · missing or unreviewed 1
Unranked
LiquidAI: LFM2.5-24B-A2B
Unranked · observed YES 0/12 reviewed · fit coverage 12/13
YES 0 · NO 11 · uncertain 1 · missing or unreviewed 1
Unranked
Mistral: Mistral Large 4
Unranked · observed YES 5/12 reviewed · fit coverage 12/13
YES 5 · NO 6 · uncertain 1 · missing or unreviewed 1
Unranked
Qwen: Qwen3.5-9B
Unranked · observed YES 0/0 reviewed · fit coverage 0/13
YES 0 · NO 0 · uncertain 0 · missing or unreviewed 13
Unranked
Qwen: Qwen3.8 27B
Unranked · observed YES 0/5 reviewed · fit coverage 5/13
YES 0 · NO 5 · uncertain 0 · missing or unreviewed 8
Unranked
Qwen: Qwen3.8 Flash
Unranked · observed YES 1/6 reviewed · fit coverage 6/13
YES 1 · NO 4 · uncertain 1 · missing or unreviewed 7
Unranked
Qwen: Qwen3.8 Max Prime
Unranked · observed YES 4/12 reviewed · fit coverage 12/13
YES 4 · NO 5 · uncertain 3 · missing or unreviewed 1
Unranked
Xiaomi: MiMo-V2.6-Flash
Unranked · observed YES 1/5 reviewed · fit coverage 5/13
YES 1 · NO 3 · uncertain 1 · missing or unreviewed 8
Unranked
Z.ai: GLM 5.3 Prime
Unranked · observed YES 10/12 reviewed · fit coverage 12/13
YES 10 · NO 2 · uncertain 0 · missing or unreviewed 1

Each full bar represents 13 fit prompts. YES, NO, uncertain, and missing are different categories; missing includes incomplete and unreviewed selected answers. Model names link to exact answers and attempt histories.

What this score does—and does not—say

Score 0–13: each completed, reviewed fit prompt labeled YES earns 1. NO and uncertain earn 0. They remain different labels.
Official eligibility: all 13 fit prompts must be completed and reviewed. Partial models show observed YES / reviewed count and coverage, and remain explicitly unranked. Missing, failed, truncated, and unreviewed cells are not negative answers.
Ties: equal scores share competition rank (1, 1, 3). Names break display-order ties only.
Confidence stays separate: P4, P5 are excluded from this score. A NO under a certainty instruction need not deny physical fit. This is not confidence calibration.
Selection: earliest completed attempt in run order, then attempt index. If none completed, show the latest actual attempt; if absent, show unattempted. An early successful answer is never replaced to improve its score. Every attempt remains archived.
Limits: one item and an owner's observation, not general accuracy, intelligence, validated physics, or a uniform one-shot head-to-head. Added assumptions about hatch closure or safety can change what an answer means.

Follow one wording change

One question, every model.

Choose a question for a model-by-model comparison, review reasons, and links to exact answers. This is a prompt navigator, not a second leaderboard. Focus this region to scroll with arrow keys, or swipe on small screens.

15 prompts across 30 models; counts use selected answers only. Confidence variants do not contribute to rank.
Prompt	Wording change	Reviewed labels & missing coverage	Drill down
P1 · Inches + centimetersFit agreement	Add approximate centimeter equivalents to the base question's inch measurements.	YES 3 · NO 25 · uncertain 11 missing / unreviewed of 30	Compare models
P2 · Inches onlyFit agreement	Base question: imperial dimensions, vehicle generation and year, with no loading instructions.	YES 6 · NO 19 · uncertain 23 missing / unreviewed of 30	Compare models
P3 · Seats specifiedFit agreement	Specify folded rear seats and a fully reclined passenger seat.	YES 5 · NO 22 · uncertain 12 missing / unreviewed of 30	Compare models
P4 · More than 90% sureConfidence only	Specify the seats and add an instruction favoring an affirmative answer above 90% certainty.	YES 11 · NO 14 · uncertain 41 missing / unreviewed of 30	Compare models
P5 · 100% sure or NOConfidence only	Specify the seats and require NO unless the model is 100% certain; this is a confidence instruction, not a separate physical fit test.	YES 0 · NO 29 · uncertain 01 missing / unreviewed of 30	Compare models
P6 · Dimensions in centimetersFit agreement	Replace the dimension values with exact centimeter conversions; retain the product's 98-inch size name.	YES 5 · NO 19 · uncertain 33 missing / unreviewed of 30	Compare models
P7 · Dimensions reorderedFit agreement	Keep the same inch measurements but present depth, thickness, then length.	YES 4 · NO 19 · uncertain 34 missing / unreviewed of 30	Compare models
P8 · Not just cargo volumeFit agreement	Ask for rigid-object placement rather than relying on the car's cargo-volume figure.	YES 2 · NO 20 · uncertain 17 missing / unreviewed of 30	Compare models
P9 · Opening versus interiorFit agreement	Separate passing through the hatch from positioning inside, and ask the model to identify assumed dimensions.	YES 5 · NO 17 · uncertain 53 missing / unreviewed of 30	Compare models
P10 · Answer first, brieflyFit agreement	Require a clear opening verdict and a short explanation without prescribing the verdict.	YES 3 · NO 19 · uncertain 44 missing / unreviewed of 30	Compare models
P11 · Skeptical first impressionFit agreement	Add a skeptical guess, explicitly asking the model to assess independently; pair with P12.	YES 8 · NO 18 · uncertain 13 missing / unreviewed of 30	Compare models
P12 · Optimistic first impressionFit agreement	Add the opposite guess while keeping the independence instruction identical to P11.	YES 2 · NO 20 · uncertain 26 missing / unreviewed of 30	Compare models
P13 · Test the objectionFit agreement	Ask the model to challenge its conclusion with the strongest geometric objection.	YES 10 · NO 11 · uncertain 54 missing / unreviewed of 30	Compare models
P14 · Consider a diagonalFit agreement	Suggest diagonal placement as a possibility, without supplying an interior measurement or the answer.	YES 3 · NO 17 · uncertain 73 missing / unreviewed of 30	Compare models
P15 · Unknown is not impossibleFit agreement	Ask the model to distinguish missing measurement evidence from evidence that fitting is impossible.	YES 3 · NO 11 · uncertain 133 missing / unreviewed of 30	Compare models

Read the base question, exact prompts, and word-level differences

A different question

What do confidence instructions change?

Confidence wording is not fit-agreement score or probability calibration. Under “100% sure or NO,” NO may express uncertainty rather than physical impossibility. Each badge opens its answer and history; this region scrolls independently if needed.

Confidence variants only — excluded from all ranks
Model	P4More than 90% sure	P5100% sure or NO
Google: Gemini 3.5 Flash Lite	YES	NO
StepFun: Step 5 Preview	YES	NO
DeepSeek: DeepSeek V4.1 Flash 1017	NO	NO
Meta: Muse Wonder	NO	NO
Anthropic: Claude Sonnet 5.5	Uncertain	NO
Meta: Muse Glimmer 30B	YES	NO
OpenAI: GPT-6 Luna	NO	NO
SpaceXAI: Grok 4.7	NO	NO
Anthropic: Claude Haiku 5.5	Uncertain	NO
Anthropic: Claude Opus 5.5	NO	NO
Google: Gemini 3.8 Flash	NO	NO
Google: Gemini 4 Argon	NO	NO
IBM: Granite 4.2 8B	NO	NO
Meta: Muse Spark Open	NO	NO
Mistral: Ministral 3 3B 2512	YES	NO
NVIDIA: Nemotron 3.5 Lightning	NO	NO
NVIDIA: Nemotron 3.5 Super VL	NO	NO
OpenAI: GPT-6.1 Sol	Uncertain	NO
Upstage: Solar Mini 4	NO	NO
Cohere: Command A+	Uncertain	NO
DeepSeek: DeepSeek V4 Pro 0813	YES	NO
inclusionAI: Ling 3.1 Flash	NO	NO
LiquidAI: LFM2.5-24B-A2B	NO	NO
Mistral: Mistral Large 4	YES	NO
Qwen: Qwen3.5-9B	Transport error	Transport error
Qwen: Qwen3.8 27B	YES	NO
Qwen: Qwen3.8 Flash	YES	NO
Qwen: Qwen3.8 Max Prime	YES	NO
Xiaomi: MiMo-V2.6-Flash	YES	NO
Z.ai: GLM 5.3 Prime	YES	NO

Coverage, errors & cost

The denominator matters.

30 models × 15 prompts = 450 selected-answer cells. A successful recovery does not erase an earlier failure. There are 513 actual requests in the full history, including 111 noncompleted attempts. The ledger below describes selected cells, not a request-total denominator.

Selected completed final answers: 402
Of those, awaiting review: 0
Selected truncated responses — not scored: 2
Selected empty final answers — not scored: 3
Selected API errors: 5
Selected transport errors: 38
Selected interrupted requests: 0
Not attempted / no actual request record: 0

All-attempt spend. Explicit unknowns.

Known cost · all actual attempts: $11.21190183
Actual attempts with unknown cost: 83
Disclosed spending authorization / admission amount: $18

These totals include prior failures and every actual historical request, not only selected answers. Unavailable cost is not zero. API-reported costs are not an independently reconciled invoice; catalog prices are not per-answer bills.

New uncapped requests use observed-spend admission, not a fabricated worst-case token reservation. In-flight requests may finish after admissions stop and can overshoot an admission threshold or encounter provider limits. The disclosed amount is not a guaranteed invoice ceiling.

Per-attempt archives show provider, returned model, exact sanitized request, deadline, status, timings, warnings, usage, and cost. Missing information stays unknown.