Did it finish?
Job-specific acceptance checks, not a universal intelligence score.
Lab discipline + reviewer judgmentStandard v0.1
Find the breakpoint.
Show the receipts.
A benchmark asks whether a model can answer a test. A buyer needs to know whether it can finish their work repeatedly, what remains for the human, and when the next tier changes the outcome enough to justify its price.
01
Define the unit
A real job ends in a checkable artifact.
We do not treat a clever answer as proof that a model can complete work. Each test begins with a bounded objective and ends with something that can be inspected.
Objective→Materials→Environment→Execution→Artifact→Verification
The bar must exist first.
Examples include tests passing, citations resolving, a schema validating, required fields matching, or cleanup remaining below a declared threshold. “Looks good” is not enough on its own.
02
Escalate on evidence
Start cheap. Stop when the bar clears.
For each job we test a small, realistic buying ladder: the lowest-cost serious contender, the balanced default, and the frontier option. Difficulty rises until the lower tier is no longer dependable.
- CheapCan it clear the bar repeatedly?
If yes, stop. More model is overkill for this job.
- DefaultDoes it resolve the observed miss?
If yes, measure whether lower cleanup and failure justify the increase.
- FrontierDoes the hardest work change materially?
The flagship earns its premium only where the work shows it.
03
Measure useful work
Token price is only the receipt at the door.
The lower-priced run can cost more once a person has to steer, inspect, correct, rerun, and recover it. We report only measures the test can actually support.
Did it finish again?
Repeated attempts and the spread between strong and weak runs.
What stayed on the desk?
Prompting, supervision, corrections, verification, and recovery.
How did it miss?
Detectability, severity, blast radius, self-detection, and repair cost.
What is it like to use?
Initiative, restraint, literalness, calibration, context, and recurring friction.
What did success cost?
Inference plus the human work required to reach acceptable output.
04
Evidence hierarchy
Never let a benchmark become a bigger claim than it supports.
Provider claim
What the maker says changed. Preserved as a claim, not converted into an independent finding.
External evidence
Model cards, independent benchmarks, research, pricing, and lifecycle documentation—with scope and date visible.
Field observation
What happened in our exact job, harness, settings, repetitions, and working environment.
Editorial verdict
The narrow buying decision the combined evidence supports—and the limits it does not cross.
SupportedDirectionally supportedMixedNot observedInsufficient evidence
05
Identity and time
Every conclusion is dated.
“Model name” is not enough. Reviews bind the underlying model or API identity, alias state, settings, reasoning level where relevant, tools, harness, environment, dates, and repetitions.
ObjectModel / implementation / product
Kept distinct
IdentityExact ID, snapshot, or alias
Lifecycle noted
SetupSettings, tools, harness
Environment visible
TimeFirst tested / last verified
History preserved
Current
No known material change to the reviewed buying decision.
Retest required
A meaningful input changed and the verdict needs a fresh run.
Stale
Useful historically; unsafe as current buying advice.
Retired
The reviewed option is no longer available for the use.
06
Editorial identity and limits
Transparent about the desk behind the work.
PublisherAIModel.Reviews
BylineAIModel.Reviews Editorial Desk
VerificationAIModel.Reviews Verification Desk
Content is produced with AI assistance and published under stable, role-based editorial aliases. These aliases are organizational roles; they do not represent human identities, credentials, professional licenses, legal approval, or a monitored support promise.
The current build publishes the research method and product direction. It does not claim completed original model tests, current price comparisons, pass rates, breakpoints, sponsorship relationships, or a proprietary historical corpus. Those statements require their own evidence.
Proposed first experiment
Repository coding: five tasks, three configurations.
The fixture design and acceptance rules are available. Paid model runs and outcome measurements have not been completed.
Status · proposed, not run
Luna medium, Luna high, Terra medium.
Inspect the fixed task types, controls, repetition policy, and exact evidence a result would need.
Read the proposed protocol →Method before marketing
The standard is public. The verdicts must now earn it.
Use the external-research guide now and return for firsthand results only when their artifacts exist.
Open the buyer guide