The question
Does a stronger model earn its token premium on repository maintenance—and is raising the cheaper model’s reasoning effort a better first move? The proposed configurations are Luna at medium effort, Luna at high effort, and Terra at medium effort.
Five task types
| Task type | Planned work | Acceptance focus |
|---|---|---|
| Repository understanding | Trace a missing-ID operation | Correct call chain; no invented write; no edits |
| Bounded bug | Fix filtering with a limit | Correct order, edge cases, regression test |
| Multi-file feature | Add service export and CLI JSON | Correct interface; unchanged plain output |
| Cross-layer failure | Repair state after a failed save | Propagate failure; never report false success |
| Compatibility refactor | Add a versioned, atomic storage format | Old data reads; failed writes preserve the file |
Controls
- Every run starts from the same fixed commit in an isolated environment.
- The runner, tools, task text, permissions, and budgets are fixed per comparison.
- Human hints, retries, fallback, refusals, and budget exhaustion remain visible.
- Three repetitions per task and configuration would create a 45-run pilot.
- A passing existing suite is insufficient; each task has extra acceptance and scope checks.
The small fixture can validate instrumentation and expose task-specific differences. It cannot establish a general production-repository reliability rate.
What results would include
Every attempt would retain its patch or explanation, acceptance outcome, complete billed usage, timing, interventions, review/correction effort, and failure shape. Failed and inconclusive runs remain in the denominator.
How to use this protocol now
Adapt the task and checks to your own repository without presenting this design as a finished benchmark. Freeze the changed fixture as a new version and keep its outcomes separate. Read the buyer guide for the selection rule and the Luna–Terra comparison for the economic question.
Reader questions
Frequently asked questions
Has AIModel.Reviews run this repository-coding test?
No. The protocol is proposed and zero original model tests have been completed from it.
Why repeat each task?
Repeated attempts show consistency and prevent one unusually strong or weak run from becoming the entire conclusion. The small pilot still cannot establish a universal reliability rate.
Primary reading
Sources and update conditions
Rates, availability, model identity, and provider behavior can change. Recheck the linked sources before making a consequential purchase or sending repository data.