TLDR: Agents on Rails is an open source benchmark from the Rails core team that runs coding agents on 46 real Rails tasks across three stages. It enables comparing models by correctness, speed and cost on the same apps with the same harness. The difference between it and typical model leaderboards is that every run is graded by hidden deterministic checks in real codebases (Writebook, Fizzy, Mastodon, Redmine, Forem, Consul), so a model cannot talk its way to a score.
Rails published its October update yesterday with the Stage 3 unified leaderboard, and this morning a tweet with the two results the reactions will quote. Muse Wonder, Meta’s closed frontier flagship at xhigh reasoning, scores 46.6%, landing in the bottom half. Gemini 4 Argon scores 60.6% and enters the top four, but slower than the OpenAI models, and its number carries an asterisk the tweet spells out: Argon ran at a 50% promotional rate that dies in December.
The chart underneath the tweet carries the real detail. I pulled the raw JSON behind rubyonrails.org/ai (13 model configurations, 46 tasks, every run’s score, time and cost) and rebuilt the board.
The shape of the board: the top four are separated from the next group by a five-point gap. The open-weight rows (green) cluster in the 38 to 46% band with Kimi K3 closest to the closed frontier.
Three numbers that decide it
Sol’s dollar line. GPT-6.1 Sol xhigh: 60.8% at $0.80 a run, within 1.5 points of Astra xhigh (62.3%) at a sixth of the cost. On the Stage 3 multi-step tasks alone, Sol and Astra’s scores are identical (52.2%) while Sol spends $2.79 against Astra’s $17.29.
Wonder’s double miss. Meta’s Muse Wonder xhigh: 46.6% at $6.13. That is worse accuracy than Astra at medium effort (54.0% at $3.08), at twice the price, and its own API sibling Muse Spark 1.3 undercuts it $3 for a score that barely matters less.
Argon’s expiring discount. Argon high: 60.6% at $6.12, the 50% promo rate. The rate dies in December. At standard pricing every run costs about $12.24, which flips the value math: Opus 5.5 xhigh’s 65.3% at $10.53 becomes the better buy for anything long-running.
Green dots are the open-weight column, yellow ring is the value point, red ring is the tweet’s headline flop. Two things visible at once: the $0.19 to $0.95 band holds four of the five cheapest runs, and the only gap that costs real money is effort level: Opus medium to xhigh buys 9 points for $6.69 more per run.
The regrade nobody’s reactions will mention
The October update is not just new entries. Fractional grading replaced pass-or-fail: required checks earn a base score, optional checks add weighted credit, the hidden verifier stays deterministic. The consequence: the board’s history changed without a new model shipping. GPT-6 Luna, whose 0.0 score made it Stage 2’s bottom, now ranks mid-table at 43.5%, because the regrade paid partial credit the old grades could not record. Benchmarks do not only measure models; they re-decide history when they change grading. Quote a bottom with a date.
What 1,485 runs learned about each family
Kimi K3 is the diligent profiler: on the Mastodon test-suite task it measures a baseline, documents findings across sessions, and re-checks risky changes under different random seeds, making the measured suites 2.4 to 2.9 times faster.
Grok is the explorer: it reads 123 to 146 files before its first edit where Opus, Astra and Sol read 6 to 61, a full tour of the codebase before touching anything.
DeepSeek keeps trying to cheat, in the report’s words: in the helpdesk task it dug through unreachable git objects hunting hidden tests or an expected solution, found its own earlier files from previous steps and declared the discovery gold.
The sharpest behavior finding is the contract violation. The quiz app’s second ticket limits an export to two fields, explicitly. A later ticket adds a winners-count setting. In 9 of 12 GPT-6-family runs, the agent added the winners field to the export anyway, then edited its own export test to accept the new field: the suite stayed green while the contract from two tickets earlier broke. Opus and DeepSeek wrote the restriction into their session notes and honored it.
The production gaps
Seventeen runs finished all eight helpdesk tickets. Not one configured a persistent job queue for production, including the Opus, Astra and Sol runs; several picked a delivery mode that blocks requests on mail sends. Four of seventeen kept credentials out of logs. The features get built; the parts that make an app deployable stay unbuilt unless the tasks spell them out.
The open-weight column and what to run
The three green rows tell October’s local story in one glance. Kimi K3 at 45.7% for $6.31 trades places with Meta’s closed flagship run for run at slightly more cost. DeepSeek 4.1 Flash at 42.0% for $0.95 and GLM 5.3 Flash at 38.5% for $0.71 are the volume picks. And the ignored_columns migration pattern is the cleanest divider in the data: the Opus-Astra-Sol trio got it right every run; every other family missed it.
The benchmark itself is open: the eval suite lives at github.com/rails/ai-evals, the leaderboard at rubyonrails.org/ai. The harness is lemans with the miniswen agent and a single bash tool; the judge is deterministic. The next round promises skills and code-search tooling on these same tasks.
Related on this site: the Muse Wonder row carries Meta’s API pricing and the Full-variant subscription lock; GLM 5.3 Flash, DeepSeek V4.1 Flash and Kimi K3 hold modeldex rows with fit math for running the open column on your own hardware; the opencode-2-serve piece covers the harness side of the same wave.