TLDR
Z.ai shipped GLM-Legal, a post-train of the MIT-licensed GLM-5.3-Flash, and Harvey’s evaluation partner verified the claimed 19.2 percent all-pass within the same day: rank 6 on the leaderboard at $0.15 a task. It enables contract review, evidence analysis and compliance work at a price 14 times under the current leader and 100 times under Anthropic’s configs. The difference between the verified number and the leaderboard crown is two things: 6.25 points of task-pass against Muse Spark 1.2, and a hallucination-gated version of the benchmark that scored the base model near the bottom and has not run the legal one yet.
Caption: the cost ladder on Harvey’s benchmark, October 12: the green bar is the whole story, 19.17 percent at 15 cents.
The number is partner-verified, same day
The unusual part of this launch is not the score, it is whose scoreboard it sits on within hours. Vals AI operates Harvey’s Legal Agent Benchmark leaderboard and updated it October 12 with GLM Legal at rank 6: 19.17 percent of tasks fully passed, 92.5 percent of rubric criteria passed, one hour twelve minutes per task, $0.15 per task. Z.ai’s own card lists the same three competitor rows Vals carries (Kimi K3, Fable 5, Astra), at the same numbers Vals publishes. The launch claim and the partner artifact agree, on the headline number and on the competitor rows.
The base-model tie settles what the post-train bought. GLM-5.3-Flash sits on the same leaderboard at 6.67 percent for $0.57 per task. GLM-Legal’s 19.17 for $0.15 means twelve and a half points of accuracy and 3.8 times less spend per task: the post-train and whatever harness work shipped with it reduced the token bill (fewer wasted passes inside a one-hour task window) while adding the legal competence. That pairing, accuracy up and cost down, is what a domain post-train is supposed to do and rarely demonstrates with same-day partner verification.
Caption: the efficiency frontier: the green point’s score-per-dollar is an order of magnitude under everything to its right; the open question is what hallucination gating does to it.
The cost column reframes the whole benchmark
Harvey’s leaderboard has been a frontier-model tournament: the top rows trade 20 to 25 percent all-pass at $2 to $5 a task, and the Anthropic/OpenAI premium configurations sit at $19 to $46 for single-digit percentages. GLM-Legal changes the shape at the cheap end: at $0.15 it is priced like a utility bill, which moves agentic legal work from “enterprise pilot budget” to “comparable with a paragraph of a junior associate’s time.” The right comparison for a law firm is not Muse Spark’s 25.42 percent; it is what a compliance team currently pays for the exact tasks where GLM-Legal already passes all criteria.
The gating question keeps it honest. Artificial Analysis’s v1.1 methodology zero-credits a task for any material hallucination, and its three-judge reading scored the GLM-5.3-Flash base at 0.8 percent with 4.04 material hallucinations per task, versus more than 60 percent of otherwise-passing results contaminated across models. AA has not run GLM-Legal on that harness. If the legal-post-train’s hallucination rate did not drop proportionally with the all-pass gain, the gated score could compress the advantage hard; if the domain tuning did exactly what it was built for (grounding citations and holdings in the matter files), the gated score would lead the board. The data needed to resolve this exists and the benchmark’s own infrastructure runs it weekly; both readings are legitimate, and only one has been published.
The deployment play, two doors
No weights at announcement: GLM-Legal ships through a design-partner application and through AutoClaw, Z.ai’s desktop agent application (Mac, Windows, Linux; 200 million tokens free to new accounts; browser automation, IM integration into Telegram and Slack, and document generation bundled). The base model remains MIT on Hugging Face, so Z.ai is running the release pattern from the GLM-5.3 launch again: the open row carries the architecture, the gated product carries the domain post-train, whatever harness code the benchmark run used, and the compliance framing legal buyers’ procurement teams treat as a feature. Whether the legal post-train ever reaches the open row, the way the base model’s weights did in August, is the open question a law firm IT department should be asking Z.ai directly.
What to watch
- Artificial Analysis running GLM-Legal on the gated harness: the single number that decides whether the price-performance story survives the strictest reading.
- Whether Vals’s GLM-Legal row updates with a fallback flag when Z.ai ships the design-partner cohort’s fixes; the 19.17 is a single snapshot.
- Anthropic’s response on price: Fable 5 at $19.23 per task against a 19-percent competitor at $0.15 is the first time the benchmark’s cost column matters commercially rather than descriptively.
- Whether the legal post-train lands on the open GLM line eventually: the Z.ai play has alternated product-gated and open releases for two generations, and legal is the highest-price vertical where open deployment would hit hardest.
- Design-partner cohort reports: the API access form asks about practice-area needs, which telegraphs the next vertical (evidence analysis and compliance are the card’s stated targets, both procurement-heavy).
Sources: Z.ai announcement - Vals AI leaderboard - Harvey LAB-AA v1.1 - AA v1.1 methodology post - Harvey’s initial results - AutoClaw
Related on this site: GLM 5.3 launch
Discussion
Be the first to commentStart a discussion
Got a take on this, a rig to show off, or a benchmark that says otherwise? Sign up and start the thread - your comment publishes instantly once you're in.