What is a tool-calling quality benchmark?

It measures whether an LLM can use tools well in an agentic workflow: picking the right tool from many, passing accurate parameters, chaining calls across turns, handling errors, respecting safety boundaries, and knowing when not to call a tool at all. It is narrower than a full agent benchmark - it focuses on the tool-use substrate that everything else builds on.

How is the score computed?

Each scenario is scored 2 (pass), 1 (partial), or 0 (fail), weighted by scenario count to a 0-100 percentage. A safety gate caps the rating if the safety category scores below 50 percent, and infrastructure failures (timeouts, 5xx) are excluded rather than counted as zero. Scores on this page are Tokenstead's editorial assessment aligned with that methodology and sourced from public agentic evals.

Can I run this myself?

Yes. The benchmark is an open Python tool that runs against any OpenAI-compatible endpoint (vLLM, LiteLLM, llama.cpp, SGLang). Install it, point it at your local model, and it produces the same per-scenario traces and final score you see summarized here.

Tool-calling quality

Which LLMs actually call tools well?

Throughput tells you a model is fast. A quality score tells you it reasons. Neither tells you whether it can use tools: pick the right one from a dozen, pass the right parameters, chain calls across turns, recover when a call fails, and refuse when it should not act at all. That is what a tool-calling quality benchmark measures - the substrate every agentic workflow is built on. Below is how it works and where the models we have assessed land.

Browse all models The tok/s benchmark

What it measures

The benchmark runs a model against a fixed set of scenarios with a shared system prompt, a fixed date context, and a defined toolset (12 or 52 tools). Each scenario has deterministic mock tool responses with realistic payload noise, so a run is reproducible and comparable to anyone else's.

Tool selection. Pick the right tool from many options.
Parameter precision. Get units, dates, and multi-value parameters right.
Multi-step chains. Thread data across turns, parallel calls, async polling.
Restraint and refusal. Know when not to call a tool.
Error recovery. Handle failures without losing data integrity.
Safety and boundaries. Prompt injection, authority escalation, parameter validation. A safety gate caps the rating if this category scores below 50%.

Scoring is 2 (pass) / 1 (partial) / 0 (fail) per scenario, weighted to a 0-100 percentage. Infrastructure failures (timeouts, 5xx) are excluded rather than counted as zero, so a flaky endpoint does not sink a good model. Scores shown here are Tokenstead's editorial assessment aligned with this methodology and sourced from public agentic evals (Artificial Analysis agentic Elo, Terminal-Bench 2.1, SWE-bench).

Ranked by tool-calling score

Models Tokenstead has assessed, sorted by tool_calling_score. Open a card for the full score breakdown and architecture notes.

Qwen3.8-Flash-Next MoE

DeepSeek V4 Flash 0731 MoE

Ornith-1.5-35B-A3B MoE

Nemotron 3.5 Lightning MoE

Run it yourself

The benchmark is an open Python tool that runs against any OpenAI-compatible endpoint - vLLM, LiteLLM, llama.cpp, or SGLang. Install it, point it at your local model, and it produces the same per-scenario traces and final score you see summarized above. It is a single package with no telemetry beyond the run you choose to keep.

uv tool install git+https://github.com/SeraphimSerapis/tool-eval-bench.git
tool-eval-bench run --short

The short run is a fast subset; the full run is 69 scenarios (84 with the opt-in hard mode). See the repo for the scenario taxonomy, the held-out pack methodology, and the per-category breakdown.

Source on GitHub Measure your tok/s