Which LLMs actually call tools well?
Throughput tells you a model is fast. A quality score tells you it reasons. Neither tells you whether it can use tools: pick the right one from a dozen, pass the right parameters, chain calls across turns, recover when a call fails, and refuse when it should not act at all. That is what a tool-calling quality benchmark measures - the substrate every agentic workflow is built on. Below is how it works and where the models we have assessed land.
What it measures
The benchmark runs a model against a fixed set of scenarios with a shared system prompt, a fixed date context, and a defined toolset (12 or 52 tools). Each scenario has deterministic mock tool responses with realistic payload noise, so a run is reproducible and comparable to anyone else's.
- Tool selection. Pick the right tool from many options.
- Parameter precision. Get units, dates, and multi-value parameters right.
- Multi-step chains. Thread data across turns, parallel calls, async polling.
- Restraint and refusal. Know when not to call a tool.
- Error recovery. Handle failures without losing data integrity.
- Safety and boundaries. Prompt injection, authority escalation, parameter validation. A safety gate caps the rating if this category scores below 50%.
Scoring is 2 (pass) / 1 (partial) / 0 (fail) per scenario, weighted to a 0-100 percentage. Infrastructure failures (timeouts, 5xx) are excluded rather than counted as zero, so a flaky endpoint does not sink a good model. Scores shown here are Tokenstead's editorial assessment aligned with this methodology and sourced from public agentic evals (Artificial Analysis agentic Elo, Terminal-Bench 2.1, SWE-bench).
Ranked by tool-calling score
Models Tokenstead has assessed, sorted by tool_calling_score. Open a card for the full score breakdown and architecture notes.
Run it yourself
The benchmark is an open Python tool that runs against any OpenAI-compatible endpoint - vLLM, LiteLLM, llama.cpp, or SGLang. Install it, point it at your local model, and it produces the same per-scenario traces and final score you see summarized above. It is a single package with no telemetry beyond the run you choose to keep.
uv tool install git+https://github.com/SeraphimSerapis/tool-eval-bench.git tool-eval-bench run --short
The short run is a fast subset; the full run is 69 scenarios (84 with the opt-in hard mode). See the repo for the scenario taxonomy, the held-out pack methodology, and the per-category breakdown.