Tool-use review
Created · Instruction-Tuned
Deployment Verdict
Conditional deploy, because tool execution is strong but synthesis fidelity is too weak and the tool call was not consistently valid. For production MCP pipelines, raw execution strength alone is insufficient here.
Tool Execution Profile
The model demonstrates genuine tool intelligence, not merely rigid pattern matching. On the Web Search & Tool Selection test — which checks whether web_search is chosen over fetch without any hint — it identifies the correct access path very reliably. On the URL Construction test, which evaluates deriving a target URL from the model’s own knowledge, it performs adequately but less deterministically. This points to sensible tool selection, but not consistently precise execution.
The critical issue remains that the tool call was not marked as valid overall. This is not a retry issue and therefore represents more of a protocol or format risk on the first run than a simple careless error. For MCP infrastructures this means: solid planning logic, but additional guardrail requirements for call validation and schema checking.
Synthesis Fidelity
How well does it condense tool results? Only to a limited degree. P2 performance at 56.67 is the model’s clear weak point. This is most visible on EU License Research and Multilingual Search & Synthesis: it retrieves information but then condenses it too coarsely, drops relevant caveats, or prioritizes peripheral aspects. For pure retrieval pipelines this is still manageable. For compliance, policy, or decision-adjacent summaries it is too imprecise.
Does it stay within tool results or fall back on training? On the Honeypot EU License Research test — which checks whether current license restrictions come from web sources rather than training — it does not hallucinate. That is the central trust anchor. The very low P2 score there therefore reflects not free fabrication but inadequate evaluation of actually retrieved content. That is better than hallucination, but still insufficient for sensitive domains.
Error Resilience
On the 404 test, which checks for transparent handling of failing tool calls, the model behaves in a production-ready manner. It does not fabricate page content and communicates the failure in a comprehensible way. This is exactly the behavior a tool pipeline requires: a visible error rather than plausible-sounding fiction.
Sovereignty Profile
Locally deployable, Apache-2.0-licensed, and therefore sovereignly operable. At 71.29 Combined it sits n/a points above the fleet average of 68.12. In practical terms: locally deployable and yet fleet-competitive, but not robust enough to operate as a standalone tool synthesis instance without an additional control layer.
Summary & Recommendation
Suitable for local research and orchestration pipelines in which the model selects tools, initiates search paths, and passes raw results to downstream validators or more rigorous summarizers. Not suitable as the final authority for compliance summaries, multilingual condensation, or any pipeline in which the model must itself bindingly interpret tool results. Recommendation: deploy as a local tool operator with hard call validation and separate answer verification — not as an autonomous final decision-maker.
This analysis was generated automatically based on benchmark data. Model used: GPT-5.4 by OpenAI. The raw data and full methodology are documented in the GitHub project.
