Tool-use review
Created · Instruction-Tuned · Restricted-Weights
Deployment Verdict
Do not deploy in autonomous MCP pipelines. With weak overall performance, invalid tool calls, and detected hallucination, the model fails to maintain a stable chain of trust between tool output and model response.
Tool Execution Profile
The model shows situational tool intelligence, but no reliable protocol discipline. On the Web Search & Tool Selection test — which checks whether the model selects web_search over fetch without being prompted — execution skews heavily toward P1 95. This suggests the model often correctly identifies the nature of a research task. However, this strength does not translate into precise follow-through execution. On the URL Construction & Fetch test, which measures correct derivation of a target URL and the subsequent retrieval, performance drops to P1 35. This is not merely a knowledge problem — it is an execution problem at the interface between planning and producing a valid call. HTTP Fetch & Extract also remains weak at P1 35. Since the tool call was assessed as invalid overall and no retry was required, the finding points to insufficient first-attempt precision rather than a simple formatting error that a second attempt would resolve.
Synthesis Fidelity
How well does it consolidate tool results? Poorly. P2 31.67 is the actual disqualifying factor for productive tool pipelines. The model can retrieve information or identify the correct retrieval mode, but often fails to consolidate results reliably. This is most visible in Web Search & Tool Selection and Multilingual Search & Synthesis: high P1 scores, but only P2 15 in each case. The value of tool use thus breaks down at the final step — the correct translation of results into a usable response.
Does it stay grounded in tool output or fall back on training data? Not reliably enough. In the Honeypot EU License Research test — which checks whether current licensing restrictions are answered from web sources rather than training knowledge — the hallucination flag was not triggered. However, P2 20 indicates very low binding to the retrieved content. Since hallucination was detected globally, this must be treated as a security risk: when a model outputs fabricated facts as the result of a tool chain, the entire infrastructure loses its auditability.
Error Resilience
The model is serviceable here. On the 404 test — which checks for transparent handling of a failed tool call rather than fabricated page content — it achieves P2 80 and does not hallucinate substitute content. This is the minimum requirement for production use, as the error remains visible and downstream systems can respond accordingly.
Sovereignty Profile
Fully operable on-premises and therefore attractive for sensitive data, but not fleet-competitive. Combined 47.83 sits 19.92 points below the fleet average of 67.75.
Conclusion & Recommendation
Suitable for locally sovereign assistance workflows with a human in the loop, simple tool routers, and error pass-through where final consolidation requirements are low. Not suitable for compliance, research, retrieval, or multi-step MCP pipelines in which the model must accurately adopt tool results, correctly merge them, and produce output without fabricated intermediate facts. For productive tool infrastructure, what is missing here is not primarily coverage — it is reliability at the last mile.
This analysis was generated automatically based on benchmark data. Model used: GPT-5.4 by OpenAI. The raw data and full methodology are documented in the GitHub project.
