Tool-use review
Updated · Native Quantisierung · Harmony-Format
Deployment Verdict
Conditional deploy: tool execution is strong, but a detected hallucination in the honeypot and an inconsistently valid tool call mean the model does not reliably maintain the baseline trust required for critical MCP pipelines.
Tool Execution Profile
GPT-OSS 120B generally recognizes tools well and does not operate on rigid pattern-matching alone. In the Web Search & Tool Selection test — which requires choosing between search and direct fetch without any hint — it selects the correct tool confidently. This points to genuine tool selection in open pipelines. In the URL Construction test, it constructs the target URL adequately, but not precisely enough for deterministic pipelines. That is exactly where the limit shows: the model understands the workflow but does not always produce a formally sound call.
The P1 score is strong overall, but the finding “Tool-Call valid: False” is production-relevant. Not because the model cannot handle tooling, but because individual calls may carry protocol or structural errors. On the positive side, no retry was required. This reads more like a precision issue within the call itself than a fundamental misunderstanding of the MCP flow.
Synthesis Fidelity
How well does it consolidate tool results? Only with limited reliability. Synthesis Quality sits clearly below the execution level. In HTTP Fetch & Extract and Multilingual Search & Synthesis, it does pull information from sources, but consolidates it unevenly and loses detail. For summarization-based assistance this is still usable. For compliance, regulatory work, or precise extraction chains, it is too imprecise.
Does it stay within the tool result or fall back on training? This is where the real risk lies. In the honeypot EU License Research test — which checks whether current license restrictions are answered from web sources rather than training knowledge — the model hallucinates and achieves only P2=15. This is not merely a quality shortcoming; it is a security risk. When a model outputs fabricated facts as an apparently tool-grounded response, it undermines the reliability of the entire infrastructure.
Error Resilience
In the 404 test, which checks for transparent handling of a failing tool call, the model remains acceptable. It does not fabricate page content and communicates the failure in an essentially open manner. This property matters for production: a tool failure does not immediately escalate to misinformation. Error communication is not excellent, but it is workable.
Operational Profile
Call 1: 4.99s. MCP latency: 1.17s. Call 2: 31.44s. Total: 225.58s. Cost/run: local. Direct assessment: inexpensive locally, but slow for the response fidelity delivered.
Conclusion & Recommendation
Suitable for local research and orchestration pipelines where tool selection matters more than precise final consolidation and where a downstream verifier checks every response against raw sources. Not suitable for compliance, licensing, policy, or other high-trust pipelines where the model must strictly reproduce tool results. Anyone deploying GPT-OSS 120B should treat it as a tool operator with external verification — not as a trusted synthesis authority.
This analysis was generated automatically based on benchmark data. Model used: GPT 4.5 by OpenAI. The raw data and full methodology are documented in the GitHub project.
