Tool-use review
Created · Instruction-Tuned · Long Context
Deployment Verdict
Conditional deploy, as overall tool use is viable, but invalid tool calls and only moderately reliable synthesis limit confidence in an MCP pipeline.
Tool Execution Profile
Qwen 3.5 27B demonstrates genuine tool selection rather than mere schema-following. On the Web Search & Tool Selection test — which checks whether web_search is chosen over fetch without any hint — it operates confidently and identifies the correct access path. It also reliably retrieves the required sources on Multilingual Search & Synthesis and EU License Research. This speaks to a solid agentic baseline.
Protocol fidelity in the details is weaker. The global finding “tool call valid: false” is more relevant for production than the clean P1 score. On the URL Construction test — which requires deriving the target URL from internal knowledge and then executing fetch correctly — performance is workable but not deterministic enough. The model thus appears intelligent in its choice of tool, but less precise at the last mile of the call. No retry was needed, pointing to execution accuracy issues rather than comprehension failures.
Synthesis Fidelity
How well does it consolidate tool results? Only limitedly reliable. P2 performance falls well short of tool execution. Particularly on HTTP Fetch & Extract — structured extraction of real page content — the consolidation is too imprecise. The pattern: it locates information more often than it correctly and concisely converts it into usable responses.
Does it stay within tool output or fall back on training? On the Honeypot EU License Research test, which forces retrieval of current license restrictions from web sources, the trust signal remains mixed. It does not hallucinate overtly, but P2 20 shows that the response barely translates the queried web content into reliable, assertable form. Since hallucination was detected in the overall run, this is a safety risk. In a tool pipeline, what matters is not only whether the model uses tools, but whether stated facts can be unambiguously traced back to tool output.
Error Resilience
On the 404 test — which measures transparent handling of failing tool calls — the model responds acceptably. It does not fabricate page content despite the error. P2 60 is not a strong score, but far more important for production here is transparency: errors tend to be left standing rather than silently overwritten. This is a workable foundation for orchestration with external retry or fallback.
Operational Profile
Total 764.95s: slow.
Call 1 7.55s, Call 2 117.62s, MCP latency 2.32s.
Cost/run: local. Cheap in money, expensive in runtime relative to performance.
Conclusion & Recommendation
Suitable for local, cost-controlled pipelines with human review, clear tool schemas, and external validation of the final response. Particularly useful where tool selection matters more than precise consolidation — for example, research pre-stages, collection agents, or multi-step routing tasks. Not suitable for compliance, fact-checking, or extract-transform-report pipelines where the final response must be strictly derived from tool output and formally reliable.
This analysis was generated automatically based on benchmark data. Model used: GPT 4.5 by OpenAI. The raw data and full methodology are documented in the GitHub project.
