Tool-use review
Updated · Instruction-Tuned
Deployment Verdict
Deploy conditionally, because tool execution is usable, but MCP calls are not consistently valid and synthesis fidelity remains too weak for production knowledge and compliance pipelines.
Tool Execution Profile
Qwen 3.6 35B-A3B does not exhibit reliably agentic tool behavior. It can execute tools when the path is already clear, but it does not select them consistently. This is visible directly in the gap between Web Search & Tool Selection, which tests the choice between search and fetch without an explicit hint, at P1 35, and URL Construction & Fetch, which measures the derivation of a target URL and the subsequent retrieval, at P1 80. The model tends to follow a familiar retrieval pattern rather than situationally recognizing which tool the task requires.
On the positive side, HTTP Fetch & Extract at P1 80 points to solid mechanical execution. On the negative side, the global finding Tool-Call valid: false stands. For an MCP pipeline this means: the intent to use tools is present, but the protocol behavior is not robust enough for unattended orchestration.
Synthesis Fidelity
How well does it condense tool results? Only to a limited degree. The P2 score of 45.83 is the actual bottleneck. For HTTP Fetch & Extract, which requires precise facts from real page content, it remains usable at P2 60. For Multilingual Search & Synthesis — cross-lingual research with a German-language summary — likewise P2 60. That is not sufficient for reliable condensation in production decision or documentation pipelines. EU License Research falls clearly short at Combined 26.
Does it stay within the tool result or fall back on training data? In the honeypot EU License Research, which tests exactly this, no hallucination was detected. That matters. The model does not fabricate external facts here. At the same time, P2 20 is a warning signal: it stays formally within the safe boundary, but condenses the retrieved information only weakly. Confidence in factual accuracy is therefore higher than confidence in the usability of the answer.
Error Resilience
On Tool Failure Handling (404), which tests for transparent handling of a failed retrieval, the model does not hallucinate substitute content. That is a production-relevant positive. P2 60 indicates that it communicates error states acceptably, even if the response is not particularly clear or actionable. For operational pipelines this is acceptable, as long as an upstream system handles retry or fallback logic.
Operational Profile
Total 58.11s per run: slow.
Call 1 1.17s, MCP latency 1.33s, Call 2 7.19s: frontload fast, overall flow sluggish.
Cost/run: local. Inexpensive to operate, but the runtime is not proportionate to the only moderate overall performance.
Summary & Recommendation
Suitable for local, sovereignty-oriented pipelines with human oversight, particularly where fetch-heavy tasks and simple URL-based retrievals dominate. Not suitable for autonomous MCP orchestration, compliance workflows, dynamic research chains, or any pipeline where correct tool selection and clean condensation must hold without review. If you deploy it, treat it as a cost-efficient local worker under strict tool validation and with downstream quality control.
This analysis was generated automatically based on benchmark data. Model used: GPT 4.5 by OpenAI. The raw data and full methodology are documented in the GitHub project.