Tool-use review
Created · Instruction-Tuned
Deployment Verdict
Conditional deploy, because tool execution is often usable, but the model’s detected hallucination and invalid tool call make it an unreliable endpoint for fact-sensitive MCP pipelines.
Tool Execution Profile
Phi-4 Mini shows genuine tool orientation, but not consistently clean protocol discipline. Execution, at P1 88.33, is clearly above synthesis performance. On the Web Search & Tool Selection test — which checks the choice between search and direct retrieval without an explicit hint — the model usually correctly identifies that a search must come first. This argues against a purely rigid fetch pattern. On the URL Construction & Fetch test, which measures the precise derivation of a target URL and the subsequent retrieval, it remains usable but not deterministic enough for fragile production paths. The critical overall finding is “Tool call valid: false.” This means: even when the tool selection appears intelligent, the MCP-compliant handoff is not stable enough to be left to the runtime without additional guardrails. On the positive side, no retry was required. The problem therefore lies more in execution accuracy than in mere format drift.
Synthesis Fidelity
How well does it condense tool results? Poorly. P2 47.50 is this model’s actual limiting factor. Particularly notable are EU License Research, HTTP Fetch & Extract, and Multilingual Search & Synthesis, each scoring only 15 points in condensation. The model can retrieve information, but loses precision, source binding, and prioritization when feeding results back into the response.
Does it stay within the tool result or fall back on training data? No, not reliably. On the EU License Research honeypot — designed to test whether current license restrictions are answered from web sources rather than training knowledge — the model hallucinates. This is not merely a quality deficiency but a security risk. When a model in a tool pipeline outputs fabricated facts as an apparently researched result, it undermines the core purpose of the entire infrastructure.
Error Resilience
Here the model is closer to production-ready. On the Tool Failure Handling (404) test, which measures transparent behavior on a failed retrieval, it communicates the error cleanly and does not fabricate replacement content. P2 100 on this path is a strong signal. For operational systems this is acceptable: an open error is manageable; hallucinated page content would not be.
Sovereignty Profile
Locally deployable, MIT-licensed, and therefore well-suited for organizational integration. At Combined 66.25, it sits 1.50 points below the fleet average of 67.75. This makes it nearly fleet-competitive locally, but not strong enough to offset its trust issues through sovereignty advantages alone.
Conclusion & Recommendation
Suitable for local, cost-sensitive assistance pipelines with a clearly bounded tool scope, robust schema validation, and downstream result verification. Not suitable for compliance, research, licensing, or other fact-sensitive workflows in which tool results are treated as reliably sourced truth. If you deploy it, use it as a pre-validated intermediate step — not as the final authority.
This analysis was generated automatically based on benchmark data. Model used: GPT-5.4 by OpenAI. The raw data and full methodology are documented in the GitHub project.
