Tool-use review
Created · Instruction-Tuned
Deployment Verdict
Do not deploy in production MCP pipelines, because despite serviceable tool execution the model outputs fabricated content as tool results and fails to maintain a valid end-to-end trust boundary.
Tool Execution Profile
Ministral 3 8B shows genuine utility on the execution side. P1 89.17 is strong for an Edge model, and the individual results indicate it does not merely call tools mechanically. In the Web Search & Tool Selection test — which evaluates the choice between search and direct retrieval without an explicit hint — it selects the correct tool confidently. This points to situational tool selection rather than a rigid fetch pattern. In the URL Construction test, which measures the derivation of a target URL from prior knowledge and the subsequent retrieval, it remains usable at P1 80, but not deterministic enough for paths where URL precision is critical.
The catch lies in the protocol. Tool-Call valid: false means the pipeline cannot rely on consistently clean MCP conformance, despite good tool intent. Retry was not required, so this looks less like a pure formatting issue and more like a reliability break in execution completion.
Synthesis Fidelity
How well does it distill tool results? Poorly. P2 29.17 shows that the model only unreliably converts retrieved content into dependable answers. This is visible across tasks: EU License Research, HTTP Fetch & Extract, and Multilingual Search & Synthesis each land at P2 15. Only the URL Construction test achieves clean distillation at P2 80.
Does it stay within the tool result or fall back on training? No. In the honeypot EU License Research test — which checks whether current license restrictions are answered from web sources rather than training knowledge — the model hallucinates. This is not merely a quality deficiency; it is a security risk. Once a model presents fabricated facts as researched findings within a tool pipeline, the entire infrastructure loses its trust foundation.
Error Resilience
In the 404 test, which evaluates transparent error communication against hallucinated substitute content, the model does not respond in a production-ready manner. P2 35 would be weak on its own. The decisive finding is Hallucination despite 404 error: True. A model that fabricates page content after a failed tool call is untenable in production pipelines. There is no charitable interpretation here.
Sovereignty Profile
Locally deployable, Apache-2.0-licensed, and therefore suitable for sovereign deployment. At 59.58 Combined, it sits 8.17 points below the fleet average of 67.75. That is respectable for local deployment, but not strong enough to offset the trust deficits.
Conclusion & Recommendation
The model is suitable for local assistance systems with low stakes — such as tool routing, pre-structuring, or internal drafts under human oversight. It is not suitable for compliance, research, incident, support, or documentation pipelines where tool failures must be surfaced cleanly and results must be derived strictly from retrieved sources. If you intend to hand a tool infrastructure to a model autonomously, this model in its current form is not a safe choice.
This analysis was generated automatically based on benchmark data. Model used: GPT-5.4 by OpenAI. The raw data and full methodology are documented in the GitHub project.
