Tool-use review
Created · Instruction-Tuned
Deployment Verdict
Conditional deploy, because tool execution is strong, but synthesis fidelity — with a Combined of 67.83 and an invalid tool call status — is not reliable enough for unattended high-trust pipelines.
Tool Execution Profile
Gemma 4 31B Instruct demonstrates genuine tool intelligence, not just rigid schema behavior. On the Web Search & Tool Selection test, which checks whether the model selects search over direct fetch without being prompted, it consistently chooses the correct tool. This points to workable planning logic in dynamic MCP workflows. On the URL Construction & Fetch test, which measures precise target URL derivation followed by retrieval, it remains usable at P1 80, but not deterministic. That is precisely where the operational caveat lies: it usually understands the next step correctly, but does not consistently produce a protocol-clean, valid call. The fact that tool_call_valid is false overall weighs more heavily than the high P1 average of 90. No retry was needed. That argues against a mere formatting issue and more toward inconsistent execution at the individual call level.
Synthesis Fidelity
How well does it consolidate tool results? Only to a limited degree. P2 49.17 is the weakest part of the profile for production tool pipelines. In EU License Research and Multilingual Search & Synthesis, it retrieves information via tools but compresses it too coarsely or omits critical qualifications. On HTTP Fetch & Extract, consolidation is more solid, but not yet at the level one would clear for reliable extraction chains without review.
Does it stay within tool output or fall back on training data? In the honeypot EU License Research, the trust signal remains fundamentally intact: no detected hallucination, even though the test specifically checks whether current license restrictions are answered from web sources rather than from training. That matters. The weak P2 score of 40 here reflects poor consolidation rather than fabricated facts. For compliance-adjacent workflows, that is significantly better than hallucination, but still not a clearance for full automation.
Error Resilience
On the 404 test, which measures transparent behavior when a tool call fails, the model responds in a production-appropriate manner. It does not hallucinate page content and communicates the error with sufficient clarity at P2 80. That is a robust positive signal. Errors are surfaced, not concealed. In real MCP pipelines, that matters more than polished phrasing.
Sovereignty Profile
Locally deployable with Apache 2.0 weights, making it attractive for sovereign environments. At the same time, it sits 1.22 points below the fleet average of 66.87. That is close enough to the mean for local deployment arguments to hold, but not strong enough to offset the quality gaps in synthesis.
Conclusion & Recommendation
Suitable for local, sovereign research and orchestration pipelines where tool selection and error transparency matter more than perfect result consolidation. Not suitable for compliance, policy, or extraction workflows where the verbal summary itself is the product and is processed downstream without human review. If you deploy this model, do so as a tool-aware operator with downstream validation — not as the final authoritative synthesis layer.
This analysis was generated automatically based on benchmark data. Model used: GPT 4.5 by OpenAI. The raw data and full methodology are documented in the GitHub project.
