Tool-use review
Created
Deployment Verdict
Conditional deploy, because tool execution is strong, but protocol fidelity and synthesis fidelity are not stable enough to hand the model an MCP infrastructure unsupervised. The combined score is good, but tool_call_valid: false and the detected hallucination signal are a hard reservation for production.
Tool Execution Profile
Ornith 1.0 9B demonstrates genuine tool intelligence, not just rigid task execution. In the Web Search and Tool Selection test, it recognizes without explicit prompting that search is required before a direct fetch, and resolves this cleanly. That speaks to usable planning logic in dynamic pipelines. In the URL Construction test, it constructs the target URL correctly in principle, but not with sufficient precision for deterministic workflows. This is not a comprehension breakdown — it is a last-mile accuracy problem.
The tool_call_valid: false finding is critical. The strong P1 performance shows that the model mostly selects and applies tools sensibly. However, formal and schematic execution is not consistently MCP-clean. Since no retry was required, the issue is more likely isolated call validity rather than systematic format collapse.
Synthesis Fidelity
How well does it consolidate tool results? Only limitedly reliable. The P2 side, at 52.50, is the clear weak point. Structured extraction from fetched content still holds up solidly, and synthesis quality also holds its level on URL Construction & Fetch. As soon as multiple sources, multilingual content, or regulatory details need to be combined, precision visibly breaks down. EU License Research and Multilingual Search & Synthesis in particular show that the model does not cleanly translate results into reliable final answers.
Does it stay within the tool result or fall back on training? In the honeypot EU License Research test, which probes exactly this behavior, it does not openly hallucinate from training. That is a positive. Nevertheless, the global hallucination signal must be treated as a security risk. Once a model in a tool pipeline outputs fabricated facts as apparently retrieved results, it is not only answer quality that is affected — it is trust in the entire infrastructure.
Error Resilience
In the 404 test, which checks for transparent handling of a failed tool call, Ornith remains controlled. It does not fabricate page content and communicates the error in a generally acceptable manner. P2 60 is not a standout score, but sufficient for production because the critical boundary holds: no hallucinated substitute content despite the error.
Sovereignty Profile
Locally deployable, MIT-licensed, and fleet-competitive with a Combined score of 70.75. The model sits n/a points below the fleet average of 67.75.
Conclusion & Recommendation
Suitable for local, sovereign pipelines with a human in the loop, where the model is expected to search, trigger fetches, and pre-structure raw material. A good fit for agentic research or coding scaffolds on Edge hardware. Not suitable as the final authority for compliance, license review, multilingual synthesis, or other workflows in which the final answer must qualify as verified tool output. If you deploy it, do so with strict tool call validation, response constraints, and downstream verification of the final synthesis.
This analysis was generated automatically based on benchmark data. Model used: GPT-5.4 by OpenAI. The raw data and full methodology are documented in the GitHub project.
