Signal 3.8 27B

Signal 3.8 27B by AgentionAI is a token-efficiency fine-tune on Qwen 3.8 27B: via self-distillation, the model generates roughly 57 percent fewer response tokens and thinking chains half the length, at equal or better quality. Licensed under Apache 2.0, available in a mid-range Q5 quantization with a 262,000-token context and local inference.

AgentionAI Version 3.8 Commercial use permitted Dense 27.8 B (27.8 B active) 262 K Context 12/2025 locally tested

Open Weights
Workstation
llama.cpp
Text
Vision
Instruction-Tuned
Batch

Sovereign Risk: MEDIUM TODO

Tool-use details

Asset performance, reliability, and runtime profile

Name	P1	P2	Combined
EU License Research	100	20	52
HTTP Fetch & Extract	80	80	80
Tool Failure Handling (404)	80	80	80
Web Search & Tool Selection	100	80	91
URL Construction & Fetch	80	80	80
Multilingual Search & Synthesis	100	60	80

Name

Combined

EU License Research

100

HTTP Fetch & Extract

Tool Failure Handling (404)

Web Search & Tool Selection

100

URL Construction & Fetch

Multilingual Search & Synthesis

100

CrucibleMark evaluates tool use across 6 independent tests. Click a test name for details.

Reliability

Tool Call Valid: No

Retry: Not required

Hallucination: Not detected

Reliability measures how consistently a model actually executes tool calls: Tool Call Valid schema and format accepted, Retry Required successful only after retry, Hallucination Flag fabricated tools or parameters detected. All three green means production-ready.

Runtime profile

Call 1: 3.43; First Request

MCP: 1.3; Protocol Latency

Synthesis: 30.18; Response Generation

Total: 209.43; Sum of All Phases

Token: 10989; Input + Output

Cost: $0; Cost per Run

The runtime profile shows the latency and cost metrics for the model run: Call 1 First Request, MCP Protocol Latency, Synthesis Response Generation, Total sum of all phases, supplemented by Token input and output and Cost cost per run.

Deployment Verdict

Conditional deploy: tool execution is strong, but the trust foundation breaks at a critical point — valid tool calls were not consistently achieved, and synthesis remains too imprecise for unattended pipelines on knowledge-critical web tasks.

Tool Execution Profile

Signal 3.8 27B demonstrates genuine tool intelligence, not just rigid sequential execution. On the Web Search & Tool Selection test — which checks whether the model selects search over fetch without being prompted — it achieves full execution reliability. This suggests the model recognizes its information need before issuing the tool call. On the URL Construction test, which requires deriving the correct target URL from prior knowledge, it is usable but not deterministic enough. That is precisely where the operational limit lies: the model often knows which tool is needed, but does not always produce a cleanly valid call. For MCP pipelines, this means good planning capability, but a wrapper should enforce URL normalization, argument schema validation, and call validation.

Synthesis Fidelity

How well does it consolidate tool results? Only adequately — not reliably. The P2 score of 66.67 reveals a pattern: HTTP Fetch & Extract, Tool Failure Handling (404), and Multilingual Search & Synthesis are workable, but consolidation is not consistently precise. For pure retrieval summaries, this is often sufficient. For compliance, policy, or contract surfaces, it is not sufficient without downstream review.

Does it stay grounded in tool output, or does it fall back on training data? This is exactly where the distrust signal appears. On the EU License Research Honeypot — which tests whether current license restrictions are drawn from web sources rather than training — synthesis drops sharply to P2=20. No hallucination was formally flagged. In practice, however, the test shows that the model does not bind tightly enough to its source. For production tool infrastructure, this is a trust problem, even without a hard hallucination flag.

Error Resilience

On the 404 test, which measures transparent behavior when a tool call fails, Signal 3.8 27B responds acceptably. It does not fabricate page content and communicates the failure with sufficient transparency. This is production-ready. Failing calls do not automatically compromise factual integrity here.

Sovereignty Profile

Fully locally operable, Apache-2.0 licensed, and therefore attractive for sovereign deployments. No sovereignty penalty is recorded; the reference score remains n/a points below the fleet average of 67.75. Local operation is thus not a discernible performance disadvantage in this test picture.

Conclusion & Recommendation

Suitable for MCP-backed research, routing, and assistant pipelines where tool selection matters and a downstream validator secures both the calls and result grounding. Not suitable as an unattended synthesis endpoint for compliance, license review, or other current web facts carrying significant liability exposure. Those who want to deploy locally and openly get a strong tool-use foundation model. Those who require strict source fidelity in the final response step should use it only with guardrails and verification in place.

This analysis was generated automatically based on benchmark data. Model used: GPT-5.4 by OpenAI. The raw data and full methodology are documented in the GitHub project.