Skip to content
CrucibleMark

Qwen 3 4B

Four billion parameters, Apache 2.0 license, and an optional thinking mode: Qwen 3 4B is designed for mobile applications and edge setups where every watt counts. Locally deployable under Q6 quantization, with a 128,000-token context window — generously sized for a Nano-class model. The manufacturer’s jurisdiction of China occasionally results in censored or evasive responses on politically sensitive topics.

Alibaba Version 3 Commercial use permitted Dense 4 B (4 B active) 128 K Context 09/2024 locally tested

  • Open Weights
  • Nano
  • llama.cpp
  • Text
  • Real-Time

Sovereign Risk: LOW The model is run locally without a cloud connection; the CLOUD Act and data transfer risks associated with cloud usage do not apply here. Provenance remains traceable through community quantization, though the operational risk is low with purely local inference.

Tool-use profile: 6 assets in detail

Comparison of asset performance (P1/P2/Combined) against the fleet average

Asset performance (radar)

Score Breakdown vs. Fleet Average

Tool-use details

Asset performance, reliability, and runtime profile

Name P1 P2 Combined
EU License Research 100 60 76
HTTP Fetch & Extract 80 35 57.5
Tool Failure Handling (404) 80 80 80
Web Search & Tool Selection 100 60 82
URL Construction & Fetch 80 60 70
Multilingual Search & Synthesis 100 15 57.5

CrucibleMark evaluates tool use across 6 independent tests. Click a test name for details.

Reliability

  • Tool Call Valid: Yes
  • Retry: Not required
  • Hallucination: Detected

Reliability measures how consistently a model actually executes tool calls: Tool Call Valid schema and format accepted, Retry Required successful only after retry, Hallucination Flag fabricated tools or parameters detected. All three green means production-ready.

Runtime profile

Call 1
2.24
First Request
MCP
0.79
Protocol Latency
Synthesis
7.06
Response Generation
Total
60.49
Sum of All Phases
Token
7838
Input + Output
Cost
$0
Cost per Run

The runtime profile shows the latency and cost metrics for the model run: Call 1 First Request, MCP Protocol Latency, Synthesis Response Generation, Total sum of all phases, supplemented by Token input and output and Cost cost per run.

Tool-use review

Created

Deployment Verdict

Conditional deploy, because Qwen 3 4B produces valid tool calls and demonstrates usable tool execution overall, but synthesis quality — given the hallucination findings — is not stable enough for production decision or compliance pipelines.

Tool Execution Profile

The model is clearly stronger on the execution side than on the response side. Tool calls were valid, MCP-protocol-compliant, and executable without retry. This argues against a formatting issue and in favor of fundamentally clean interface behavior.

On Web Search & Tool Selection — the test of whether web_search is chosen over fetch without an explicit hint — Qwen 3 4B reliably identifies the correct tool type. That is a good signal for dynamic pipelines. On the URL Construction test, which checks whether the model can derive a target URL on its own and then retrieve it via fetch, it is usable but not deterministic enough. It thus demonstrates genuine tool selection rather than rigid pattern recall, but loses precision as soon as it has to construct the target address itself.

Synthesis Fidelity

How well does it condense tool results? Only reliably to a limited extent. P2 performance is the clear bottleneck: extraction precision drops noticeably on HTTP Fetch & Extract, and condensation essentially collapses on Multilingual Search & Synthesis. For pipelines that require reliable short answers, structured reasoning, or clean factual summaries from tool output, this is too weak.

Does it stay within the tool result or fall back on training? On the Honeypot EU License Research test — which checks whether current license restrictions are actually retrieved from web sources — the model stays cleanly within the tool result. That is the important trust signal. At the same time, the global hallucination finding is a security risk: when a model in a tool pipeline outputs fabricated facts as tool results, it undermines the reliability of the entire infrastructure.

Error Resilience

On the 404 test, which checks for transparent behavior when a tool call fails, Qwen 3 4B responds in a production-appropriate manner. It communicates the error rather than fabricating page content. This behavior is precisely what is acceptable for operational pipelines, because the orchestrator can then make clean downstream decisions.

Sovereignty Profile

Locally deployable, Apache-2.0-licensed, and therefore operationally attractive for sovereign deployments. On the performance side, it sits only 0.75 points below the fleet average of 66.55. For a Nano model, that is a solid ratio of control, resource requirements, and usable tool competence.

Conclusion & Recommendation

Suitable for local MCP pipelines in which the model primarily selects tools, executes calls cleanly, and reports errors transparently. Not suitable as the final authority for fact-critical synthesis, multilingual research condensation, or compliance-adjacent outputs without downstream verification. Recommendation: deploy as a lightweight tool operator or upstream stage in Edge and sovereignty setups, but delegate the final response layer to a stronger model or a strict validator.

This analysis was generated automatically based on benchmark data. Model used: GPT 4.5 by OpenAI. The raw data and full methodology are documented in the GitHub project.