Skip to content
CrucibleMark

Llama 4 Scout 17B

Llama 4 Scout is Meta’s multimodal fourth-generation Llama model, combining general language processing with image understanding in an efficient MoE architecture. Of 109 billion total parameters, only 17 billion are active per token; the context window spans 128,000 tokens. Available under the Llama 4 Community License, which contains restrictions for EU-based users regarding self-hosting and deployment.

Meta Version 4 Commercial use permitted MoE 109 B (17 B active) 128 K Context 12/2024 $0.11 / $0.34 per 1M

  • Restricted Weights
  • Server
  • Groq
  • Text
  • Vision
  • Instruction-Tuned
  • Real-Time

Sovereign Risk: MEDIUM Meta is a US company and subject to the CLOUD Act, which may allow government access to data when using the API. Weights are publicly available. The Llama 4 Community License excludes multimodal Llama 4 models for EU-domiciled entities with respect to self-hosting/deployment; end-user access via third-party APIs is to be assessed separately.

Tool-use profile: 6 assets in detail

Comparison of asset performance (P1/P2/Combined) against the fleet average

Asset performance (radar)

Score Breakdown vs. Fleet Average

Tool-use details

Asset performance, reliability, and runtime profile

Name P1 P2 Combined
EU License Research 35 0 14
HTTP Fetch & Extract 35 40 37.5
Tool Failure Handling (404) 35 40 36.8
Web Search & Tool Selection 35 40 37.3
URL Construction & Fetch 35 40 37.5
Multilingual Search & Synthesis 35 40 37.5

CrucibleMark evaluates tool use across 6 independent tests. Click a test name for details.

Reliability

  • Tool Call Valid: No
  • Retry: Required
  • Hallucination: Not detected

Reliability measures how consistently a model actually executes tool calls: Tool Call Valid schema and format accepted, Retry Required successful only after retry, Hallucination Flag fabricated tools or parameters detected. All three green means production-ready.

Runtime profile

Call 1
0.97
First Request
MCP
0
Protocol Latency
Synthesis
1.05
Response Generation
Total
12.08
Sum of All Phases
Token
4405
Input + Output
Cost
$0.0011
Cost per Run

The runtime profile shows the latency and cost metrics for the model run: Call 1 First Request, MCP Protocol Latency, Synthesis Response Generation, Total sum of all phases, supplemented by Token input and output and Cost cost per run.

Tool-use review

· Instruction-Tuned

Deployment Verdict

Do not deploy in production MCP tool pipelines: tool calls were not valid, a retry was required, and overall execution fails to establish reliable protocol trust — despite the absence of hallucinations.

Tool Execution Profile

Llama 4 Scout 17B does not exhibit reliable tool behavior. P1 remains flat at 35 across all tasks — a conspicuously uniform result. This does not suggest situational tool selection; it points to a rigid or only partially protocol-compliant pattern. In the Web Search & Tool Selection test, which forces a choice between search and direct fetch without an explicit hint, the model does not robustly recognize the requirement. In the URL Construction & Fetch test, which requires deriving the correct target URL from prior knowledge and retrieving it via fetch, execution likewise appears non-deterministic. The fact that retry_required=true is set here suggests a format or protocol issue rather than a pure content-comprehension failure. For MCP operation, this is precisely what matters: a model must not be “almost correct” in its tool output.

Synthesis Fidelity

How well does it consolidate tool results? Only to a limited degree. In five of six assets, P2 sits at 40 — a basic, serviceable level of consolidation. As soon as a task demands clean, source-bound synthesis of current web content, however, performance visibly collapses. The outlier is EU License Research, with P2=0.

Does it stay within tool results, or does it fall back on training data? The trust verdict is negative, even though no formal hallucination was flagged. In the EU License Research honeypot — which tests whether current licensing restrictions are answered from web sources rather than training knowledge — the model fails content verification entirely. Content Verification State B2 at P2=0 means in practice: the response is not reliably anchored to the retrieved sources. For compliance, policy, or regulated research pipelines, this is insufficient.

Error Resilience

The model responds acceptably to tool failures. In the Tool Failure Handling (404) test, which checks for transparent handling of a failed retrieval, it does not hallucinate substitute content and achieves P2=40. This is production-capable in the narrow sense: errors are surfaced rather than concealed. This strength does not, however, compensate for weak tool validity.

Sovereignty Profile

Locally operable, but not fleet-competitive. The Sovereignty Gap stands at -1.37 points below the fleet average of 67.84. There is also a hard constraint: under Meta’s license, EU-domiciled organizations are not permitted to self-deploy Llama 4. For sovereign EU deployments, the model is effectively ruled out.

Conclusion & Recommendation

Suitable at most for internal assistant workflows with human oversight, basic web summarization, and tolerable retries. Not suitable for autonomous MCP pipelines, compliance research, deterministic tool orchestration, or any chain in which tool calls must be formally correct and source-bound synthesis is mandatory. The model is often serviceable at the text level, but not robust enough to serve as an infrastructure component.

This analysis was generated automatically based on benchmark data. Model used: GPT 4.5 by OpenAI. The raw data and full methodology are documented in the GitHub project.