Qwen 3.8 27B (Thinking)

With an overall score of 77.44%, Qwen 3.8 27B delivers a remarkably mature showing in the Thinking mode evaluated here. The model is classified as a Generalist, runs in the Workstation class, and enters the field with 27.8 billion dense parameters. For that, it plays broadly, technically, and often with genuine substance. The Speed Profile Badge “Batch DevOps Expert” fits surprisingly well: not snappy, but strategic, thorough, and clearly designed for longer technical workloads. As a locally usable Open Weights model, this is the kind of system that has no interest in being polished and charming — it wants to be useful.

Header Grades: Stability and Reliability

Architecture and Character

The pre-assigned categorization hits the mark fairly well. Qwen 3.8 27B is not a narrow specialist, though its coder- and agentic-leaning DNA is clearly perceptible. As a Generalist, it must hold its own across the board. That is precisely where it usually convinces through structure rather than elegance. As a dense model, its 27.8 billion parameters reflect its actual active capacity. There is no MoE trick here that looks large on paper and shrinks at runtime. The claim for the Workstation class is therefore high — and it is legitimate.

The specific test mode also matters: this report refers explicitly to Thinking mode. That is not a cosmetic switch; it shapes the character of the entire run. Responses are longer, more analysis-heavy, and often more robust. Particularly for logic and code tasks, that is desirable. The price, however, shows up brutally in stability and outliers.

The model’s multimodality remains naturally peripheral in this text-centric benchmark. Qwen 3.8 27B is natively multimodal, but this test measures almost exclusively its text work. That is not a weakness of the benchmark — just a clean qualification: anyone seeking a verdict on image or video understanding will find only half the biography here.

Speed and Runtime Profile

On the local reference system ASUS GX10 / NVIDIA DGX Spark (GB10 Grace Blackwell Superchip, ~115 GB Unified Memory — no practical memory limit for tested model sizes), Qwen 3.8 27B exhibits the behavior its badge already hints at: “Batch DevOps Expert” describes a model that does not feel trained for quick prompts, but for longer technical sessions, reviews, and structured problem decomposition. Generation quality is moderate to low. In practice, that means: plausible for background work, unpleasant for tightly timed interaction.

A second effect compounds this. In the Thinking run, this model is clearly more verbose than the fleet average. In Reasoning and metacognition, that is acceptable and often sensible. But across several practical modules, Qwen 3.8 27B also produces noticeably more output text than the median of other tested models. Particularly striking is UX Writing, where it exceeds the expected verbosity range. For a local model, such overruns are primarily a latency signal. More text does not mean higher cost per API call here — but it does mean more waiting time. And waiting time is something this run has in ample supply.

Code and Security: Technically Sharp, Practically Valuable

Qwen 3.8 27B leaves its strongest calling card in the technical core. The Code Quality section reveals a model that not only identifies security vulnerabilities but ranks, explains, and makes them actionable with meaningful granularity. In the audit at hand, it works through a PHP application with rare completeness: SQL Injection, XSS, CSRF, Session Fixation, Path Traversal, insecure cookies, weak tokens, Type Juggling, IDOR. Even implicit vulnerabilities such as Mail Header Injection or secondary injection paths are cleanly named. This is not a mere keyword collection — it is a model that has understood the mechanics behind the flaw.

In a security context, that matters. Many models find the loudest vulnerabilities and stumble on the quiet ones. Qwen 3.8 27B does the opposite. It flags the less obvious attack chains as well, even if its explanations do not always reach the final level of didactic rigor. The Judge notes somewhat terse justifications and less fully developed attack-chain context compared to the reference standard. That is a valid deduction, but not one that calls the substance into question. The security-related competence here is real.

The flip side is operational. In the Code Quality module, stability is devastating. A model that analyzes brilliantly but fails in four out of five runs behaves like a brilliant auditor who is only reachable on Fridays. For supervised one-off cases, that is still workable. For automated pipelines, it is dangerous.

CLI, Tool Use, and Agentic Quality

The strong CLI performance aligns cleanly with the agentic classification. Qwen 3.8 27B plans technical steps well, structures command sequences clearly, and rarely seems lost in terminal-adjacent tasks. Models like this do not need to produce every one-liner with the cold precision of a shell specialist, as long as they decompose the task strategically. That is precisely where a strength lies here.

There is, however, a serious flaw in the Tool Use area. In one task, the model hallucinated content that did not originate from the actual tool result. The score was consequently capped by the hallucination penalty. For content-critical tasks — research, reports, or fact-bound agentic workflows — this is not a cosmetic issue but a warning signal. An agent that uses the toolbox but then freely supplements the measurement result is not reliable; it is dangerously creative.

This does not eliminate the agentic suitability entirely, but it sets clear limits. For technical planning, analysis, and workflow preparation, Qwen 3.8 27B is well suited. For tool-bound factual records, it requires oversight, guardrails, and ideally a verification step.

Reasoning: Strong, Structured, but Not Maximally Deep

In the Reasoning module, Qwen 3.8 27B demonstrates exactly the kind of thinking one expects from a Thinking run. It solves classic logic problems correctly, structures its responses into traceable steps, and considers multiple approaches rather than selling the first plausible idea as truth. On the guard puzzle, for instance, it cleanly works out the double inversion and explains the solution clearly. That is not spectacularly novel, but it is solidly mastered. And solidly mastered reasoning is worth more in a benchmark than stylized brilliance.

At the same time, a pattern remains visible: the model is frequently correct, but not always maximally developed. Tables, comparative structures, and alternative formulations are occasionally absent where they would have made the line of reasoning more robust. The gap from the reference standard lies less in logic than in didactic completeness. Qwen 3.8 27B thinks correctly. It just does not always explain down to the last detail.

This, incidentally, fits the dual role of Reasoning and Instruct rather well. On one hand, the model wants to argue thoroughly; on the other, it stays task-oriented and avoids unnecessary theoretical detours. The result is often useful, sometimes a little dry. But dryness is easier to forgive in a technical environment than ornamental emptiness.

UX Writing: Strong Craft, Impractical Verbosity

In the UX Writing section, Qwen 3.8 27B delivers substantively impressive work. Jargon is reliably removed, progressive disclosure is correctly applied, psychological principles are cleanly named, and the identified problems are explicitly addressed with solutions. The Judge credits the model with a complete, precise response that is in parts even substantively stronger than the reference standard. That deserves respect, especially since UX Writing does not belong to the classic comfort zone of technically oriented models.

But the medal has a very ugly edge here. This module is simultaneously one of the most unstable in the entire run. Moreover, Qwen 3.8 27B is far too verbose here. It produces considerably more text than necessary and is the only module to exceed the expected verbosity range. For a human reader, this may still pass as thoroughness. For productive workflows, it is dead weight. Good microcopy must be precise. A model that delivers it with analytical excess misses part of the discipline the genre demands.

Content Transformation and Cultural Intelligence: Talent with a Compliance Weakness

Here Qwen 3.8 27B becomes more human — and in this case, that is not a compliment. In content transformation, it can build strong texts, structure production-ready scripts, and place good hooks, pattern interrupts, screen cues, and CTA elements. The material clearly shows that the model understands story and format mechanics. The dramaturgical hand is visible, particularly in the video script example.

But then comes the old affliction of many high-performing models: it does not cleanly follow the rules. In one content task, Qwen 3.8 27B ignored the explicit German target language and responded in English. In another task in the same module, it did so again. This is no longer a slip — it is a structural signal. Across multiple tasks in the Content Transformation section, the model shows a consistent pattern: when faced with simultaneous constraints on language, length, and format, it drops the language requirement first. In production editorial or marketing setups with a fixed output language, this is a real deployment risk.

There is also an automatic Hard Constraint violation. In one Content Transformation task, the model exceeded the explicit word limit of 900 words, reaching 1,345 words — 149% of the limit. The system applied an automatic deduction of 20%, equivalent to 16.20 points off the achievable sub-score. The content quality of the response is therefore irrelevant. The penalty applies regardless. This is precisely where a benchmark must remain strict.

In the same module, a substantively critical gap also appeared: a required troubleshooting section in a video script was missing entirely. That is not a matter of taste — it is a completion failure on an explicitly required structural component.

The Cultural Intelligence side then shows the other face of the model. There, Qwen 3.8 27B identifies toxic or culturally inappropriate phrasing well, rewrites inclusively, and hits the professional tone. Only once does it lose the final level of instruction discipline: instead of outputting exclusively the revised target text, it appends explanatory justifications. That is intelligent, but disobedient. An editor appreciates it. A production system does not.

Documentation: Technically Strong, Linguistically Not Clean Enough

In the Documentation section, Qwen 3.8 27B performs strongly overall. The module score is high, which aligns with its technical profile. It can document, structure, and present complex relationships in a way that keeps technical depth from collapsing under its own weight. For a model with coder and agentic leanings, this is almost the ideal secondary competency.

But here too there is a documented language error. In one Documentation task, the model ignored the explicit language instruction and responded in English. This is not a technical defect — it is a weakness in instruction following. Together with the two language violations in the Content module, a clear pattern emerges: Qwen 3.8 27B is strong at thinking and formulating, but not reliably enough in language adherence under multiple simultaneous constraints.

Hallucinations

The Dangerous Moment in Tool Use

Because at least one significant hallucination case occurred, the topic warrants its own look. The critical finding comes from a Tool Use task. There, Qwen 3.8 27B generated content that could not be derived from the actual tool result. This is not poetic embellishment — it is a fabricated factual layer. The benchmark deliberately caps the score in such cases, because for research, factual reports, and agents with external data sources, this is precisely where the question of trust is decided.

This is all the more frustrating because the model otherwise frequently conveys the impression of a methodical, controlled system. Hallucinations of this kind damage that impression more severely than a plain knowledge error. Anyone seeking to couple Tool Use with reliable factual fidelity must not be lulled by the model’s otherwise strong competence.

Privacy and Data Sovereignty

A dedicated privacy block is not necessary for this run, as Qwen 3.8 27B was operated with local weights. The provenance of the weights remains relevant nonetheless: the Weights Provenance Risk is rated MEDIUM, as the model was developed by Alibaba in China. In practice, this risk is substantially mitigated by the Apache 2.0 license and local operation, since no data is transmitted to the developer.

Conclusion

Qwen 3.8 27B is an interesting piece of model engineering: a generalist, dense Workstation model that, in Thinking mode, comes across as more intelligent than many nominally larger candidates across large stretches of the benchmark. Code, Security, CLI, and technical Reasoning are clearly among its strongest suits. Documentation also lands strongly. The model has substance, and it shows that it prefers to analyze rather than improvise.

But one should not be deceived by that substance. The header grades are a massive warning. The timeout rate is disastrous for unattended deployment, the tail latency is critical, and on language constraints, Qwen 3.8 27B shows a genuine compliance weakness across multiple tasks. Add to that the hallucination case in Tool Use. For agents that pull facts from tools and process them onward without human oversight, this is a problem with sharp edges.

In a direct family comparison, the Thinking run shows a clearly stronger overall character than the standard run of the same model: higher score, better technical depth, noticeably more mature in Code and Documentation. The price is equally clear: more text, more batch character, more operational weight. Those who deploy Qwen 3.8 27B locally for security reviews, code audits, technical documentation, or deeper analytical tasks get a serious tool. Those looking for a fast, failure-resistant everyday model or an unattended production agent should move on. This model is not a sprinter. It is a workshop instrument with a sharp blade and a temperamental switch.

This evaluation was generated automatically based on the benchmark data. Model used: GPT 4.5 by OpenAI. The raw data and full methodology are documented in the GitHub project.