Skip to content
CrucibleMark

Test protocol: Qwen3.8-27B thinking modes in CLI benchmark

Note – This page is the raw protocol for the article "But wait… Can machines think too much?" and serves as supporting documentation only. It is not included in the magazine index.

Qwen 3.8 27B NVFP4 — thinking mode comparison (quality / tokens / TPS)

  • Model: Qwen 3.8 27B NVFP4 (Dense, MTP, 1:3-Interleaving 16 full_attn + 48 Mamba2, 262K Context)
  • Hardware: DGX Spark (asus_gx10_blackwell)
  • Inference: vLLM 0.27.2rc1.dev126 (MOD thinking-budget-by-effort)
  • Provider: vllm_spark
  • Judge: Anthropic (Hybrid)
  • Module: cli_benchmark (6 assets)
  • Date: 2026-08-17
  • Pipeline: v5.1.5 (real provider tokens, TPS = output_tokens / wall-time)

TL;DR

Mode Quality Out-tokens TPS Time Verdict
instruct (off) 86.0 425 17.9 24 s strong budget option
low 82.4 2 515 23.5 107 s worst
medium 93.0 3 244 26.3 124 s Sweet-Spot
xhigh 86.7 15 878 11.5 1 385 s Overkill (≈ instruct)

Key takeaways

  1. medium is the Sweet-Spot — highest quality (93.0) and fastest TPS (26.3) at moderate token cost.
  2. xhigh yields no quality gain — 86.7 ≈ instruct (86.0), but 5× the tokens and 58× the time. The extra thinking is noisy and actually degrades individual tasks (cli006: 58 vs. medium 100).
  3. low is counterproductive — 82.4 is below instruct (86.0). A minimal thinking budget pays the token cost without delivering the quality gain.
  4. instruct (off) is the best latency/budget option: 86.0 in 24 s (1/58 of xhigh time).

Methodology

Each mode runs the same cli_benchmark module (cli001–cli006) once, sequentially against the running vLLM server. The 4 configurations:

Mode Model ID enable_thinking reasoning_effort max_tokens
instruct qwen3_8-27b-nvfp4 off — 16 384
low qwen3_8-27b-nvfp4-thinking on low 32 768
medium qwen3_8-27b-nvfp4-thinking on medium 32 768
xhigh qwen3_8-27b-nvfp4-thinking on xhigh 32 768
  • enable_thinking is controlled via chat_template_kwargs per request (dual-profile expansion).
  • reasoning_effort is sent per request; the MOD thinking-budget-by-effort derives the thinking_token_budget from it. No server restart between modes (per-request parameter).
  • The CSV is append-based with a unique run_id per run. For each mode, the new run_id was determined via before/after diff and the 6 rows saved as a snapshot (outputs/analysis/qwen3_8_thinking_modes/).
  • --force enforced a fresh run (cache ignored). reasoning_effort was reset to the committed value (medium) after the run.

Metric definitions (v5.1.5):

  • tokens_per_second (TPS) = output_tokens / execution_time — actual decode throughput incl. thinking tokens.
  • output_tokens = visible response + thinking; reasoning_tokens = thinking portion; input_tokens = prompt.
  • Aggregated TPS is weighted by execution_time.

Core metrics per mode

Mode Avg. score input output reasoning TPS (wtd.) Total time
instruct 86.0 679 425 0 17.87 23.8 s
low 82.4 823 2 515 1 454 23.54 106.8 s
medium 93.0 667 3 244 2 376 26.28 123.5 s
xhigh 86.7 895 15 878 15 475 11.46 1 385.1 s

Quality per asset (score 0–100)

Asset instruct low medium xhigh
cli001 Disk Cleanup 72.0 88.4 100.0 86.0
cli002 Library Install 72.0 72.0 86.0 86.0
cli003 Repo Clone 86.0 100.0 100.0 100.0
cli004 Zshrc Alias 100.0 100.0 86.0 100.0
cli005 SwarmUI Docker 86.0 62.0 86.0 90.0
cli006 Ollama Symlink 100.0 72.0 100.0 58.0
Avg. 86.0 82.4 93.0 86.7

How to read this: medium wins clearly 3× (cli001, cli002, cli006) and shares the top spot 2× (cli003). xhigh is high-variance: best individual score at cli005 (90), but by far the worst at cli006 (58) and weaker than medium at cli001 (86 vs. 100). The extra thinking is therefore not consistent — it helps on some tasks and hurts on others.

Token consumption & TPS per asset

Asset Mode output reasoning TPS Time
cli001 instruct 114 0 18.82 6.1 s
cli001 low 1 045 256 20.37 51.3 s
cli001 medium 1 357 778 25.71 52.8 s
cli001 xhigh 6 392 6 222 6.96 918.8 s
cli002 xhigh 2 024 2 000 19.46 104.0 s
cli005 instruct 185 0 24.85 7.4 s
cli005 medium 690 523 30.44 22.7 s
cli005 xhigh 5 871 5 768 20.73 283.2 s
cli006 xhigh 567 529 20.29 27.9 s

(Selection of the most relevant rows; complete data in the snapshots.)

Token burn concentrates on the hard tasks: xhigh burns 12,263 tokens on cli001 (6,392 out) and cli005 (5,871 out) combined — that is 77% of xhigh's output budget. On the easy tasks (cli003, cli006), xhigh stays moderate (146 and 567 out respectively). Time distribution: cli001 (919 s) + cli005 (283 s) = 1,202 s = 87% of total xhigh runtime.

Efficiency metrics

Mode Thinking share* Tokens/task Score / 1k tokens Score / second
instruct 0.0 % 184 467.4 21.70
low 57.8 % 556 148.1 4.63
medium 73.2 % 652 142.7 4.52
xhigh 97.5 % 2 796 31.0 0.38

* reasoning_tokens / output_tokens — what share of the output consists of pure thinking.

  • Thinking share: xhigh is 97.5% thinking (almost entirely reasoning, barely any visible response). medium 73.2%, low 57.8%.
  • Score/1k tokens (quality per token): instruct dominates (467), because there is no thinking overhead. xhigh is the least efficient (31) — 15,878 tokens for only 86.7 score.
  • Score/second (quality throughput): instruct is fastest (21.7), xhigh is slowest (0.38).

Important: "Score/1k tokens" and "Score/second" favor instruct because these metrics do not capture the absolute quality gain on hard tasks. instruct only scores 72 on cli001 (medium: 100). The efficiency metrics are a cost-reduction signal, not a quality signal.

Analysis & findings

  1. medium = clear recommendation. Best average quality (93.0) combined with the fastest TPS (26.3) and moderate cost (3,244 output tokens, 124 s). The medium thinking budget hits the Sweet-Spot: enough reasoning for the hard tasks (cli001 → 100) without the xhigh overhead.

  2. xhigh is overkill with noise. The quality gain over medium is negative (86.7 < 93.0) and over instruct negligible (86.7 ≈ 86.0). The price: 5× tokens, 58× time, TPS halved. The extra thinking is inconsistent — it lifts cli004/cli005 but drags down cli001/cli006. Not justifiable for routine tasks; only sensible for one-off hard problems.

  3. low is the worst mode. 82.4 is below instruct (86.0). A minimal thinking budget pays the token/time cost without delivering sufficient quality gain — on cli005 (62) and cli006 (72) it is even clearly worse than no thinking at all. Small budget ≠ proportional quality.

  4. instruct (off) is the latency banker. 86.0 in 24 s. For latency-sensitive or high-volume use, the best price-to-performance ratio — only 7 points below medium, but 1/5 of the time and 1/3 of the tokens.

  5. TPS caveat (xhigh): The low xhigh TPS (11.5) is partly a measurement artifact — cli001 had request retries/timeouts (918 s) that inflate wall-time while output_tokens counts only the final response. The actual decode speed is higher; the effective throughput slowdown is real, but not purely hardware-related.

Limitations / caveats

  • n = 6 assets per mode — small sample size; individual outliers (cli006 xhigh = 58) carry significant weight.
  • One module (cli_benchmark) — representative of practical CLI problem-solving, not of all task types. More reasoning-heavy modules (reasoning, code_quality) could show different patterns.
  • One run per mode — no repetitions/variance; scores may shift on re-runs.
  • xhigh retries distort the TPS measurement (see finding 5).
  • CLI tasks are relatively short; with longer, more complex outputs the quality/token ratio could shift.

Recommendation

Scenario Recommended mode Rationale
Default / best quality medium 93.0 score, fast, moderate
Latency / budget / high-volume instruct (off) 86.0 in 24 s, 1/5 of the time
Hard single problem (1 task) xhigh only where extra reasoning is worth the 58× time cost
Avoid low worst quality (82.4)

Concrete suggestion for provider_config.yaml: reasoning_effort: medium as the default for qwen3_8-27b-nvfp4 (already set). xhigh not as default — only as an explicit, deliberate choice for hard problems.

Appendix

Run IDs & snapshots (outputs/analysis/qwen3_8_thinking_modes/):

Mode run_id Model ID Snapshot
instruct f319329c062d qwen3_8-27b-nvfp4 instruct.csv
low f509a27713d0 qwen3_8-27b-nvfp4-thinking low.csv
medium bacf2a69dd4b qwen3_8-27b-nvfp4-thinking medium.csv
xhigh 661a9b7658df qwen3_8-27b-nvfp4-thinking xhigh.csv