Deterministic output agreement
Reference execution and GPU-routed expert execution generated identical output across the fixed qualification corpus
Evidence
Measured results with explicit scopeWhat MER has demonstrated—and what it has not.
The evidence separates deterministic output qualification, a controlled earlier warm-path test, real NVIDIA L4 validation, and historical CPU baselines. None of these results is presented as completed GPU-native generation or commercial throughput.
What the evidence answers today
Each result answers a specific engineering question. No card below is a claim about completed production throughput.
Controlled warm-path decode
Earlier host-driven NVIDIA L4 test: 1.3798886916 to 1.5612554031 generated tokens/s; not current GPU-native throughput
Fewer host synchronizations
The controlled batching test reduced warm-path synchronization operations from 6,144 to 768
Intermediate host re-entry
The NVIDIA-L4-qualified GPU foundation uses no intermediate production maps or readbacks
1 / Does it produce correct output?
Deterministic output qualification
A CPU reference that reproduces the production expert input/output boundary and the GPU-routed expert path produced identical greedy generated output across the fixed qualification corpus. This establishes a correctness reference for deeper GPU integration without implying that every ordinary CPU control run must match.
2 / Has reducing host work helped?
Controlled warm-path batching test
- Baseline mean decode
- 1.3798886916 generated tokens/s
- Batched-path mean decode
- 1.5612554031 generated tokens/s
- Observed decode improvement
- +13.1436%
- Baseline mean end-to-end
- 1.3290345164
- Batched-path mean end-to-end
- 1.4933532175
- Observed end-to-end improvement
- +12.3638%
- Warm host synchronizations
- 6,144 → 768
- Reduction
- 87.5%
Controlled NVIDIA L4 alternating-order result from an earlier host-driven GPU-expert path. It is not the current GPU-native loop, production throughput, or evidence that the ≥10 TPS target is achieved.
3 / Has the GPU path run on real hardware?
GPU-native foundation on NVIDIA L4
Device-resident execution state
Request-local hidden and residual state, authoritative WGPU device reuse, and non-mappable intermediate buffers.
Persistent weights and core operations
Persistent F32 and native Q8_0 weights, GPU GEMV, embedding lookup, request scratch, and binding-limit chunking; NVIDIA-L4 qualified.
Q/K/V, RoPE, and KV append
RMSNorm and residual transitions, grouped per-head Q/K normalization, Q/K/V projections, RoPE, and request-local F32 GPU KV append; NVIDIA-L4 qualified.
4 / What historical baselines exist?
Historical CPU full-transformer baseline
The earlier strict Qwen Q8 CPU measurements remain useful provenance. They no longer represent the project’s top-level architecture or latest engineering milestone.
Layer-qualified experts
48 transformer layers × 128 experts per layer
Learned expert routing
Real LinearGate selection on the verified Qwen path
Required dense tensors loaded
Strict checkpoint mode with no seeded fallback
Generated-output parity
Matching generated output across measured runs
Historical baseline environment
The result is configuration-specific.
Model
- Checkpoint
- Qwen3-Coder-30B-A3B-Instruct Q8_0
- Architecture
qwen3_moe- Geometry
- d_model 2048, d_ff 768, 48 layers, 128 experts per layer
- Routing
- Learned top-8 selection
Environment
- Machine
- GCP g2-standard-32, 32 vCPUs, 128 GB RAM
- CPU
- Intel Xeon reported at 2.20 GHz, AVX-512 available, CPU-only
- Storage
- Boot-disk-backed model path
- Converted model
- Approximately 31 GB
Historical CPU real-inference baseline
1,536 cache slots / 25% expert residency
- Prompt throughput mean
- 0.502 tokens/s
- Decode throughput mean
- 0.551 generated tokens/s
- TTFT p50
- 61.780 seconds
- Mean total runtime
- 292.135 seconds
- Cache hit rate
- Approximately 71.9%
- Expert misses per run
- Approximately 17,043
- Expert data read per run
- Approximately 85.7 GB
- Foreground SSD stall
- Approximately 231.9 seconds per run
- RSS
- Approximately 17.8–18.1 GiB
- Expert compute
- Approximately 29.7 seconds per run
- Output parity
- True
Cache tradeoff
Memory savings increased foreground storage work.
Decode throughput
Generated tokens per second by real Qwen cache configuration.
Approximate RSS
Midpoint of the observed resident-set-size range.
Foreground SSD stall
Critical-path expert storage wait per measured run.
| Cache configuration | Residency | Raw cache | Prompt | Decode | TTFT | Hit rate | Misses | Data read | SSD stall | RSS |
|---|---|---|---|---|---|---|---|---|---|---|
| 768 slots | 12.5% | 3.59 GiB | 0.306 tok/s | 0.330 tok/s | 101.513 s | 53.04% | 28,492 | 142.96 GB | 429.98 s | 9.9–10.2 GiB |
| 1,536 slots | 25% | 7.18 GiB | 0.502 tok/s | 0.551 tok/s | 61.780 s | 71.9% | 17,043 | 85.7 GB | 231.9 s | 17.8–18.1 GiB |
Synthetic cache matrix
A separate systems benchmark, with separate units.
Synthetic iteration rate
Benchmark iterations per second across cache configurations.
Synthetic hit rate
Resident expert hit rate under the skewed synthetic workload.
Synthetic SSD stall
Foreground storage stall during the synthetic run.
| Cache slots | Residency | Raw cache | Synthetic iterations/s | Hit rate | Misses | I/O share | SSD stall |
|---|---|---|---|---|---|---|---|
| 384 | 6.25% | 1.79 GiB | 101.502 | 92.79875% | 5,761 | 65.58% | 63.5101 s |
| 768 | 12.50% | 3.59 GiB | 125.761 | 94.68625% | 4,251 | 49.05% | 31.9680 s |
| 1,536 | 25.00% | 7.18 GiB | 76.703 | 96.13625% | 3,091 | 29.54% | 14.1372 s |
| 2,976 | 48.44% | 13.91 GiB | 60.212 | 96.6575% | 2,674 | 27.27% | 12.9498 s |
The 768-slot configuration produced the highest synthetic iteration rate, but that result did not transfer to real learned Qwen routing. Synthetic routing is useful for systems analysis but cannot replace full-transformer validation.
Methodology
A strict, repeatable execution scope.
- Execution
- CPU-only, strict converted Qwen checkpoint, greedy decoding
- Sequence
- 31 prompt tokens and 128 completion tokens
- Transformer work
- 158 full transformer forwards per run
- Runs
- One warmup and three measured runs
- Cache policy
- Reset policy: keep
- Prediction
- Neural speculator disabled
- Correctness
- Generated-output parity checked across measured runs
These results are internal benchmark measurements from the listed configuration. Reproduction artifacts and independent third-party validation remain part of the ongoing validation roadmap.
5 / What has not been proven?
What remains to be proved
- Complete GPU-native autoregressive generation
- Production throughput
- ≥10 real generated tokens/s on an oversized virtualized MoE
- A universal model-to-VRAM multiplier
- Production-qualified predictive NVMe → RAM → VRAM overlap
- 235B production qualification