Deterministic headroom
Account for persistent non-expert weights, context and active-request memory, hidden and residual state, temporary GPU work, in-flight uploads or evictions, and allocator/driver safety margin.
Technical overview
A GPU-owned loop over virtualized expert memory.
MER is a Rust execution engine for routed Mixture-of-Experts models. Its target architecture separates accelerator-owned transformer compute, managed expert residency across VRAM/RAM/NVMe, and a CPU control plane that services misses without owning the normal warm loop.
Execution architecture
Each plane has a different ownership boundary. The capability labels distinguish hardware-validated components from current integration work and later architecture stages.
- Validated foundation
State, weights, embedding, GEMV, norms, Q/K/V projection, RoPE, F32 KV append
- Currently being integrated
Causal attention scores, softmax, value aggregation, output projection
- Next architecture stage
GPU routing, Q4 expert execution, weighted combine
- Available today
SSD/NVMe expert store, RAM cache/staging, prediction machinery
- Upcoming architecture stage
VRAM expert arena, safe budget controller, tier-aware predictive movement
- Target path
Demand and predicted movement through NVMe → RAM → VRAM
- Current
Startup, loading, storage work, orchestration, telemetry, status
- Validated correctness
CPU reference and GPU-expert output qualification
- Target boundary
No warm hidden-state host re-entry after GPU integration
These capability labels separate validated components from integration work; the complete GPU-native generation loop is not finished.
- Warm target path
- GPU state → attention → router → resident expert → combine → next layer.
- Miss path
- NVMe/SSD → RAM → VRAM changes residency beneath the compute plane.
- Predictive path
- Likely-needed experts begin moving toward VRAM before demand, when confidence and resource budgets justify the work.
- Validated today
- GPU state, persistent dense weights, embedding, GEMV, RMSNorm, residual transitions, grouped Q/K normalization, Q/K/V projections, RoPE, and F32 GPU KV append.
- Not complete today
- Causal attention and output, GPU router/top-k, routed Q4 expert arena and combine, residency reconnection, and full autoregressive generation.
VRAM budget policy
Spend nearly all remaining safe VRAM on expert residency.
Managed VRAM expert arena
Give essentially the entire remaining safe budget to layer-qualified expert residency. The objective is maximum safe occupancy, not literal 100% allocation.
RAM and NVMe namespace
RAM stages and caches expert payloads while NVMe/SSD provides the larger backing capacity. Demand and eviction determine what remains hot in VRAM.
I/O path
Storage specifications are not routed-workload measurements.
Expert blobs use page-aligned layouts and buffers. On Linux, MER supports O_DIRECT-oriented positional reads; an optional io_uring backend uses registered fixed buffers and batched submission. Portable development paths can fall back when direct I/O is unavailable.
Routed inference is shaped by random expert access, queue contention, filesystem behavior, storage virtualization, and concurrent prefetch. A drive’s sequential specification does not predict the latency of this access pattern by itself.
Residency integration
Predictive movement belongs in the memory architecture.
MER already contains predictor, speculator, and admission-control machinery for expert prefetch. As GPU residency is reconnected, that machinery will be adapted so predicted experts can move through NVMe → RAM → VRAM before demand arrives. Admission must respect prediction confidence, memory headroom, and I/O contention, with foreground inference taking priority. The final GPU-tier path is not production qualified today.
Validated GPU foundation
Current hardware-qualified capabilities.
- Request-local GPU hidden and residual state
- Authoritative existing WGPU device reuse
- Persistent model-scoped F32 and native Q8_0 weights
- GPU embedding lookup and GEMV with binding-limit chunking
- RMSNorm, grouped per-head Q/K normalization, and residual state transitions
- Q/K/V projections, RoPE, and request-local F32 GPU KV append
- No intermediate maps/readbacks in the qualified production primitives
Capability status
Correctness, diagnostics, and integration are not interchangeable.
| Status | Capability | Meaning |
|---|---|---|
| Validated output qualification | Reference execution compared with GPU-routed expert execution | Deterministic generated output agreed across the fixed corpus; no TPS claim. |
| Measured diagnostic | Controlled earlier warm-path batching on NVIDIA L4 | Measured decode improvement and fewer host synchronizations; not current GPU-native throughput. |
| Validated GPU foundation | State, weights, dense operations, norms, Q/K/V, RoPE, F32 KV append | NVIDIA-L4 validated without intermediate production host readbacks. |
| Currently being integrated | Causal attention, softmax, value aggregation, output projection | Required before attention is complete in the GPU-owned loop. |
| Next architecture stage | GPU routing and Q4 experts; residency, budgeting, and predictive movement | Required to reconnect MER’s virtualized expert memory to GPU execution. |
| Engineering target | Complete GPU-owned generation at ≥10 real generated tokens/s | Not achieved; oversized-model proof must include measurable tier movement. |