VRAM hit
If the selected expert is resident in the managed VRAM arena, the target warm path uses it directly while hidden and residual state remain device-resident.
Execution path
How MER runs beyond VRAM.
A sparse Mixture-of-Experts model activates only part of its expert pool for each token. MER uses that selectivity to keep the active experts in VRAM while RAM and NVMe provide progressively larger capacity behind them.
From token to selected expert
The model’s learned router decides which experts are needed. MER manages where their weights live, how misses move toward the GPU, and which likely experts should start moving before demand arrives.
- 01Route
The learned router selects the experts needed for this token
- 02Hit or miss
Check the managed VRAM resident set
- 03Move the miss
Bring missing weights from NVMe through RAM toward VRAM
- 04Execute
Run the selected resident expert on the GPU
- 05Retain and predict
Keep useful experts hot and anticipate likely demand
Validated today
The GPU-native foundation now reaches attention preparation.
Hardware-validated components now cover request-local GPU hidden and residual state, persistent dense weights, embedding lookup, GPU matrix-vector operations, RMSNorm, residual transitions, attention preparation through Q/K/V projections and RoPE, and per-request GPU context storage.
Causal attention scores, softmax, weighted value aggregation, attention output, GPU routing, routed Q4 expert execution and combine, expert-residency reconnection, and the complete GPU-owned token loop remain in development.
Hit or miss
A miss changes residency underneath compute.
Expert miss
MER sources the missing expert from RAM or loads it from NVMe through RAM toward VRAM. Storage latency is real; the final tier-aware overlap path is not yet production qualified.
Maximum safe occupancy
Keep useful expert weights in the VRAM that remains.
MER first reserves deterministic headroom for persistent non-expert weights, context and active-request memory, temporary GPU work, in-flight uploads or evictions, plus allocator and driver safety margin.
Nearly all of the remaining safe budget becomes managed expert residency. More VRAM should therefore mean a larger hot set and fewer misses, not a different model-loading compatibility boundary. MER does not claim literal 100% allocation.
Residency policy
Retention and prediction follow demand.
MER tracks layer-qualified expert identity and uses the safe budget available on the selected device. There is no universal cache-slot count that applies to 24 GB, 48 GB, and 80 GB GPUs.
MER is designed to anticipate likely expert demand so useful weights can begin moving from NVMe through RAM toward VRAM before they are needed. This GPU-tier path is being integrated, and foreground inference keeps priority when confidence, memory, or I/O budgets are tight.
Performance
Five factors shape the operating point.
Safe VRAM budget
How much of the expert namespace can remain hot.
Routing locality
How often learned demand reuses the resident set.
Miss service
NVMe latency, RAM staging, upload, and queue contention.
GPU kernels
Dense, attention, routing, expert, and combine efficiency.
Request shape
Context length, active requests, concurrency, and temporary GPU work.
Correctness foundation
Every faster path must preserve generated output.
Historical all-CPU execution remains an informational control. The authoritative qualification compared a CPU reference that reproduces the production expert boundary with GPU-routed expert execution; deterministic generated output agreed across the fixed corpus. That correctness standard remains in place as more of the loop moves to the GPU.
What MER is not
- Not a model or a model API wrapper.
- Not a completed GPU-native autoregressive loop today.
- Not proof that SSD latency is hidden or free.
- Not a claim that the ≥10 generated tokens/s target has been achieved.