GPU / VRAM
- Backend and compute support
- Usable VRAM after safety reserves
- Persistent dense state and weights
- Context, active-request memory, and temporary GPU work
- Managed expert hot-set capacity
Hardware
VRAM is the hot tier, not the capacity ceiling.
MER plans around the whole path: GPU compute and usable VRAM first, then system RAM, local NVMe or SSD, CPU control, and memory for context, active requests, and temporary GPU work.
The system is one connected path.
GPU / VRAMHot transformer compute, request state, and managed expert residencySystem RAMExpert staging, cache, miss buffers, and host headroomNVMe / SSDCapacity backing the larger routed-expert namespaceCPU controlStartup, storage path, miss orchestration, and telemetryRequest shapeContext length, active requests, concurrency, and temporary GPU work
What matters
Plan around the full hierarchy.
System RAM
- Expert staging and cache
- Miss buffering and upload headroom
- Startup and model state where applicable
- Operating-system headroom
Local NVMe / SSD
- Expert backing capacity
- Random-read latency
- Queue depth and contention
- Sustained routed-workload behavior
- Filesystem and direct-I/O support
CPU / host control
- Miss and storage orchestration
- Startup and conversion work
- Telemetry and status
- Verified historical CPU execution
Request shape
- Context length
- Memory for active requests
- Concurrency
- Temporary GPU work and safety headroom
Current qualification device
NVIDIA L4 validation, not a universal minimum.
- Device
- NVIDIA L4 with 23,034 MiB visible VRAM
- Qualified scope
- GPU state, F32/Q8_0 weights, embedding, GEMV, RMSNorm, residual transitions, grouped Q/K normalization, Q/K/V projections, RoPE, and F32 KV append
- Readback boundary
- No intermediate maps/readbacks in production primitives; hardware tests map only a tiny aggregate validation status
- Not established
- Minimum supported GPU, complete production GPU-native token loop, or commercial throughput
One hardware qualification device is evidence for those primitives, not a minimum GPU specification.
VRAM planning
Reserve, then fill the safe remainder.
- 1 / Persistent
- Dense and other non-expert model weights
- 2 / Requests
- Context, active-request state, and concurrency
- 3 / Workspaces
- Kernels and temporary GPU buffers
- 4 / Residency operations
- In-flight uploads and evictions
- 5 / Safety
- Allocator and driver margin
- 6 / Expert budget
- Nearly all remaining safe VRAM for managed expert residency
Planning framework
Capacity follows the model and workload.
GPU and request pressure depends on
- Persistent dense/non-expert weights
- Target expert hot-set size
- Quantization
- Context length and memory for active requests
- Temporary GPU work and in-flight expert movement
- Concurrency
RAM and storage pressure depends on
- Expert payload size
- Miss frequency
- Routing locality
- Staging and upload behavior
- Random-read latency and queue contention
- Sustained routed workload, not advertised sequential bandwidth