LLM context, KV cache and memory bandwidth: why RAM is only one constraint

Capacity and bandwidth answer different questions

A model can fit in memory and still be frustrating to use. On our M3 Ultra Mac Studio with 256GB RAM, memory capacity alone did not make the models we tried useful for OpenClaw. Keep two questions separate: can this configuration hold the workload, and can it complete the task at a useful speed?

For Apple silicon comparisons, keep nominal specification bandwidth beside the exact chip. Apple lists M2 Max at 400 GB/s and M3 Ultra at 819 GB/s. The current M5 Ultra specification lists 1.2 TB/s, equivalent to 1,200 GB/s in decimal units. These are hardware specifications, not this site’s measured sustained bandwidth or inference results.

Why generation can depend on bandwidth

During autoregressive decoding, each new token depends on the state built by previous tokens. For some dense-model single-stream workloads, moving weights and cache data to the processor can limit performance. More usable bandwidth can help in that regime, but it does not remove every other bottleneck.

Prompt processing has a different computation pattern and may respond differently to GPU compute and kernel optimization. Batching can reuse work across requests and improve aggregate throughput while worsening an individual user’s first-token latency. Report the phase and concurrency instead of publishing one unqualified “AI speed” number.

KV-cache demand changes with the workload

A standard full-attention decoder cache stores keys and values across layers, context positions and concurrent requests. A useful planning expression multiplies two caches by layers, KV heads, head dimension, tokens, concurrent requests and bytes per element. Actual runtimes may use different cache layouts, compression, sliding windows or hybrid architectures, so architecture-specific settings matter.

Ollama documents context configuration and its relationship to memory demand. Use the model’s real settings and observe the runtime. Reducing context can reduce memory pressure, but it can also remove evidence the task needs. The objective is a correct completion within a practical resource budget, not the smallest possible context at any cost.

Do not turn a file-size ratio into a MoE prediction

A rough dense-model bandwidth argument may divide bytes moved per token into usable bytes per second. The denominator must represent actual traffic. Dividing nominal bandwidth by the entire file size of a mixture-of-experts model is not a valid speed prediction when only selected experts are active for a token.

Our M3 Ultra record uses Qwen3.6-35B-A3B-8bit. Its displayed model size and measured generation rate should be retained as separate observations. We do not infer a cross-machine winner or an expected M5 Ultra token rate from the bandwidth ratio alone.

Design the comparison you need

Keep the model artifact, quantization, input length, generated length, acceleration settings and cache policy constant. Measure first-token delay, generation throughput and peak memory separately. Add repeated trials and retain failures. If a smaller-memory machine requires a different model, present that as a different workload rather than an identical hardware race.

Use the memory planner, inspect the M3 Ultra records, and compare repeats with the benchmark analyzer. Confirm the exact chip before assigning a bandwidth figure to an M1-generation MacBook Pro.

Implementation map

Sources and limits

Apple M2 Max, M3 Ultra specifications, M5 Ultra specifications and Ollama context. Documentation checked October 14, 2026. Recommendations and worked examples are editorial methods, not claims of a completed client engagement.

Owner evidence: M2 Max 64GB