EXPERIMENT / 01
Is 16GB Enough for Serious Local LLMs on a Mac?
Testing memory pressure, useful context and responsiveness on a 16GB M1 MacBook Pro.
Explore the experiment planLOCAL AI / THE HARDWARE LAB
Running LLMs on real hardware.
How much hardware do you actually need to run useful AI models locally? I test models, machines, inference engines, memory limits, and practical workloads to find out.
HOW THE PIECES FIT
A model on your machine. A workload worth testing.
The model runs locally, but memory, context and inference overhead still matter. A practical evaluation connects the hardware to the task you need it to perform.
Read the engineering method ↗LOCAL INFERENCEIllustrated, not live
The hardware test bench
APPLE SILICON / 01
M1 MacBook Pro
16GB unified memory
Measurements pendingAPPLE SILICON / 02
M2 Max MacBook Pro
64GB unified memory
Four owner-provided benchmark recordsTEST BENCH / 03
M3 Ultra Mac Studio
256GB unified memory · owner confirmed
Four owner-provided benchmark recordsMore memory didn’t fix the model. Finding the right model did.
Even our M3 Ultra Mac Studio with 256GB unified memory could run models that failed to do the work we needed. Qwen3.6 35B MoE became our sweet spot for agentic AI, coding and basic operations.
- Did not meet our needs
Smaller models tried
Frequent hallucinations and insufficient reasoning for our OpenClaw workflows.
- Did not meet our needs
Other Qwen variants
The variants we tried did not sustain the agent workflows we needed.
- Did not meet our needs
“Mistral 70B” — owner’s label
Still unsuccessful in our setup, including on the 256GB M3 Ultra. Exact artifact unconfirmed.
- Our operational sweet spot
Qwen3.6 35B MoE
Worked for our agentic AI, coding and basic operations; our preferred model among those tried.
REPORTED OPERATIONAL OUTCOME16 OpenClaw agents
Running in parallel for months of basic operations, according to the owner’s experience with Qwen3.6 35B MoE.
Our recommendation: test the workflow before buying more RAM.
For an OpenClaw setup like ours, start by evaluating this 35B MoE model against the tasks your agents actually perform. Check reasoning, hallucinations, tool use and completed work, then measure latency and memory. A model fitting into RAM does not establish that it can run the workflow successfully.
Evidence scope and how to read this verdict
This is Fazal Shah’s retrospective account of models tried in his own setup, not a universal claim that other models fail. “Failed” means the model did not meet his operational needs; it does not necessarily mean it failed to load. Smaller models reportedly hallucinated and could not support the reasoning and work required by his OpenClaw agents.
Exact alternative model repositories, quantizations, dates, prompts, failure counts and intervention rates have not been supplied. “Mistral 70B” preserves the owner’s wording, not a verified model identifier. The 16-agent account does not establish 16 simultaneous inference requests, uninterrupted uptime, unattended operation or a measured success rate. Its exact machine allocation and deployment artifact remain unconfirmed.
The screenshot benchmarks test 1×, 2× and 4× batching with 128 output tokens, not the 16-agent operational deployment. They show Qwen3.6-35B-A3B-8bit on the M3 Ultra and a differently named, truncated artifact on the M2 Max. They cannot independently verify months of agent operation or identify the exact deployment artifact.
Next evidence to collect: a redacted OpenClaw agent roster, dated run history, exact model artifact and configuration, representative completed tasks, failed attempts and human interventions. Keep task quality and throughput as separate measurements.
Measure usefulness, not just throughput.
Record first-token latency, tokens per second, load time, peak memory, quantization, context length, coding task success, agentic task success and system responsiveness.
Owner-provided results: M2 Max 64GB and M3 Ultra Mac Studio 256GB. Eight screenshot-transcribed records. Different model artifacts and runtime versions; these are observations, not a controlled hardware ranking.
Generation speed
Time to first token
| Input | Batch | TTFT (ms) | Prefill tok/s | Generation tok/s | E2E (s) | Peak GB reported |
|---|---|---|---|---|---|---|
| 1024 | 1 | 1076.3 | 951.4 | 81 | 2.662 | 19.89 |
| 4096 | 1 | 3956.5 | 1035.3 | 78.4 | 5.595 | 21.79 |
Owner-supplied screenshots; not independently reproduced. M2 Max RAM is owner-confirmed at 64GB; M3 Ultra RAM is owner-confirmed at 256GB; both machines’ CPU/GPU cores and macOS remain unconfirmed. Full model repositories/revisions, M2 quantization, acceleration/cache settings, measurement dates and repeat-run spread are unknown. The model artifacts and oMLX versions differ: this is not a controlled hardware ranking. Peak memory uses the application-reported GB label. Batch generation TPS is aggregate; batch TTFT is an average. Dashboard averages are separate from benchmark records.
APPLE SPECIFICATIONS / NOT MEASURED BANDWIDTH
Capacity holds the model. Bandwidth feeds the GPU.
Apple-rated unified memory bandwidth in decimal GB/s. 1.2 TB/s = 1,200 GB/s in decimal units. Reference chips are not additional tested machines.
What changes token speed?
Single-stream dense-model decoding can be limited by how fast weights and cache data reach the GPU. More bandwidth can help when memory traffic is the bottleneck; extra RAM alone does not guarantee faster tokens.
Prompt processing uses a different computation pattern. Batching, GPU compute, context, kernel efficiency and quantization can shift the bottleneck. MoE models do not read every expert weight for every token, so dividing bandwidth by the entire model file is not a valid speed prediction for this Qwen result.
Read the bandwidth and context guide →| Machine | Known RAM | Nominal bandwidth | Measurement status |
|---|---|---|---|
| M1-generation MacBook Pro | 16GB, owner brief | Unconfirmed — exact chip required | No recorded inference runs |
| M2 Max MacBook Pro | 64GB, owner confirmed | 400 GB/s | Four owner-provided oMLX result rows |
| M3 Ultra Mac Studio | 256GB, owner confirmed | 819 GB/s | Four owner-provided oMLX result rows |
| M5 Ultra reference | Not part of our measured bench | 1,200 GB/s (1.2 TB/s) | Specification only — no inference runs supplied |
Apple sources: M5 Ultra specifications, M3 Ultra specifications, M2 Max, M1 Pro / Max. The name “MacBook Pro” does not establish whether the chip is base M1, M1 Pro or M1 Max.
M2 Max benchmark and all-time usage
OWNER EVIDENCE / M2 MAX · 64GB
Measured workload, not a hardware ranking.
Owner-supplied oMLX v0.7.0 screenshots show Code (Python), 128 output tokens, and a model label beginning Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Opti… . The full artifact and quantization are not visible.
| Input / batch | Generation tok/s | TTFT ms | E2E seconds | Peak GB reported |
|---|---|---|---|---|
| 1024 / 1× | 81 | 1076.3 | 2.662 | 19.89 |
| 4096 / 1× | 78.4 | 3956.5 | 5.595 | 21.79 |
| 1024 / 2× | 133.2 | 2085.7 | 4.47 | Not shown |
| 1024 / 4× | 213.1 | 3871.2 | 8.526 | Not shown |
At 2× and 4× batching, generation is aggregate throughput, not the rate each user receives. The screenshot reports 1.64× and 2.63× speedup.
All-time serving: a separate view
Prefill tokens (rounded)72.9M
Cached tokens (rounded)69.8M
Displayed cache efficiency95.7%
The All-Time / All Models dashboard reports 215.6 tok/s prompt processing excluding cached tokens and 33.2 tok/s generation. Its date range, model mix and request count are unknown. These are mixed-use averages, not another benchmark or proof of a latency/cost saving. The empty Today history does not invalidate the all-time counters.
The M2 Max specification is 400 GB/s memory bandwidth; the M3 Ultra is 819 GB/s. The supplied tests use different artifacts and oMLX versions (v0.7.0 versus v0.5.4), so their token speeds cannot isolate bandwidth or establish which chip is faster. Inspect both datasets, bandwidth sources and original screenshots →
oMLX v0.7.0: two single-request tests and 2×/4× aggregate batching. One capture per condition; repeat-run variation unknown.
Code (Python), pp1024/pp4096, tg128 and 2×/4× batches. Model label is truncated; quantization and collapsed advanced settings are unconfirmed. ANE-aligned prompts is unchecked.
All-Time / All Models: rounded token counters and mixed-workload averages. The Today usage-history section is empty and is a different scope.
M3 Ultra source evidence
Mac Studio with 256GB unified memory, confirmed by the owner. oMLX v0.5.4, Qwen3.6-35B-A3B-8bit, Code (Python), pp1024/pp4096 and tg128. Inspect generation, latency and batching above.
The benchmark output used for the four result records. Batch throughput is aggregate; single-request results use 128 generated tokens.
Qwen3.6-35B-A3B-8bit; Code (Python); pp1024, pp4096 and 2×/4× batching. Advanced settings were collapsed in the supplied capture.
oMLX v0.5.4 session averages across the selected dashboard scope. These averages are separate from the benchmark records.
Cache state shown in the supplied dashboard capture. This does not establish whether benchmark requests reused cached prompts.
Inference tools on the research list
MLX, oMLX, Ollama and other local inference workflows. Each comparison will record exact software versions and settings.
Upcoming experiments
EXPERIMENT / 02
What's the Cheapest Mac That Can Run an LLM?
A practical test plan for finding useful local inference without overspending.
Explore the experiment planEXPERIMENT / 03
Can a Local LLM Replace Claude or Codex?
A planned comparison of local models and paid coding assistants on the same tasks.
Explore the experiment planA first memory estimate
This planning tool estimates weights, overhead and KV-cache memory from explicit assumptions. It is not a hardware benchmark or a compatibility guarantee.
Planning model, not a compatibility test. Defaults describe an illustrative standard decoder architecture. Enter your model's actual KV dimensions. Hybrid, compressed-cache and sliding-window architectures can differ.
Save the inputs to a file and reopen them later. Nothing is stored automatically.
Memory formulas and limits
Raw weights = total parameters × bits / 8, converted to GiB. KV cache = 2 × attention layers × KV heads × head dimension × context tokens × parallel requests × cache bytes. The factor 2 represents keys and values. Values are binary GiB; a vendor's decimal GB display may differ.
The allowance is a user assumption, not a universal overhead percentage. This does not estimate speed or guarantee that a runtime supports the chosen settings. A MoE model still needs a budget for resident weights beyond the active parameter count.
Build a private AI setup around your workload.
Hardware choices should follow the actual task, privacy requirements and measured performance.
Discuss a Private AI Deployment ↗BUILD IN PUBLIC
Tools, skills and field notes you can inspect.
I publish Mac utilities, agent infrastructure and reusable workflows on GitHub. Start with the code and documentation, then adapt the parts that fit your setup.
Auto Pause Mac Apps
A macOS menu bar utility for pausing background apps and managing memory headroom before local model work.
Explore on GitHub ↗Turnstile
A concurrency-capping FIFO proxy in front of a local model endpoint, designed to keep multi-agent requests from overwhelming one GPU.
Explore on GitHub ↗Claude skills
Reusable skills covering living documentation, self-healing MCP services and coverage-verified RAG indexing.
Explore on GitHub ↗FROM THE ENGINEERING NOTES
Practical guides from the Workshop
Pick a problem you are working on. Each guide includes a method, sources and a tool or example to try.