How to design a reproducible local LLM test

Define useful before fast

Pick a task before picking a winner. A coding comparison needs a test suite or a reviewed patch; an extraction task needs expected fields. Run the same workload on each configuration and keep the failures. Tokens per second tell you how output arrived, not whether it solved the problem.

Record the complete setup

Record the chip, memory, operating system, model version, inference engine version, quantization, context settings and prompt. Keep other applications and power settings consistent across runs.

Separate loading and generation

Report model-loading time, first-token latency and generation throughput separately. Distinguish cold and warm runs. Note whether compilation or cache reuse is included.

Measure memory under the real workload

Context length and simultaneous requests change memory use. Record peak memory and system responsiveness, not just the model file’s size. Avoid translating a model fitting in memory into a claim that it is practical.

Repeat and retain failures

Repeat each configuration, keep raw measurements and document failed or interrupted runs. Use consistent prompts and output limits. If a configuration cannot complete the task, report that outcome.

Publish limitations

Results describe a specific machine, model and workload. They do not prove that every model in the same size range will behave similarly. The M2 evidence below is an observed example; it is separate from this repeatable test protocol.

Download the blank benchmark CSV or visit the Local AI Lab.

Further implementation guides

Read the Mac setup guide and MLX versus Ollama comparison. These are documentation-led review editions; speed measurements remain pending.

Compare latency and throughput separately

A short interaction can feel slow even when its generation rate is high if loading or prompt processing delays the first token. Report time to first token, generation throughput and end-to-end latency as separate values. State whether the measurement begins before model loading, after warmup or at the request boundary.

For concurrent serving, distinguish aggregate output from per-request experience. Four requests producing a large combined token rate do not give each person that combined rate. Record average or per-request first-token delay and the output length used in the test. Check the benchmark implementation before assuming another tool uses identical metric definitions.

Keep a hardware specification ledger

Store chip name, CPU/GPU cores, memory capacity, nominal bandwidth and operating-system version. Memory capacity affects what can remain resident; bandwidth can affect decode performance when moving model data is the bottleneck. Neither specification is a measured token rate. Use the bandwidth guide for the distinction and keep hardware reference entries separate from machines actually tested.

The owner’s M3 Ultra screenshots now provide four result rows, with configuration and source images on the hardware-lab page. Exact RAM, advanced settings, model revision and repeat-run spread are still missing. Those limitations remain attached to the observations rather than being replaced with guessed values.

Prepare a repeatable run sheet

Create one configuration label for each distinct combination of model, quantization, context, cache policy and acceleration settings. Save the individual run number and timestamp. Record failed attempts in a separate outcome field rather than entering a zero as if it were a successful but slow generation.

Start with at least three identical runs for an initial inspection. Use medians to summarize the observations and retain the raw rows. With few trials, a percentile near the tail is often just the maximum and should not be described as a reliable service-level estimate. The benchmark analyzer exposes its calculation method so readers can interpret the summary.

Source context

Hugging Face’s memory-estimation documentation distinguishes loading a model from the additional demands of inference. Artificial Analysis publishes a methodology that separates latency and throughput across serving conditions. These references inform the measurement categories; no external leaderboard result is claimed as our own. Memory estimation; performance methodology.

Owner evidence: M2 Max 64GB