Same-artifact GGUF comparison
Compare the three GGUF targets with the same file hash and settings; wrapper and bundled-engine differences remain variables.
AI infrastructure · model runtimes / RESEARCH IN PROGRESS
Which local model runtime fits a 32GB or 64GB Mac?
Compare a pinned open model across local runtimes, then measure what changes with four simultaneous requests. The procedure is untested; no speed, memory-fit or quality winner has been established.
A scoped answer, with choices.
- Use separate 32GB and 64GB Apple Silicon machines; record chip, GPU cores, memory bandwidth, macOS and power mode. RAM capacity alone cannot explain cross-machine differences.
- Use the same source-model revision, tokenizer, chat template and synthetic prompts. Hash every downloaded artifact.
- Compare the identical GGUF artifact across llama.cpp, Ollama and LM Studio; evaluate MLX conversions separately. Matching bit counts do not make quantizations equivalent.
- Keep cloud inference and external tools disabled; preload permitted model files before the offline measurement phase.
Separate MLX conversion comparison
Compare MLX-LM and LM Studio MLX using the same MLX artifact. Assess output quality before comparing conversion tracks.
Claims you can inspect.
Loads supported models from GGUF files.
Supported model and configured local backend; documentation only, no execution recorded.
Source: llama.cpp overview
Description; supported backends; quick start · Checked 2026-09-15
38a5b42d9a3e82e0a586bcd1caed121f36c87a73
Inspect original source ↗Provides conversion to quantized model weights.
README conversion example uses 4-bit weights; not equivalent to GGUF 4-bit artifacts.
Source: MLX-LM documentation
Quick Start; Python API; quantization; long prompts and generations · Checked 2026-09-15
872ae88d1fac77350db23c8c04fe8dd372a9e3e8
Inspect original source ↗Can process multiple requests to the same model in parallel.
Requires sufficient memory; parallel request count multiplies context memory and excess work queues.
Source: Ollama FAQ
Local data; disabling cloud; server binding; concurrent requests · Checked 2026-09-15
Living documentation; runtime not pinned
Inspect original source ↗Documents parallel requests for the llama.cpp engine; MLX support is listed as forthcoming.
Partial across the target engines: llama.cpp continuous batching only at capture date; configure Max Concurrent Predictions.
Source: LM Studio parallel requests
Max Concurrent Predictions; engine support · Checked 2026-09-15
Living documentation; runtime not pinned
Inspect original source ↗Measure a pinned local model under one and four requests
This is a procedure draft, not a working implementation. Package versions and executable artifacts are not yet pinned.
Agree on execution boundaries
Obtain explicit authorization for the selected machines, runtime installations, model license, downloads and resource budget before executing this recipe.
Expected: A named test environment and approved resource limits.
Pin the candidates
Record chip, GPU cores, RAM, macOS, power mode, runtime and bundled-engine versions. Hash source-model, tokenizer, template and derived weight artifacts. Separate GGUF and MLX tracks.
Expected: A manifest that distinguishes identical artifacts from conversions.
Fix workload and settings
Use synthetic prompts of fixed token lengths, fixed output limits, seed and sampling settings, context size, GPU offload and concurrency. Disable speculative decoding for the baseline. Record cache and batching settings.
Expected: A reproducible baseline with one and four simultaneous requests; unsupported configurations stay explicitly unmeasured.
Preload and isolate
Acquire only approved models and runtimes, disable cloud and remote tools, then disconnect networking for the local measurement phase. Record startup failures and missing dependencies.
Expected: A documented local-only run boundary; no prompt leaves through an external tool.
Measure repeated runs
Separate cold model-load runs from warm inference. Capture startup time, time to first token, prefill and decode rates, per-request latency, peak memory and swap. Repeat the fixed workload, retaining raw samples and errors.
Expected: Per-machine and per-artifact measurements, with sample counts and p50/p95 latency where enough samples exist.
Check correctness and interpret
Check completion length, structured-output validity where claimed, and task correctness on the fixed prompt set. Compare within a machine and artifact track first. Report failures and avoid attributing all cross-machine differences to RAM.
Expected: A bounded report with quality checks and no unsupported universal winner.
What would count as evidence?
Not attempted
Every compared same-artifact run records an identical model hash, tokenizer and template.
Not attempted
Model inference completes with networking disconnected and external tools disabled, or the run records why it failed.
Not attempted
One-request and four-request results include queueing, errors, memory and per-request latency; backend restrictions remain visible.
Not attempted
Output validation and fixed task checks accompany performance metrics.
Not attempted
Raw samples, versions and settings are retained; no executed finding is added to schema v1 documentation claims.
Resolve before calling this a recipe that works
- User-selected machines, model, installation boundaries and download budget are not established.
- No runtime, latency, quality, concurrency or memory measurement has been collected.
- App and bundled-engine versions must be pinned separately; missing capabilities remain Unknown.