repository · locator & rights
38a5b42d9a3e82e0a586bcd1caed121f36c87a73
Features; --model; /completion; /v1/chat/completions; cache_prompt; speculative decoding
Link and original paraphrase only; upstream rights retained.
Serving / FEATURE
Concurrent inference
Process multiple requests concurrently within documented backend and memory limits; accepting queued requests alone is insufficient.
3connected products
Standing condition: these are documentation claims at the source capture date, with outcome “claimed”; no runtime execution is recorded. Dossier applicability and limitations apply throughout. Expand a row for its statement, specific scope and contradictions. Numbered citations resolve to the source locator and capture date below. Missing evidence means Unknown, not No. Sources are unpinned unless marked “Pinned”; this marks source stability, not stronger support or runtime verification. Reuse counts refer to claims in the full dossier; repeated citations do not represent independent sources.
Yes Documented within scope Partial Limited form of the feature No Explicit evidence of absence Unknown No evidence recorded. These are support assessments, not quality scores or runtime verification.
3 products have Yes or Partial documentation. Review freshness and attestation counts describe the dossier, not runtime quality. Gaps below are product-wide, not necessarily specific to this feature.
Concurrent inferenceYes
Explore Concurrent inference →
Supports parallel decoding and continuous batching.
Scope: Configured server slots and available resources; no throughput measured.
2 open gaps · applicability & verification plan
What we do not know yet about llama.cpp
- Which pinned model, quantization, context length and runtime versions fit the selected Mac?
- What are measured latency, throughput, peak memory and swap under one and four concurrent requests?
Applicability
Repository revision 38a5b42d9a3e82e0a586bcd1caed121f36c87a73; living documentation separately dated.
Self-managed local build and server; pin compiled backends and model files.
Excludes: Runtime verification and performance rankings; Model weights license or quality evaluation; Cloud inference, remote hosts and external tools; Unlisted model architectures, builds and hardware
Recheck when
- Runtime, bundled engine, backend, model format or license changes.
- Documentation changes affecting the cited configuration.
Needs recheck · due
Verification plan
- Choose hardware and obtain explicit approval for installations, downloads and execution resources.
- Pin runtime and engine versions, model artifact hashes, tokenizer, template, quantization and inference settings.
- Run the local-model-mac investigation with synthetic prompts and capture raw measurements; do not record documentation as execution.
Provenance & review
Testedknowhow model runtime research
Commissioned by a Nimbalyst maintainer. Documentation synthesis; no independent review or product execution.
Revision 2026-09-15.1 · unreviewed · checked
0 attestations · Content: Unlicensed prototype; redistribution decision pending
Sources · 1
Unknown — no evidence recorded
3 open gaps · applicability & verification plan
What we do not know yet about MLX-LM
- Which pinned model, quantization, context length and runtime versions fit the selected Mac?
- What are measured latency, throughput, peak memory and swap under one and four concurrent requests?
- Concurrency and JSON schema output are Unknown in this research package; no cited absence is asserted.
Applicability
Repository revision 872ae88d1fac77350db23c8c04fe8dd372a9e3e8; living documentation separately dated.
Local Apple Silicon execution with MLX; model conversion and server configuration remain operator-owned.
Excludes: Runtime verification and performance rankings; Model weights license or quality evaluation; Cloud inference, remote hosts and external tools; Unlisted model architectures, builds and hardware
Recheck when
- Runtime, bundled engine, backend, model format or license changes.
- Documentation changes affecting the cited configuration.
Needs recheck · due
Verification plan
- Choose hardware and obtain explicit approval for installations, downloads and execution resources.
- Pin runtime and engine versions, model artifact hashes, tokenizer, template, quantization and inference settings.
- Run the local-model-mac investigation with synthetic prompts and capture raw measurements; do not record documentation as execution.
Provenance & review
Testedknowhow model runtime research
Commissioned by a Nimbalyst maintainer. Documentation synthesis; no independent review or product execution.
Revision 2026-09-15.1 · unreviewed · checked
0 attestations · Content: Unlicensed prototype; redistribution decision pending
Concurrent inferenceYes
Explore Concurrent inference →
Can process multiple requests to the same model in parallel.
Scope: Requires sufficient memory; parallel request count multiplies context memory and excess work queues.
2 open gaps · applicability & verification plan
What we do not know yet about Ollama (local runtime)
- Which pinned model, quantization, context length and runtime versions fit the selected Mac?
- What are measured latency, throughput, peak memory and swap under one and four concurrent requests?
Applicability
Repository revision a43fad18b088095de20fbd7a8f0de50824cf5d27; living documentation separately dated.
Local Ollama server with OLLAMA_NO_CLOUD=1 and preloaded local models; excludes Ollama Cloud and web search.
Excludes: Runtime verification and performance rankings; Model weights license or quality evaluation; Cloud inference, remote hosts and external tools; Unlisted model architectures, builds and hardware
Recheck when
- Runtime, bundled engine, backend, model format or license changes.
- Documentation changes affecting the cited configuration.
Needs recheck · due
Verification plan
- Choose hardware and obtain explicit approval for installations, downloads and execution resources.
- Pin runtime and engine versions, model artifact hashes, tokenizer, template, quantization and inference settings.
- Run the local-model-mac investigation with synthetic prompts and capture raw measurements; do not record documentation as execution.
Provenance & review
Testedknowhow model runtime research
Commissioned by a Nimbalyst maintainer. Documentation synthesis; no independent review or product execution.
Revision 2026-09-15.1 · unreviewed · checked
0 attestations · Content: Unlicensed prototype; redistribution decision pending
Sources · 1
vendor-documentation · locator & rights
Local data; disabling cloud; server binding; concurrent requests
Link and original paraphrase only; upstream rights retained.
Concurrent inferencePartial
Explore Concurrent inference →
Documents parallel requests for the llama.cpp engine; MLX support is listed as forthcoming.
Scope: Partial across the target engines: llama.cpp continuous batching only at capture date; configure Max Concurrent Predictions.
3 open gaps · applicability & verification plan
What we do not know yet about LM Studio (local runtime)
- Which pinned model, quantization, context length and runtime versions fit the selected Mac?
- What are measured latency, throughput, peak memory and swap under one and four concurrent requests?
- The schema has no proprietary source-access value: Not established does not override the explicitly recorded proprietary app terms.
Applicability
Living documentation accessed 2026-09-15; app and installed engine versions are not pinned.
Local LM Studio app and server; pin the separately installed llama.cpp or MLX engine. Excludes remote hosts, Bionic workflows and external tools.
Excludes: Runtime verification and performance rankings; Model weights license or quality evaluation; Cloud inference, remote hosts and external tools; Unlisted model architectures, builds and hardware
Recheck when
- Runtime, bundled engine, backend, model format or license changes.
- Documentation changes affecting the cited configuration.
Needs recheck · due
Verification plan
- Choose hardware and obtain explicit approval for installations, downloads and execution resources.
- Pin runtime and engine versions, model artifact hashes, tokenizer, template, quantization and inference settings.
- Run the local-model-mac investigation with synthetic prompts and capture raw measurements; do not record documentation as execution.
Provenance & review
Testedknowhow model runtime research
Commissioned by a Nimbalyst maintainer. Documentation synthesis; no independent review or product execution.
Revision 2026-09-15.1 · unreviewed · checked
0 attestations · Content: Unlicensed prototype; redistribution decision pending
Sources · 1
vendor-documentation · locator & rights
Max Concurrent Predictions; engine support
Link and original paraphrase only; upstream rights retained.