# llama.cpp

C/C++ inference library and server with GGUF models and selectable hardware backends.

Revision: 2026-09-15.1 · Review: unreviewed

Commissioned by a Nimbalyst maintainer. Documentation synthesis; no independent review or product execution.

## Applicability

Repository revision 38a5b42d9a3e82e0a586bcd1caed121f36c87a73; living documentation separately dated.. Self-managed local build and server; pin compiled backends and model files..

## Claims

- **yes · documented**: Supports inference on a local machine. [llama.cpp overview](https://github.com/ggml-org/llama.cpp/blob/38a5b42d9a3e82e0a586bcd1caed121f36c87a73/README.md) Scope: Supported model and configured local backend; documentation only, no execution recorded.
- **yes · documented**: Documents Metal acceleration on Apple Silicon. [llama.cpp overview](https://github.com/ggml-org/llama.cpp/blob/38a5b42d9a3e82e0a586bcd1caed121f36c87a73/README.md) Scope: Supported model and configured local backend; documentation only, no execution recorded.
- **yes · documented**: Documents CPU inference with architecture-specific optimizations. [llama.cpp overview](https://github.com/ggml-org/llama.cpp/blob/38a5b42d9a3e82e0a586bcd1caed121f36c87a73/README.md) Scope: Supported model and configured local backend; documentation only, no execution recorded.
- **yes · documented**: Provides a CUDA backend for supported NVIDIA GPUs. [llama.cpp overview](https://github.com/ggml-org/llama.cpp/blob/38a5b42d9a3e82e0a586bcd1caed121f36c87a73/README.md) Scope: Supported model and configured local backend; documentation only, no execution recorded.
- **yes · documented**: Loads supported models from GGUF files. [llama.cpp overview](https://github.com/ggml-org/llama.cpp/blob/38a5b42d9a3e82e0a586bcd1caed121f36c87a73/README.md) Scope: Supported model and configured local backend; documentation only, no execution recorded.
- **yes · documented**: Supports multiple integer weight quantization formats. [llama.cpp overview](https://github.com/ggml-org/llama.cpp/blob/38a5b42d9a3e82e0a586bcd1caed121f36c87a73/README.md) Scope: Supported model and configured local backend; documentation only, no execution recorded.
- **yes · documented**: Provides a local HTTP inference server. [llama.cpp server documentation](https://github.com/ggml-org/llama.cpp/blob/38a5b42d9a3e82e0a586bcd1caed121f36c87a73/tools/server/README.md) Scope: Self-managed server; configure bind address and access controls.
- **yes · documented**: Exposes an OpenAI-style chat-completions endpoint. [llama.cpp server documentation](https://github.com/ggml-org/llama.cpp/blob/38a5b42d9a3e82e0a586bcd1caed121f36c87a73/tools/server/README.md) Scope: The server explicitly avoids promising full OpenAI API compatibility.
- **yes · documented**: Streams completion output using server-sent events. [llama.cpp server documentation](https://github.com/ggml-org/llama.cpp/blob/38a5b42d9a3e82e0a586bcd1caed121f36c87a73/tools/server/README.md) Scope: The documented /completion stream option.
- **yes · documented**: Accepts JSON schema constraints for generated responses. [llama.cpp server documentation](https://github.com/ggml-org/llama.cpp/blob/38a5b42d9a3e82e0a586bcd1caed121f36c87a73/tools/server/README.md) Scope: Supported schemas and model configuration; semantic correctness untested.
- **yes · documented**: Supports parallel decoding and continuous batching. [llama.cpp server documentation](https://github.com/ggml-org/llama.cpp/blob/38a5b42d9a3e82e0a586bcd1caed121f36c87a73/tools/server/README.md) Scope: Configured server slots and available resources; no throughput measured.
- **yes · documented**: Can reuse a matching prompt prefix through the KV cache. [llama.cpp server documentation](https://github.com/ggml-org/llama.cpp/blob/38a5b42d9a3e82e0a586bcd1caed121f36c87a73/tools/server/README.md) Scope: cache_prompt; reuse can affect determinism across batch sizes.
- **yes · documented**: Exposes draft-model speculative decoding controls. [llama.cpp server documentation](https://github.com/ggml-org/llama.cpp/blob/38a5b42d9a3e82e0a586bcd1caed121f36c87a73/tools/server/README.md) Scope: Compatible draft setup required; acceleration is not guaranteed.

## Open questions

- Which pinned model, quantization, context length and runtime versions fit the selected Mac?
- What are measured latency, throughput, peak memory and swap under one and four concurrent requests?

## Verification plan

1. Choose hardware and obtain explicit approval for installations, downloads and execution resources.
2. Pin runtime and engine versions, model artifact hashes, tokenizer, template, quantization and inference settings.
3. Run the local-model-mac investigation with synthetic prompts and capture raw measurements; do not record documentation as execution.

Last checked: 2026-09-15. Recheck by: 2026-10-15.

Content rights: Unlicensed prototype; redistribution decision pending.