llama.cpp

llama.cpp is a C/C++ engine for running large language model inference on your own hardware, with no Python runtime or heavyweight dependencies required. It loads models in the GGUF format and runs them locally on CPU, or accelerated with CUDA, Metal, Vulkan, ROCm, SYCL, and several other backends, across x86, ARM, and RISC-V. It's the project behind the ggml tensor library, and it's aimed at developers who want to run open-weight language models, and a growing list of multimodal, embedding, and reranking models, directly on their own machines or servers instead of calling a hosted API.

Key features

• Plain C/C++ implementation: no external runtime dependencies, with first-class support for Apple Silicon (ARM NEON, Accelerate, Metal) and x86 (AVX, AVX2, AVX512, AMX).
• Quantization: supports 1.5-bit through 8-bit integer quantization, trading precision for lower memory use and faster inference.
• Wide backend support: CUDA and HIP for NVIDIA and AMD GPUs, Vulkan, SYCL for Intel GPUs, MUSA for Moore Threads GPUs, CANN for Ascend NPUs, and CPU+GPU hybrid inference for models bigger than available VRAM.
• llama-server: a lightweight, OpenAI API-compatible HTTP server with a built-in web UI, support for concurrent requests, speculative decoding, embedding and reranking endpoints, and constrained (grammar-based) output.
• llama-cli: a command-line tool for interactive conversation mode, custom chat templates, and grammar-constrained generation.
• Direct Hugging Face integration: download and run a model straight from Hugging Face with -hf <user>/<model>, with downloads stored in the standard Hugging Face cache directory so other HF tools can share them.
• Broad model coverage: supports most major open-weight text models (Llama, Mistral, Mixtral, Qwen, DeepSeek, Gemma, Phi, and dozens more) plus a growing list of multimodal models like LLaVA and Qwen2-VL.

Ideal use cases

llama.cpp fits anyone who wants to run an open-weight language model on their own hardware rather than through a hosted API: developers building local-first AI features, teams that need inference to run air-gapped or on-prem for privacy or cost reasons, and hobbyists running models on a laptop or a single GPU. The OpenAI-compatible server makes it a practical drop-in for existing tooling built against that API shape, and the quantization support means models that wouldn't otherwise fit in memory can still run, at some quality cost.

It's less suited to teams that need the absolute highest throughput at scale, where dedicated serving frameworks tuned for datacenter GPUs will usually beat it, or to anyone who wants to fine-tune models rather than just run inference. It also assumes some comfort with the command line and with managing GGUF files and quantization levels; if you want a fully polished point-and-click desktop app, one of the many GUI projects built on top of llama.cpp, listed in its own README, may be a better starting point than the engine itself.

Installation

Several installation paths are documented:

• Install via brew, nix, winget, or conda-forge (see docs/install.md in the repo).
• Run with Docker, following the project's Docker documentation.
• Download pre-built binaries from the releases page.
• Build from source by cloning the repository and following the build guide.

Once installed, run a model directly from a local file or straight from Hugging Face:

# Use a local model file
llama-cli -m my_model.gguf

# Or download and run a model directly from Hugging Face
llama-cli -hf ggml-org/gemma-3-1b-it-GGUF

# Launch an OpenAI-compatible API server
llama-server -hf ggml-org/gemma-3-1b-it-GGUF

The server exposes a basic web UI at http://localhost:8080 and a chat completion endpoint at /v1/chat/completions by default. Models need to be in GGUF format; other formats can be converted using the convert_*.py scripts included in the repository, or through Hugging Face's GGUF-my-repo space for converting and quantizing directly in the browser.