llama.cpp

Name: llama.cpp
Author: ggml-org (Georgi Gerganov and contributors)

In-depth entry

by ggml-org (Georgi Gerganov and contributors) · Library · also known as llama-cpp, ggml, llama-server

C/C++ inference engine for LLMs and vision-language models with GGUF quantized weights, Metal, CUDA, Vulkan and other backends, a CLI and an OpenAI-compatible server

MIT licenseOpen sourceFreeLocalSelf-hostedSelf-hostable

Visit official site Compare

Link verified Nov 2, 2026

Overview

llama.cpp is an MIT-licensed C/C++ engine for running large language models, and increasingly vision-language models, on commodity hardware. It began in 2023 as a port of Meta's LLaMA to Apple Silicon and grew into the reference implementation behind the GGUF file format and the GGML tensor library; the repository moved from ggerganov to the ggml-org organization. Models are converted from Hugging Face checkpoints with the bundled convert_hf_to_gguf.py script and quantized to integer formats from 1.5 to 8 bits, which lets multi-billion-parameter models fit in laptop memory.

Hardware coverage is unusually wide: ARM NEON, Accelerate and Metal on Apple Silicon; AVX variants and AMX on x86; CUDA, HIP, MUSA, Vulkan, SYCL, OpenCL, WebGPU, CANN and other accelerators; plus hybrid CPU-GPU offload when a model exceeds VRAM. The project ships a CLI (llama cli) that can download models directly from Hugging Face, a completion tool with GBNF grammar-constrained output, and llama-server, an OpenAI-compatible HTTP server with a built-in web UI. Installation is a script, Docker image, prebuilt release or source build, and bindings exist in the wider ecosystem for Python, Go, Rust, Node and others.

llama.cpp is the engine inside Ollama, LM Studio and many other front ends, so it is the layer to reach for when you need the newest model architectures, fine-grained control of quantization and sampling, or a minimal dependency footprint on a server. Its limits are that it is a single-node engine optimized for latency on one or a few requests rather than for high-concurrency batching, that model support depends on community conversion work, and that the fast release cadence occasionally changes CLI flags and file layouts.

Capabilities

GGUF model format with integer quantization from 1.5- to 8-bit
Backends for Metal, CUDA, HIP, Vulkan, SYCL, OpenCL, WebGPU and CPU SIMD
Hybrid CPU+GPU offload for models larger than VRAM
`llama-server` OpenAI-compatible API with web UI
CLI with direct Hugging Face downloads and grammar-constrained generation
Multimodal (vision-language) inference
Conversion scripts from Hugging Face checkpoints
Speculative decoding and KV-cache controls

Typical workflows

Convert and quantize a Hugging Face model to GGUF for a Mac Studio
Run `llama-server` as a local OpenAI-compatible endpoint for an application
Benchmark quantization levels against quality on a given machine
Embed the C API inside a native application

Strengths

Minimal dependencies and portable C/C++ codebase
Fastest adoption of new open-weight architectures
Runs on hardware from Raspberry Pi to multi-GPU servers
Permissive MIT license

Limitations

Not designed for high-concurrency serving
Requires model conversion to GGUF
Frequent breaking changes in CLI and build options
Quality loss at aggressive quantization levels

Prerequisites

C/C++ toolchain or a prebuilt release; GPU drivers for accelerated backends
Python for conversion scripts

Who it is for

Developers embedding local inference
Researchers experimenting with quantization
Operators of edge and on-premises inference

Alternatives & comparison

Ollama and LM Studio wrap llama.cpp with model management and friendlier interfaces, trading control for convenience. vLLM is the server-grade alternative with PagedAttention and continuous batching for many concurrent users on data-center GPUs. mlx-lm is Apple's framework-native option on Apple Silicon and uses its own weight format instead of GGUF.

Core ML
Apple's on-device machine-learning runtime for iOS and macOS apps, with the open-source coremltools package for converting PyTorch and other models to its format
MixedFree
exo
Open-source software that pools several Apple Silicon Macs (and Linux hosts) into one local inference cluster, sharding large models across devices with MLX
Open sourceOpen source
LM Studio
Proprietary desktop app for discovering, downloading and running local LLMs with llama.cpp and MLX engines, a chat UI, an OpenAI-compatible server, CLI, SDKs and a headless daemon
ProprietaryFreemium
MLX
Apple's open-source array framework for machine learning on Apple Silicon, with NumPy-like Python, C++ and Swift APIs and unified CPU/GPU memory
Open sourceOpen source
MLX LM
Python package and CLI for running, serving, quantizing and fine-tuning large language models on Apple Silicon with MLX
Open sourceOpen source
Ollama
Open-source runtime and model manager for running open-weight LLMs locally on macOS, Windows and Linux, with a REST API, OpenAI- and Anthropic-compatible endpoints and optional cloud models
Open sourceOpen source
vLLM
High-throughput LLM inference and serving engine with PagedAttention, continuous batching, quantization and an OpenAI-compatible server for GPU and accelerator clusters
Open sourceOpen source

Compare llama.cpp with its alternatives →

Sources & verification

Primary sources consulted for this record. Claims are tied to the fields each source supports; the automated link checker re-verifies every URL on a schedule.

License
https://github.com/ggml-org/llama.cpp/blob/master/LICENSE
Supports: license_spdx, source_model
checked Nov 1, 2026
Repository
ggml-org/llama.cpp
https://github.com/ggml-org/llama.cpp
Supports: name, capabilities, operating_systems, interfaces
checked Nov 1, 2026

Confidence: High
Reviewed: Nov 1, 2026
Links: Verified · Nov 2, 2026

Spotted an error?

MLX

Apple Inc. · Apple Silicon ML & MLX

Apple's open-source array framework for machine learning on Apple Silicon, with NumPy-like Python, C++ and Swift APIs and unified CPU/GPU memory

MITOpen sourceFreeLocalNov 2, 2026

Ray

Ray project (Anyscale and contributors) · Distributed & parallel computing

Open-source Python framework for distributing tasks, actors and ML workloads across a cluster, with libraries for data processing, training, tuning, serving and reinforcement learning

Apache-2.0Open sourceFreeLocalSelf-hostedBroken

Unsloth

Unsloth AI · Training & fine-tuning

Open-source framework for fine-tuning and running LLMs with reduced memory, covering LoRA, QLoRA, full fine-tuning and RL, GGUF export, Dynamic quantizations and a local Studio/Desktop UI

Apache-2.0Open sourceFreeLocalSelf-hostedNov 2, 2026

Core ML

Apple Inc. · Apple Silicon ML & MLX

Apple's on-device machine-learning runtime for iOS and macOS apps, with the open-source coremltools package for converting PyTorch and other models to its format

BSD-3-ClauseFreeLocalNov 2, 2026

Transformers

Hugging Face · Training & fine-tuning

Hugging Face's model-definition library for text, vision, audio, video and multimodal models, with Pipeline, Trainer and generate APIs used as the reference by training and inference engines