Tools · SPBoucher
Menu

vLLM

In-depth entry

by vLLM project (originated at UC Berkeley Sky Computing Lab) · Library

High-throughput LLM inference and serving engine with PagedAttention, continuous batching, quantization and an OpenAI-compatible server for GPU and accelerator clusters

Apache-2.0 licenseOpen sourceFreeSelf-hostedSelf-hostable
Visit official site Compare

Link verified Nov 2, 2026

Overview

vLLM is an Apache-2.0 inference and serving engine for large language models, started at UC Berkeley's Sky Computing Lab and now maintained by a large community of academic and industry contributors. Its defining technique is PagedAttention, which manages the attention key-value cache in pages so that many requests can share GPU memory efficiently; combined with continuous batching, chunked prefill and prefix caching this gives far higher throughput than naive per-request inference. Further optimizations include speculative decoding (n-gram, EAGLE and others), FP8, INT8, INT4, GPTQ, AWQ and GGUF quantization, CUDA/HIP graphs and torch.compile.

The engine runs as a Python library or as an OpenAI-compatible HTTP server that also implements the Anthropic Messages API and gRPC, with streaming, structured output via grammar backends, tool-calling and reasoning parsers and multi-LoRA serving. It scales across tensor, pipeline, data, expert and context parallelism and supports disaggregated prefill and decode. Supported hardware covers NVIDIA and AMD GPUs, x86, ARM and PowerPC CPUs, Google TPUs, Intel Gaudi and several other accelerators, with Apple Silicon available through a hardware plugin. Over 200 Hugging Face architectures load directly, including mixture-of-experts, hybrid state-space, multimodal, embedding and reward models.

vLLM is the usual choice when an open-weight model must serve many concurrent users on data-center GPUs, whether on premises or in a cloud. It is heavier than laptop-oriented runtimes: installation depends on matching CUDA or ROCm versions, memory is pre-allocated for the KV cache, and single-user latency on a workstation is not its design goal. Apple Silicon support is a community plugin rather than a primary target.

Capabilities

  • PagedAttention KV-cache management with continuous batching and prefix caching
  • OpenAI-compatible server plus Anthropic Messages API and gRPC
  • Quantization: FP8, INT8, INT4, GPTQ, AWQ, GGUF and others
  • Tensor, pipeline, data, expert and context parallelism; disaggregated prefill/decode
  • Speculative decoding, structured outputs, tool-calling and reasoning parsers
  • Multi-LoRA serving for dense and MoE layers
  • Support for 200+ Hugging Face model architectures including multimodal and embedding models
  • NVIDIA, AMD, CPU, TPU, Gaudi and plugin-based hardware backends

Typical workflows

  1. Serve an open-weight chat model to a team behind an OpenAI-compatible endpoint
  2. Run offline batch generation over a large dataset with the Python API
  3. Deploy a quantized model across several GPUs with tensor parallelism
  4. Host embedding and reranking models alongside generation

Strengths

  • High throughput and memory efficiency on GPUs
  • Drop-in replacement for OpenAI-style clients
  • Very wide model and hardware coverage
  • Active community and frequent releases

Limitations

  • Installation sensitive to CUDA/ROCm and PyTorch versions
  • Designed for servers; not a laptop runtime
  • Apple Silicon only through a plugin
  • Pre-allocated KV cache requires tuning on small GPUs

Prerequisites

  • Linux server with supported GPU or accelerator and matching drivers
  • Python environment with PyTorch

Who it is for

  • ML platform and inference engineers
  • Research groups serving models to many users
  • Teams self-hosting open-weight models

Alternatives & comparison

llama.cpp and Ollama target single machines and CPU or consumer-GPU inference with GGUF weights, whereas vLLM targets concurrent serving on data-center accelerators from native Hugging Face checkpoints. Transformers can generate text but lacks vLLM's serving optimizations and is usually the upstream model definition vLLM loads. Together AI is a hosted alternative that removes operations work but keeps data off-premises.

  • exo

    Open-source software that pools several Apple Silicon Macs (and Linux hosts) into one local inference cluster, sharding large models across devices with MLX

    Open sourceOpen source
  • llama.cpp

    C/C++ inference engine for LLMs and vision-language models with GGUF quantized weights, Metal, CUDA, Vulkan and other backends, a CLI and an OpenAI-compatible server

    Open sourceOpen source
  • LM Studio

    Proprietary desktop app for discovering, downloading and running local LLMs with llama.cpp and MLX engines, a chat UI, an OpenAI-compatible server, CLI, SDKs and a headless daemon

    ProprietaryFreemium
  • MLX LM

    Python package and CLI for running, serving, quantizing and fine-tuning large language models on Apple Silicon with MLX

    Open sourceOpen source
  • Ollama

    Open-source runtime and model manager for running open-weight LLMs locally on macOS, Windows and Linux, with a REST API, OpenAI- and Anthropic-compatible endpoints and optional cloud models

    Open sourceOpen source
  • Together AI

    Cloud platform for open-source models: serverless and dedicated inference, fine-tuning, GPU clusters, batch jobs and code sandboxes behind an OpenAI-compatible API

    ProprietaryUsage-based
  • Transformers

    Hugging Face's model-definition library for text, vision, audio, video and multimodal models, with Pipeline, Trainer and generate APIs used as the reference by training and inference engines

    Open sourceOpen source
Compare vLLM with its alternatives →

Sources & verification

Primary sources consulted for this record. Claims are tied to the fields each source supports; the automated link checker re-verifies every URL on a schedule.

Confidence
High
Reviewed
Nov 1, 2026
Links
Verified · Nov 2, 2026

Spotted an error?

Broken link, outdated license, wrong category, or a tool that no longer exists? Tell us; reports are reviewed by the editor.

LM Studio

Element Labs, Inc. · Local LLM inference & serving

In-depth entry

Proprietary desktop app for discovering, downloading and running local LLMs with llama.cpp and MLX engines, a chat UI, an OpenAI-compatible server, CLI, SDKs and a headless daemon

ProprietaryFreemiumPaidLocalBroken

MLX

Apple Inc. · Apple Silicon ML & MLX

In-depth entry

Apple's open-source array framework for machine learning on Apple Silicon, with NumPy-like Python, C++ and Swift APIs and unified CPU/GPU memory

MITOpen sourceFreeLocalNov 2, 2026

MLX LM

Apple Inc. · Apple Silicon ML & MLX

In-depth entry

Python package and CLI for running, serving, quantizing and fine-tuning large language models on Apple Silicon with MLX

MITOpen sourceFreeLocalNov 2, 2026

Ray

Ray project (Anyscale and contributors) · Distributed & parallel computing

In-depth entry

Open-source Python framework for distributing tasks, actors and ML workloads across a cluster, with libraries for data processing, training, tuning, serving and reinforcement learning

Apache-2.0Open sourceFreeLocalSelf-hostedBroken

Unsloth

Unsloth AI · Training & fine-tuning

Open-source framework for fine-tuning and running LLMs with reduced memory, covering LoRA, QLoRA, full fine-tuning and RL, GGUF export, Dynamic quantizations and a local Studio/Desktop UI

Apache-2.0Open sourceFreeLocalSelf-hostedNov 2, 2026

Core ML

Apple Inc. · Apple Silicon ML & MLX

Apple's on-device machine-learning runtime for iOS and macOS apps, with the open-source coremltools package for converting PyTorch and other models to its format

BSD-3-ClauseFreeLocalNov 2, 2026