vllm

vLLM is a library and serving engine for running large language model inference at scale. It's built for developers and teams who need to serve LLMs efficiently in production, from single-GPU setups to large distributed clusters, without writing custom inference infrastructure.

The project grew out of research at UC Berkeley's Sky Computing Lab and is now maintained by a large open-source community spanning academic institutions and companies. It supports over 200 model architectures from Hugging Face, including decoder-only LLMs, mixture-of-experts models, hybrid state-space models, multi-modal models, and embedding/retrieval models.

Key features

• PagedAttention: manages attention key/value memory efficiently, reducing waste and enabling higher throughput.
• Continuous batching: processes incoming requests dynamically with chunked prefill and prefix caching to keep GPUs busy.
• Quantization support: FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF, compressed-tensors, ModelOpt, and TorchAO.
• Optimized kernels: FlashAttention, FlashInfer, TRTLLM-GEN, FlashMLA, Triton, and CUTLASS-based GEMM/MoE kernels.
• Speculative decoding: n-gram, suffix, EAGLE, and DFlash methods to speed up generation.
• torch.compile integration: automatic kernel generation and graph-level transformations, with piecewise and full CUDA/HIP graph support.
• Disaggregated serving: separates prefill, decode, and encode stages for more flexible deployment.
• Broad hardware support: NVIDIA and AMD GPUs, x86/ARM/PowerPC CPUs, plus plugins for Google TPUs, Intel Gaudi, IBM Spyre, Huawei Ascend, Apple Silicon, and others.
• Distributed inference: tensor, pipeline, data, expert, and context parallelism for scaling across multiple GPUs or nodes.
• API compatibility: OpenAI-compatible API server, Anthropic Messages API, and gRPC support, with tool calling and reasoning parsers.
• Multi-LoRA support: serve multiple LoRA adapters efficiently across dense and MoE layers.
• Structured output generation: via xgrammar or guidance for constrained decoding.

Ideal use cases

vLLM fits teams deploying LLMs for production inference, such as running a chatbot backend, a code-completion service, or a batch-processing pipeline that needs high request throughput on limited GPU hardware. It's a good match if you need an OpenAI-compatible API endpoint, want to serve open models like Llama, Qwen, DeepSeek-V3, or GPT-OSS without building custom serving code, or need to run multiple LoRA fine-tunes off a shared base model.

It also works well for research groups benchmarking serving throughput or experimenting with quantization, speculative decoding, or parallelism strategies at scale.

It's less suited for someone who just wants to prototype with a model locally on a laptop with no GPU. It's also not the right choice if you need training infrastructure. vLLM is an inference and serving engine, not a training framework, and setup complexity (CUDA drivers, hardware-specific builds) adds overhead for simple one-off scripts or CPU-only casual use.

Installation

Install vLLM with uv (recommended) or pip:

uv pip install vllm

Alternatively, use pip directly:

pip install vllm

For development or hardware not covered by the prebuilt wheels, build from source following the GPU installation guide.

After installation, consult the quickstart guide to launch the OpenAI-compatible server or run offline inference. Check the supported models list to confirm your target model is supported before deploying.