Agent Lightning

Agent Lightning is a reinforcement learning framework from Microsoft for training AI agents while keeping their existing harnesses (tools, context, control flow, environments) completely intact. It's built for ML engineers and researchers who already have a working agent (built with AutoGen, custom code, or another framework) and want to improve its performance on a specific task using RL, rather than rebuilding it inside a training-specific abstraction.

The core idea is a proxy-based architecture: the agent talks to a model through the Agent Lightning API Gateway with zero changes to the agent's code. The Gateway captures the interactions as training data, a Rollout Controller runs the agent (locally or as a Kubernetes Job), and the Trainer (built on verl and vLLM) uses the captured rollouts to update the policy. This separation lets the agent keep its real tools and environment in the loop during training instead of running in a simplified simulation.

Key features

• Real harness training: agents connect to the Agent Lightning proxy with no code changes, keeping their actual tools, context, and control flow during RL training.
• Small codebase: around 3,500 lines of code, which keeps the system easier to read, debug, and extend compared to larger RL training stacks.
• Kubernetes-native rollouts: the Rollout Controller can run agents directly as Kubernetes Jobs, removing the need for external sandbox services.
• Full training pipelines as examples: includes end-to-end examples (coding agent, Search-R1, ScienceWorld, GSM8K, Calc-X, LLM-in-Sandbox) with data cleaning, reward-hacking mitigation, and training scripts.
• verl and vLLM integration: the Trainer component builds on verl for policy optimization and vLLM for inference, with documented configuration for trace aggregation and async training.
• Asynchronous training support: supports collocated async rollout collection with pause/drain semantics for more efficient GPU use.

Ideal use cases

Agent Lightning fits teams that already have a functioning agent, written with AutoGen, raw API calls, or another orchestration layer, and want to improve its task performance with RL without re-architecting it. The coding agent examples show this directly: training a Qwen3.5-9B-based workflow on 6K samples moved SWE-bench Verified scores from 41.8% to 56.4%, and a larger MoE model improved from 47.8% to 61.6% with 1.8K examples.

It's also a reasonable choice for research setups exploring RL on multi-turn, tool-using agents (retrieval agents, sandboxed code execution agents, interactive text environments) where keeping the real environment during training matters more than training speed alone. Native Kubernetes support makes it a fit for teams already running GPU workloads on k8s clusters.

It's not a good fit if you need a simple inference-time agent framework, a no-GPU setup, or a plug-and-play managed service. You need access to GPU infrastructure (the examples assume CUDA GPUs and verl/vLLM), familiarity with RL training concepts, and willingness to configure a multi-component system (Trainer, Gateway, Controller). If your goal is just prompting or fine-tuning without an RL loop, standard fine-tuning tools are a better match.

Installation

The project targets CUDA GPU environments and uses uv for dependency management. A typical setup on a CUDA 13.0 machine:

cd <this-repo>
uv sync
bash scripts/setup_verl.sh 0.8.0 cu130

This installs the base Python environment with uv sync, then sets up the verl GPU stack matching your CUDA version via the setup script. The official Installation Guide covers alternative CUDA versions and more detailed environment requirements. After installation, the documentation's Quick Start guide walks through a local first run and the full end-to-end training flow (Trainer, API Gateway, Rollout Controller) before moving to the example pipelines (Calc-X, GSM8K, Search-R1, coding agent, etc.) for a working reference implementation.