polars

Polars is an analytical query engine for DataFrames, written in Rust and designed to be fast, memory-efficient, and expressive. It's built for developers and data practitioners doing data manipulation and analysis who want performance closer to a compiled language without giving up a DataFrame-style API, and who may be hitting the limits of tools like pandas on larger datasets.

Key features

• Lazy and eager execution: Queries can run immediately (eager) or be built up and optimized before execution (lazy), letting the query planner reorder and simplify operations before running them.
• Streaming execution: Larger-than-RAM datasets can be processed in a streaming fashion by collecting with engine='streaming', which reduces memory requirements enough to work with datasets far bigger than available memory.
• Query optimization: A query optimizer works with the lazy API to plan efficient execution rather than running operations in the order they were written.
• Multi-threaded and SIMD-accelerated: Polars is built to use multiple threads and SIMD instructions for performance.
• Multiple language front ends: The same underlying engine is exposed through Python, Rust, Node.js, R, and SQL front ends, all documented separately.
• Apache Arrow columnar format: Data is represented using the Apache Arrow Columnar Format, which supports efficient interop with other Arrow-based tools.
• Lightweight with zero required dependencies: The README reports Polars importing in roughly 70ms versus 104ms for numpy and 520ms for pandas, with no required dependencies.
• Rust UDFs from Python: PyO3 extensions let you extend Polars with custom functions compiled in Rust and called from the DataFrame and Series APIs.

Ideal use cases

Polars fits data manipulation and analysis workloads where speed and memory efficiency matter: cleaning and reshaping large CSV or Parquet datasets, building data pipelines, or replacing pandas in a workflow that's become slow at scale. Its streaming execution mode is particularly useful when a dataset doesn't comfortably fit in memory, since it can process the data in chunks rather than requiring everything loaded at once. Teams working across Python, Rust, Node.js, or R can share the same query semantics rather than maintaining separate tooling per language.

It's a less natural fit if you need a general-purpose SQL database with concurrent multi-user access, transactions, or persistent storage management; Polars is a query engine for DataFrames, not a database server. If your datasets are small and pandas already performs well enough for your workflow, the migration effort may not be worth it, though the API is close enough to pandas that many operations translate directly.

Installation

For Python, install the latest version with pip:

pip install polars

Check the installed version and its optional dependencies with:

pl.show_versions()

The user guide covers optional dependencies and feature flags in more detail for anyone who needs extras beyond the base install.

To compile Polars from source, for a bleeding-edge release or maximum performance:

# 1. Install the Rust compiler, then maturin
pip install maturin

# 2. From the py-polars directory, choose a build target
cd py-polars
make build              # debug build, fast compile
make build-release      # optimized, no debug assertions
make build-dist-release # fastest binary, slowest compile

Set LTS_CPU=1 with the build command if targeting an older CPU without AVX2 support. If you expect more than about 4.2 billion rows, compile with the bigidx feature flag, or install polars[rt64] from Python; this isn't needed unless you actually hit that row limit, since the default build is faster and uses less memory. For older CPUs or running under Rosetta on Apple Silicon, polars[rtcompat] provides a build without AVX target features.

For a managed or distributed option, Polars Cloud is available as a separate offering.