Skip to content

Polars

A blazing-fast DataFrame library written in Rust, with a Python API. Polars parallelizes across CPU cores by default and uses lazy evaluation for large data — a strong alternative to pandas when performance matters.

Table of Contents

Installation / Setup

pip install polars            # or: uv add polars
pip install "polars[pyarrow]" # optional Arrow interop extras

import polars as pl

Polars ≥ 1.0 is stable. Zero required third-party dependencies; multithreading is on by default.

Core Syntax

Creating DataFrames

df = pl.DataFrame({
    "name": ["ada", "lin", "sam"],
    "score": [92, 85, 78],
    "group": ["a", "a", "b"],
})
df.shape            # (3, 3)
df.columns          # ['name', 'score', 'group']
df.dtypes           # per-column types
df.schema           # {name: String, ...}
Constructor Use
pl.DataFrame(dict) / pl.DataFrame(list_of_dicts) From Python data
pl.from_pandas(pdf) / pl.from_numpy(arr, schema=...) Interop
pl.DataFrame({"a": [1]}, schema={"a": pl.Int8}) Enforce schema
df.to_pandas() / df.to_numpy() Convert back

Eager vs lazy

df.filter(pl.col("score") > 80)        # eager: runs immediately

lazy = df.lazy().filter(pl.col("score") > 80)   # lazy: plan only
result = lazy.collect()                # execute now

Lazy queries are optimized (predicate pushdown, projection pushdown, join reordering) and streamed when possible.

Reading files

pl.read_csv("data.csv")                 # eager
pl.scan_csv("data.csv")                 # lazy, streaming
pl.read_parquet("data.parquet")
pl.scan_parquet("data.parquet")
pl.read_csv("f.csv", schema_overrides={"zip": pl.String}, try_parse_dates=True)

Common Operations

Selection & filtering

df.select("name")                        # column -> DataFrame
df.select(["name", "score"])
df.select(pl.col("score").alias("s"))    # rename while selecting
df.select(pl.col("^score.*$"))           # regex column selection
df.with_columns((pl.col("score") * 1.1).alias("bonus"))
df.filter(pl.col("score") > 80)
df.filter(pl.col("score").is_between(60, 90))
df.filter(pl.col("name").is_in(["ada", "sam"]))
df.filter(pl.col("name").str.contains("a"))
df.head(3) / df.tail(3)
df.unique(subset=["group"])              # drop duplicates (keep="first"/"last"/"none")

Columns, strings, datetimes

df.with_columns(
    pl.col("name").str.strip().str.to_lowercase(),
    pl.col("code").str.extract(r"(\d+)", group_index=1),
)
df.with_columns(
    pl.col("date").str.to_date("%Y-%m-%d"),       # parse strings
    pl.col("date").dt.year(),                     # .dt accessor
    pl.col("date").dt.month(),
    pl.col("date").dt.strftime("%Y-%m"),
)

Missing values

df.null_count()
df.drop_nulls()                        # any column
df.drop_nulls(subset=["score"])
df.fill_null(0)
df.with_columns(pl.col("score").fill_null(pl.col("score").median()))
df.with_columns(pl.col("score").forward_fill())   # ffill
df.with_columns(pl.col("score").interpolate())

group_by & aggregation

df.group_by("group")["score"].sum()
df.group_by("group").agg([
    pl.col("score").sum().alias("total"),
    pl.col("score").mean().alias("avg"),
    pl.len().alias("count"),
])
df.group_by(["group", "year"]).agg(pl.col("score").max())
df.group_by("group").agg(pl.col("score").sort().tail(2))   # top 2 per group

group_by (with underscore) is the modern name; groupby still works as an alias.

Sorting & ranking

df.sort("score", descending=True)
df.sort(["group", "score"], descending=[False, True])
df.with_columns(pl.col("score").rank(descending=True).alias("rank"))
df.top_k(5, by="score")                # nlargest

Joins & concatenation

df_a.join(df_b, on="user_id", how="left")     # how: inner/left/outer/cross
df_a.join(df_b, left_on="id", right_on="user")
df_a.join_asof(df_b, on="timestamp")          # nearest-time join

pl.concat([df_jan, df_feb], how="vertical")   # stack rows
pl.concat([df_x, df_y], how="horizontal")     # side by side
pl.concat([d1, d2], how="diagonal")           # union of columns, fill null

Pivot & reshape

df.pivot(index="date", columns="city", values="temp", aggregate_function="mean")
df.melt(id_vars=["user"], value_vars=["q1", "q2"], variable_name="quarter", value_name="score")
df.explode("tags")                     # one row per list element
df.unnest("struct_col")                # expand a struct column

Practical Examples

1. Lazy pipeline over many files

import glob

q = pl.scan_csv(glob.glob("sales/2024/*.csv"))   # one lazy plan, all files
out = (q
    .filter(pl.col("amount") > 0)
    .with_columns(pl.col("date").str.to_date())
    .group_by([pl.col("date").dt.year().alias("year"), "region"])
    .agg(pl.col("amount").sum().alias("revenue"))
    .sort("revenue", descending=True))
result = out.collect()                            # optimized + parallel

2. Replace a slow pandas groupby-apply

# pandas  df.groupby("id")["v"].apply(lambda s: s.diff().fillna(0))
df.with_columns(pl.col("v").diff().fill_null(0).over("id"))

3. Window functions with .over()

df.with_columns([
    pl.col("score").mean().over("group").alias("group_mean"),
    pl.col("score").rank(descending=True).over("group").alias("rank_in_group"),
])

Common Patterns

# Streaming a huge file (no full load)
q = pl.scan_csv("huge.csv").filter(pl.col("amount") > 0)
q.sink_parquet("filtered.parquet")     # streams to disk; collect() would load

# Schema-precise read
pl.read_csv("messy.csv",
            schema_overrides={"col": pl.Float64, "date": pl.Date},
            null_values=["", "NA", "n/a"])

# Chained transforms, single collect
df.lazy().filter(...).with_columns(...).group_by(...).agg(...).collect()

# Interop with pandas keeps both ecosystems usable
pdf = df.lazy().collect().to_pandas()

Best Practices

  • Default to lazy (scan_* + .collect()) for files; Polars optimizes the whole plan.
  • Prefer expressions over row-wise functions — pl.col, .over, .group_by are parallel; map_rows is a last resort.
  • Use sink_* to stream results to disk for datasets larger than memory.
  • Declare schemas on read (schema_overrides) for messy files — faster and more robust than pandas inference.
  • Use .alias() freely — expressions read best as named pipelines.
  • Keep one column type per column: Polars is strict where pandas is permissive, and that's the point.
  • Benchmark against pandas with your own data before migrating — Polars wins big on wide/long numeric data and many files.

Common Mistakes

Mistake Consequence Correct approach
.collect() on every step Kills lazy optimization Chain lazy ops, collect once
df["col"] habits from pandas Confusing API errors df.select(pl.col("col")), df.get_column("col")
Expecting NaN semantics Wrong results on missing data Polars uses null, not NaN; check with null_count()
Row-wise Python functions on big data No speedup vs pandas Rewrite as expressions
groupby without .agg Missing aggregation error Always pass .agg([...]) or a single expression
Joining on columns of mixed types Join mismatch / type errors Cast keys: pl.col("id").cast(pl.Int64) on both sides

Troubleshooting

Symptom Cause Fix
ComputeError: cannot convert column ... Type mismatch on join/op .cast(pl.Float64) etc.
Lazy query returns nothing Filter pushed into scan with bad schema Check q.explain() to see the optimized plan
OutOfMemoryError on collect Result too large for RAM Use .sink_parquet(...) or more filtering
String column parsed as Object Mixed content schema_overrides={"c": pl.String}
Surprised by null vs NaN results Different missing semantics Prefer fill_null and .is_null() filters
Slow first query Compilation/startup warm-up Normal — subsequent queries reuse cached binaries

Official Documentation

Last Updated

2026-09-06 — Initial version.