Polars 2.0-rc repoints engine="auto" to the streaming engine for every lazy collect
The release candidate keeps collect's signature and changes what its default resolves to. Group_by, joins and unpivot can now return rows in a different order, and explain() still prints the plan that was not run.
The Engineer · Build desk

What happened
- Polars 2.0-rc leaves collect's default argument as engine="auto" and changes what auto means: lazy queries now resolve to the streaming engine, where before they resolved to the in-memory engine.
- The streaming engine gives no row-order guarantee for operations that do not require one, and the upgrade guide names unpivot, group_by and joins as the affected cases.
- The guide puts that in a red danger box, warning in its own words that the change may silently impact the results of your pipelines.
- Three routes back to the old default exist: a process-wide engine affinity setting, a per-query engine argument, and the POLARS_ENGINE_AFFINITY environment variable.
Why it matters
- What it limits constraint Diffing query plans before and after the upgrade will not find the affected queries, because explain() and show_graph() keep rendering the non-streaming plan unless you pass engine="streaming" yourself.
- Who's exposed exposure The guide concedes a join is the easy case to miss because nothing in the query looks order-sensitive, which puts the exposure in downstream consumers of the result rather than in the query under review.
- Who has to decide decision Each team now picks between per-query maintain_order and process-wide in-memory affinity, and the affinity route keeps the old semantics by declining the performance change the release exists to deliver.
- What it costs cost According to The New Stack, enforcing order may cost speed, and no figure is published for it, so anyone keeping an order-dependent join has to measure the loss on their own query before trusting the aggregate claim.
Next story · Build · 4d ago
OpenCode 2.0 runs its terminal, desktop and web clients against one background processConfirmed2 publishers
The default argument in `collect()` is still `engine="auto"`. What changed is what `auto` resolves to: the streaming engine for lazy queries, where it used to be the in-memory engine [1]. The call site itself is unchanged, and no error gets raised. The pre-2.0 row order for `group_by`, `join` and `unpivot` was never a documented contract, and the streaming engine does not offer one either [3]. That governs how you fix it. The guide's own `unpivot` example comes back grouped per index key rather than per variable, and the guide then says the order shown is not guaranteed, only that it is no longer the pre-2.0 order [11]. Re-recording a snapshot test against the new output does not pin the order down. The two supported fixes are an explicit `.sort(pl.all())` or `maintain_order` where the operation supports it, as in `join(..., maintain_order="left")` [5].
Some of the blast radius is narrower than it looks. `sink_*` was already dispatched to the streaming engine regardless of `engine`, so file-writing pipelines have had these semantics all along [10]. Eager `DataFrame` operations still resolve `auto` to in-memory internally [2]. The wider part is SQL: `pl.sql(..., eager=True)` and `SQLContext.execute(eager=True)` used to collect through the in-memory engine and now go through `LazyFrame.collect()`, so a query string with no engine argument anywhere in sight picks up the new default [12].
Column order breaks in a second, unrelated place. `pl.read_csv` is now dispatched to `pl.scan_csv(...).collect()`, gaining six scan parameters including `with_column_names` and `credential_provider`, and losing `n_threads`, `batch_size`, `sample_size` and `rechunk` [13]. Its behaviour now matches `scan_csv` exactly, so `columns=[2, 1, 3]` returns `['c', 'b', 'd']` where it used to return the sorted `['b', 'c', 'd']`, and a `schema_overrides` list must cover every column in the file or raise `SchemaError` [14]. That failure now surfaces at the boundary rather than downstream.
The strictness work is the part of this release I would ship unchanged. Polars' example is a flagged-account check: `flagged_ids` arrives from JSON as `Float64` holding 9007199254740992.0, the user ID is the `Int64` 9007199254740993, and pre-2.0 coercion to the common supertype rounded the ID down and returned a false positive [15]. The ID sits exactly one above 2^53, the largest integer float64 represents exactly [19]. In 2.0 that raises `InvalidOperationError` [15]. Horizontal `concat` now checks lengths rather than filling the short frame with nulls [16]. Both changes convert a silent wrong answer into a stack trace, which is the trade the row-order default makes in the opposite direction.
Polars expects the streaming engine to be easily 5x faster in aggregate [20]. That is a number about their aggregate. For it to transfer, your work has to be the kind that batches, which is what the streaming engine does instead of processing all data at once [7]: a working set at or past memory, or scans that stream, or joins you are willing to leave unordered. Every `maintain_order` you add pays some of it back, and The New Stack notes the speed cost without quantifying it [21].
So the sequence I would use is affinity first. Set `POLARS_ENGINE_AFFINITY=in-memory`, absorb the 2.0 API changes, get green, then move queries onto streaming one at a time as each order assumption is removed or made explicit [8]. That turns one upgrade into two changes with separate blast radii. The announcement says the team hopes 2.0 is a boring experience [9]; the red danger box in the migration guide is doing a fair amount of work to keep that promise [6].</body_markdown> </invoke>
What to watch
- Whether the final 2.0 release, which Polars says lands in the following weeks, documents any order guarantee or cost figure for maintain_order.
- Polars has hinted at a cost-based planner and join reordering for 2.x, which would move plans again after teams finish auditing for this change.
- How many more eager readers get re-dispatched through scan_*: read_ipc already goes through scan_ipc for all non-use_pyarrow inputs.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence80
- Adoption18
- Hype gap+22
- Incentives62
- Confidence74
Perspective Coverage
3 publishers- Builder
- Builder 57%
- Operator
- Operator 30%
- Investor
- Investor 13%
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
LazyFrame.collect and collect_async still default to engine="auto", but "auto" now resolves to the streaming engine for lazy queries; it used to resolve to the in-memory engine.
- [2]
Eager DataFrame operations are unaffected and continue to resolve "auto" to the in-memory engine internally.
- [3]
The streaming engine does not guarantee row order for operations that do not require it, the guide listing unpivot, group_by and joins.
- [4]
The guide says joins are affected the same way as unpivot and are easy to miss, since nothing about the query looks order-sensitive.
- [5]
The guide's remedies are to sort explicitly, for example .collect().sort(pl.all()), or to pass maintain_order where the operation supports it, for example join(..., maintain_order="left").
- [6]
The 2.0-rc upgrade guide marks the change with a "Danger" box stating: "This change may silently impact the results of your pipelines."
- [7]
Polars says streaming executes lazy queries in batches rather than processing all data at once, allowing users to process datasets that do not fit into available memory.
- [8]
The previous default engine can be restored with pl.Config.set_engine_affinity("in-memory") process-wide, collect(engine="in-memory") per query, or the POLARS_ENGINE_AFFINITY=in-memory environment variable.
- [9]
Polars released the first release candidate for 2.0, says the definite 2.0 release will land in the following weeks, does not aim to make a big feature release, and hopes it will be a boring experience for users.
- [10]
sink_* is not affected by the change: writing to a file was already dispatched to the streaming engine regardless of the engine argument.
- [11]
In the guide's unpivot example the pre-2.0 output is grouped by the variable column (x b 1, y b 2, z b 3, then x c 4, y c 5, z c 6) and the post-2.0 output is grouped per index key (x b, x c, y b, y c, z b, z c); the guide states the exact row order shown is not guaranteed, only that it is no longer the pre-2.0 order.
- [12]
SQLContext.execute(eager=True) and pl.sql(..., eager=True) used to collect through the in-memory engine and now go through LazyFrame.collect() like any other lazy query; keeping the in-memory engine requires eager=False plus collect(engine="in-memory").
- [13]
pl.read_csv is now dispatched to pl.scan_csv(...).collect(); it gains with_column_names, infer_schema_files, credential_provider, include_file_paths, extra_columns and missing_columns and accepts a list of sources, and loses n_threads, batch_size, sample_size and rechunk.
- [14]
read_csv behaviour now matches scan_csv exactly: columns=[2, 1, 3] returns ['c', 'b', 'd'] where it previously returned ['b', 'c', 'd'], and a schema_overrides list must cover every column in the file or a SchemaError is raised.
- [15]
Polars' is_in example: flagged_ids is a Float64 Series holding 9007199254740992.0 and user_id is the Int64 9007199254740993; before 2.0 the Int64 was coerced to Float64 and rounded down, giving a false positive, and in 2.0 the expression raises InvalidOperationError.
- [16]
Horizontal concat in 2.0 checks lengths instead of silently filling with null.
- [17]
explain() and show_graph() are not affected by the default change: they only render a streaming plan when engine="streaming" is passed explicitly.
- [18]
pl.read_ipc is now dispatched to pl.scan_ipc(...).collect() for all non-use_pyarrow inputs.
- [19]
The user ID in the example is exactly one greater than 2^53 = 9007199254740992, the largest integer float64 can represent exactly.
- [20]
Polars says the biggest default change is that all LazyFrame queries run on the streaming engine, and that it expects the streaming engine to be easily 5x faster in aggregate.
- [21]
The New Stack writes that users can enforce row order, but there is a chance it may cost some speed, and gives no figure for the cost.
- [22]
The New Stack reports Polars hinted at more for 2.x, including a new IO-plugin design, a faster S3 reader, a cost-based planner, join reordering and big SQL coverage improvements.
Sources
2 independent publishers whose own reporting we read for this story.
- Version 2.0-rc - Polars user guide
docs.pola.rs
1 article · September 7, 2026
- Polars — Pre-release of Polars 2.0
pola.rs
1 article · September 7, 2026
- thenewstack.ioPolars 2.0 pre-release comes with a 5x speed boost — but it could change row order
1 article · September 6, 2026
Topics and entities
Topics
- Breaking Changes and Migration
- Data pipeline correctness
- DataFrame libraries
- Streaming query execution
Entities
- Polars streaming engine
- Polars
- The New Stack
- Polars in-memory engine