Skip to content

Topic

DataFrame libraries

Libraries such as pandas, Polars and Dask that provide tabular DataFrame structures for data analysis, differing in execution model, typing and API design.

Current stories

buildOne report1 publisher

Polars 2.0's streaming default changes left-join row order from run to run

Polars 2.0.0 runs lazy queries on the streaming engine by default, and a left join on that engine no longer keeps row order. A dev.to author's repro shows the change raising no error and leaving every value correct.

Publishers:dev.to

Reality

Evidence62
Adoption
Not scored
Hype gap+8
Incentives40
Confidence60
buildOne report1 publisher

Aida van de Wetering rebuilds GeoPolars as a geo namespace inside Polars in about 125 commits

Aida van de Wetering has pushed about 125 commits to GeoPolars' dev branch since mid-September, rebuilding it as a geo expression namespace inside Polars. Polars teams with spatial work now have something to test before they reach for GeoPandas, though the project is still a prototype.

Publishers:dev.to

Reality

Evidence30
Adoption
Not scored
Hype gap+35
Incentives
Not scored
Confidence35
buildOne report1 publisher

Polars 2.0's streaming default can reorder group_by and join output without an error

Polars 2.0 runs LazyFrame.collect() on its streaming engine by default, a dev.to upgrade guide reports. One environment variable pins the old engine, so teams can fix the loud API breaks before hunting row-order changes that raise no error.

Publishers:dev.to

Reality

Evidence50
Adoption
Not scored
Hype gap+5
Incentives20
Confidence50
buildOne report1 publisher

A cached strptime only wins when the same date shows up 2,740 times

A week of measurements on one Mac mini puts the lru_cache wrapper around strptime at 3x when dates repeat and slightly slower when they do not. A Polars 2.0 release candidate finishes the same report in 0.13 s.

Publishers:dev.to

Reality

Evidence70
Adoption20
Hype gap+12
Incentives30
Confidence60
buildConfirmed2 publishers

Polars 2.0-rc repoints engine="auto" to the streaming engine for every lazy collect

The release candidate keeps collect's signature and changes what its default resolves to. Group_by, joins and unpivot can now return rows in a different order, and explain() still prints the plan that was not run.

Publishers:docs.pola.rs pola.rs thenewstack.io

Perspective Coverage

3 publishers
Builder
Builder 57%
Operator
Operator 30%
Investor
Investor 13%

Reality

Evidence80
Adoption18
Hype gap+22
Incentives62
Confidence74
buildOne report1 publisher

Polars 2.0 routes every lazy collect() through the streaming engine by default

Ritchie Vink is selling 2.0 as a boring upgrade. Mostly it is, but engine="auto" now resolves to streaming by default, so joins and group_bys stop carrying incidental row order, and pipelines that leaned on it will drift quietly.

Publishers:runtimewire.com

Reality

Evidence62
Adoption20
Hype gap+10
Incentives78
Confidence58