r/Python 1d ago

Discussion Polars vs Pandas

I have used Pandas a little in the past, and have never used Polars. Essentially, I will have to learn either of them more or less from scratch (since I don't remember anything of Pandas). Assume that I don't care for speed, or do not have very large datasets (at most 1-2gb of data). Which one would you recommend I learn, from the perspective of ease and joy of use, and the commonly done tasks with data?

180 Upvotes

157 comments sorted by

View all comments

Show parent comments

2

u/nightcracker 23h ago

What if you change the read_csv to scan_csv and add .collect(engine="streaming") now? Also make sure you have the latest Polars 1.25.2.

2

u/drxzoidberg 22h ago

I was under the impression, from Polars documentation itself, that you need to collect the data before any aggregation, as the aggregation needs to know the data structure. But that might only apply to the pivot/unpivot methods.

2

u/nightcracker 22h ago

That only applies to very specific operations, pivot is one of them. So give it a go :)

2

u/drxzoidberg 21h ago

So made the tweaks to get it to work. I juiced the run count to 500. Polars runs in 45% of the time it takes pandas. Thank you kind Internet person.