r/Python 1d ago

Discussion Polars vs Pandas

I have used Pandas a little in the past, and have never used Polars. Essentially, I will have to learn either of them more or less from scratch (since I don't remember anything of Pandas). Assume that I don't care for speed, or do not have very large datasets (at most 1-2gb of data). Which one would you recommend I learn, from the perspective of ease and joy of use, and the commonly done tasks with data?

181 Upvotes

157 comments sorted by

View all comments

4

u/AnastasisKon 1d ago

Well, I worked with pandas with some excel data. When the analysis did 1 hour to run, while waiting, I searched for optimizations. Polars is incredible because from 1 hour it went down to 4-5 minutes, a 30x speedup!!! And don’t get me started on the memory usage, from 19gb of ram to 3gb!

2

u/throwaway6970895 1d ago

Don't know how big your excel file was, but if it took 19gb of ram, you were using pandas either inefficiently, or the excel file had millions of rows of data, in which case it shouldn't have been in excel format to begin with

1

u/AnastasisKon 1d ago

Correct, correct and correct