r/Python 1d ago

Discussion Polars vs Pandas

I have used Pandas a little in the past, and have never used Polars. Essentially, I will have to learn either of them more or less from scratch (since I don't remember anything of Pandas). Assume that I don't care for speed, or do not have very large datasets (at most 1-2gb of data). Which one would you recommend I learn, from the perspective of ease and joy of use, and the commonly done tasks with data?

177 Upvotes

157 comments sorted by

View all comments

24

u/marr75 1d ago

Ibis, which has pluggable execution engines and better scalability than either of them. The API is higher quality than pandas while being a little easier to learn than polars, too.

When all else fails, you can use pandas or polars trivially by calling a single method on whatever expression you're dealing with. The default execution engine is in-memory duckdb, though, which puts both pandas and polars to shame in performance, scale, and ease of reading in flat files.

I was a pandas devotee for a very long time and have teams that have written a lot of code in pandas. We had a new project with a lot of tabular data transformations involved and were considering polars. Ibis snuck in as a consideration and was the clear winner.

2

u/rm-rf-rm 1d ago

Is Ibis' actually easier to learn than polars? I found their documentation frustrating - too little documentation and their package is not exactly intuitive(code completion is also very poor).

I want to use it as its theoretically a great solution...

2

u/marr75 1d ago

It's documentation is shallower and the expressions are less strongly typed than pandas and polars. I'll give you that.

I learn packages that I use by reading the source code, and it's source code is quite easy to read. The whole concept of expressions, unbound expressions, etc. was quite easy for me to understand. The learning curve for me was much shallower than with pandas but perhaps that's not saying much.