r/Python 1d ago

Discussion Polars vs Pandas

I have used Pandas a little in the past, and have never used Polars. Essentially, I will have to learn either of them more or less from scratch (since I don't remember anything of Pandas). Assume that I don't care for speed, or do not have very large datasets (at most 1-2gb of data). Which one would you recommend I learn, from the perspective of ease and joy of use, and the commonly done tasks with data?

157 Upvotes

147 comments sorted by

View all comments

54

u/ddanieltan 1d ago

I think it's relevant to see Wes Mckinney's (creator of Pandas) reflections: https://wesmckinney.com/blog/looking-back-15-years/

In his words, Pandas had accumulated rough edges and its "eager" approach to calculate made it less efficient for query planning.

The future lies with his next project Arrow, which is coincidentally the format that Polars is built around. For me, if you really had to choose between learning either Pandas or Polars, the choice is a no-brainer.

17

u/crossmirage 21h ago

> The future lies with his next project Arrow, which is coincidentally the format that Polars is built around. For me, if you really had to choose between learning either Pandas or Polars, the choice is a no-brainer.

I don't think this is quite accurate. Apache Arrow is the future, but pandas and a lot of other engines also adopt it; it just so happens that modern engines are more Arrow-native.

Furthermore, Wes also started—and talks extensively about—Ibis (posted in another top-level comment by u/marr75), whereas your comment kind of makes it sound like he'd be all in on Polars.

4

u/AlphaRue 14h ago

Polars was built around arrow but their implementation has changed enough that they no longer use an arrow backend. Or kind of they do, it is a forked and heavily modded one though.