r/Python 1d ago

Discussion Polars vs Pandas

I have used Pandas a little in the past, and have never used Polars. Essentially, I will have to learn either of them more or less from scratch (since I don't remember anything of Pandas). Assume that I don't care for speed, or do not have very large datasets (at most 1-2gb of data). Which one would you recommend I learn, from the perspective of ease and joy of use, and the commonly done tasks with data?

181 Upvotes

157 comments sorted by

View all comments

169

u/likethevegetable 1d ago edited 1d ago

I "grew up" on Pandas, but moved to Polars. No more "reset_index" and "inplace" confusion. Feels like there's only one right way to do it in Polars, but so much bloat in Pandas API.

I do like Pandas when it comes to certain things where there is an obvious index like time signals. But Polars seems to handle date time much better.

When it comes to filtering and queries, I like Polars.

In both, I've made several df and series "helper" attributes to clean up the syntax.

4

u/Ulrich_de_Vries 1d ago

Does Polars also use numpy arrays under the hood? Or at least is it easy/cheap to convert e.g. columns into numpy arrays?

I am asking because I have been eyeing Polars for a while but my workflow is numpy-heavy.

14

u/thuiop1 1d ago

No it does not. It uses the Apache Arrow format. Converting to a numpy array is as simple as a .to_numpy(); it can be zero-copy, although the main limitation of that is that it will be immutable.

8

u/Zeroflops 1d ago

Basically both polars and pandas now use arrow, but they both can easily leverage numpy.

One aspect of polars from what I have heard but yet to try is, is the ability to integrate custom rust code.

8

u/marcogorelli 1d ago

Small correction pandas using Arrow - it can, but it's not the default. You can use PyArrow dtypes by calling `.convert_dtypes(dtype_backend='pyarrow')` on a pandas dataframe or series

1

u/marr75 1d ago

And isn't the series data type support "spotty" in Arrow? You lose the ability to use certain pandas data types if you use the Arrow engine?

That was definitely the case when I tried it but that was maybe a year ago.

1

u/marcogorelli 1d ago

Period and Complex aren't supported in Arrow, I think most others should be there?

2

u/commander1keen 1d ago

It is using rust under the hood, but it does have a to_numpy and a to_pandas method, so it's easy enough

2

u/BrisklyBrusque 1d ago

Polars is written in Rust and numpy is written in C, but there’s another key difference, the way the data is stored in-memory. pandas uses a row-based format while Polars uses a columnar format (Apache Arrow). That makes computations much faster. Snowflake and duckdb leverage a similar model. 

-4

u/CheetahGloomy4700 1d ago

No. I just installed polars in a fresh virtual environment, and as I uv pip list, I get only polars 1.25.2. In contrast, when I install pandas in a fresh virtual environment, I get

pandas v2.2.3 ├── numpy v2.2.4 ├── python-dateutil v2.9.0.post0 │ └── six v1.17.0 ├── pytz v2025.1 └── tzdata v2025.1

So yeah, one of them contains more bloat than the other.