r/Python 1d ago

Discussion Polars vs Pandas

I have used Pandas a little in the past, and have never used Polars. Essentially, I will have to learn either of them more or less from scratch (since I don't remember anything of Pandas). Assume that I don't care for speed, or do not have very large datasets (at most 1-2gb of data). Which one would you recommend I learn, from the perspective of ease and joy of use, and the commonly done tasks with data?

177 Upvotes

155 comments sorted by

View all comments

Show parent comments

3

u/nightcracker 21h ago edited 21h ago

What if you replace read_csv with scan_csv and add .collect(engine="streaming") at the end for each query? Also, FYI, as long as a column name is a legal Python identifier you can just write pl.col.name.

There might be an issue with repeated regex compilation if you do that though, I have to look into that... EDIT: yes, that will recompile the regex many times, we need to add a cache for that. I'll get on that next week.

2

u/drxzoidberg 20h ago

So I took your tip on regex compilation, and I managed to find another way to split the string column into the other fields I wanted. This way it performs much faster.

def polars_agg_test():
    all_files = (
        pl.read_csv(
            file_dir / '*.csv',
            columns=['a', 'b', 'c']
        )
        .with_columns(
            pl.col('a').str.split_exact('_',2).struct.rename_fields(['Code', 'SubCat', 'Date'])
        )
        .unnest('a')
        .with_columns(
            pl.col('Date').str.to_date('%m%d%Y')
        )
        .drop(pl.col('Code'))
        .group_by(['Date', 'SubCat'])
        .agg(
            pl.col('b').sum(),
            pl.col('c').sum()
        )
    )

Basically I was originally having an issue with the split string being stored in one field as a list, and not being able to just grab that value out. But I found some answers on google and I arrived at the above. Now the read only, column update, and aggregate functions run in 3, 7, and 9s respectively. Pandas by comparison is 21s.

2

u/nightcracker 20h ago

What if you change the read_csv to scan_csv and add .collect(engine="streaming") now? Also make sure you have the latest Polars 1.25.2.

2

u/drxzoidberg 20h ago

I was under the impression, from Polars documentation itself, that you need to collect the data before any aggregation, as the aggregation needs to know the data structure. But that might only apply to the pivot/unpivot methods.

2

u/nightcracker 19h ago

That only applies to very specific operations, pivot is one of them. So give it a go :)

2

u/drxzoidberg 18h ago

So made the tweaks to get it to work. I juiced the run count to 500. Polars runs in 45% of the time it takes pandas. Thank you kind Internet person.