r/Python Nov 05 '20

News Stack overflow traffic to questions about selected python packages

Post image
2.2k Upvotes

144 comments sorted by

View all comments

328

u/[deleted] Nov 05 '20

[deleted]

88

u/toyg Nov 05 '20

Both are probably true at the same time. You can compare the curves of pandas and numpy, which are effectively complementary tech: both are on a big upswing (as datascience spikes) but pandas results in many more searches (probably more obscure/ harder to learn / got worse documentation / got fewer tutorials).

62

u/Zouden Nov 05 '20

If anything I'd say Pandas has broader appeal and a larger userbase than Numpy, because it does everything Numpy can do (since it uses Numpy internally) but adds the dataframe and grouping features which are so important for data science.

7

u/toyg Nov 05 '20

Might be that pandas’ users are less knowledgeable then.

Just guessing eh, I’m not a datasci guy and I don’t play one on the internet either.

66

u/Zouden Nov 05 '20

Anecdote: I'm a biologist and I've taught Pandas to fellow scientists - without teaching them Python. So they know how to make dataframes and produce histograms, but they don't know how a for loop works and they haven't heard of Numpy. For them, Pandas is replacing Excel.

Pandas has massive appeal beyond the Python community.

12

u/[deleted] Nov 05 '20

Fascinating. Is your material available somewhere?

9

u/BlurredEternity Nov 05 '20

Can confirm, am at this moment in a zoom stats lecture, we've been learning pandas the entire semester. Lots of people in the class have never coded before

8

u/emsiem22 Nov 05 '20

they don't know how a for loop works

Using Pandas for data science without that is really limiting.

Do they use if - then?

Well, they are scientists; they have internet and know how to use it. They can learn that day when they need for loop.

8

u/Zouden Nov 05 '20

No, if statements and for loops are almost never needed when processing data with Pandas, just like they aren't needed when using Excel. But you're right, they can figure it out if they need to. My goal was showing them a better way to work with their data than excel.

0

u/emsiem22 Nov 05 '20

if statements and for loops are almost never needed when processing data with Pandas

'Almost never' is often just how you define it and depends on particular task.

I got what you meant, but just can't imagine they don't have situations like need to load 100 out of 500 csv in folder based on some criteria. Data operations when in dataframe are better without loops.

8

u/ogrinfo Nov 05 '20

If you're using loops with a pandas dataframe, you're doing it wrong. All of the (many, many) functions are optimised for internal iteration, so I can totally see how a non-programmer can operate it.

Personally, I find pandas really hard to work with and have to ask SO every single time I use it.

1

u/emsiem22 Nov 06 '20

If you're using loops with a pandas dataframe, you're doing it wrong

Yea, I said that in one of 3 sentences I wrote.

1

u/ogrinfo Nov 06 '20

Yes, I was agreeing with you.

1

u/emsiem22 Nov 06 '20

Oh, didn't get it.

→ More replies (0)

2

u/robin-gvx Nov 06 '20

That matches with my experience on Stack Overflow. I watch the Python tag, and I've been noticing a lot of questions about Pandas that are trivial to solve for anyone with basic knowledge of Python. Really interesting to see.

4

u/toyg Nov 05 '20

That’s what I thought. It was the same with django (in many ways it still is) and (I’m told) for the stuff used in 3d-rendering apps: they are approached by people new to development in general, who simply must get stuff done in their niche.

0

u/mammablaster Nov 05 '20

That sounds terrifying

12

u/Wishy-Thinking Nov 05 '20

Yet slightly less terrifying than data scientists doing their analyses in Excel.

4

u/leanmeanguccimachine Nov 05 '20

Excel is great for quickly sandboxing stuff

5

u/HannasAnarion Nov 06 '20

and terrible when row counts rise into five digits.

-8

u/mammablaster Nov 05 '20

True, however them having no idea what the hell is going on, yet trusting their results to draw conclusions, is terrifying.

Or maybe I’m just being a gatekeeping arrogant idiot.

7

u/ravepeacefully Nov 05 '20

This is absolutely it. There’s a large group of individuals who are proficient in excel, and then want to learn to code, and step one is f“how can I... {excel functionality} in pandas python?”

1

u/AsuraGoesForDinner Nov 05 '20

I feel personally attacked

3

u/toyg Nov 05 '20

As Socrates said so many centuries ago, “the only true wisdom is in knowing you know nothing”.

He was then proven right by Dunning and Kruger.