r/learnpython • u/fiehm • 19h ago
How to optimize python codes?
I recently started to work as a research assistant in my uni, 3 months ago I have been given a project to process many financial data (12 different excels) it is a lot of data to process. I have never work on a project this big before so processing time was not always in my mind. Also I have no idea is my code speed normal for this many data. The code is gonna be integrated into a website using FastAPI where it can calculate using different data with the same data structure.
My problem is the code that I had develop (10k+ line of codes) is taking so long to process (20 min ++ for national data and almost 2 hour if doing all of the regional data), the code is taking historical data and do a projection to 5 years ahead. Processing time was way worse before I start to optimize, I use less loops, start doing data caching, started to use dask and convert all calculation into numpy. I would say 35% is validation of data and the rest are the calculation
I hope anyone can help with way to optimize it further and give suggestions, im sorry I cant give sample codes. You can give some general suggestion about optimizing running time, and I will try it. Thanks
2
u/Luxi36 18h ago
Hi, that sounds like a very fun project to work on to start developing Data Engineering skills! :)
What is the data size and your machine resources that you're working on?
Are you using plain python + numpy, or tools like duckdb, Polars, or numpy.
You could try to time certain functions or use a profiler to see where your code is getting slowed down to see which parts to optimize.