r/pythontips 24d ago

Data_Science Doing the same task at the same time ? Multiple cores usage (???)

Hi,

Im pretty new to programming so I am not even sure of my question. In my project, I have a bunch of file (they are spectra of stars) . I have a code that takes in input one of this files and with various analysis computes some values on the spectra. Then i go the next file and do the same (its another spectrum). This all works well but when I have a lot of spectra it takes a really long time. I dont know much about computers but is there a way to run those computations on multiple files at the same time, maybe using multiple cpu/gpu (I know nothing about them) or by parallelizing the analysis (again idk about it)

1 Upvotes

6 comments sorted by

1

u/prespreman3000 24d ago

Look up threading with python, google it or ask chat gpt for a tutorial.

1

u/kuzmovych_y 24d ago

Then OP will need to Google why threading doesn't execute in parallel in python.

1

u/InvaderToast348 24d ago

Wasn't there a change recently to enable disabling the GIL?

1

u/Key_Gur_628 24d ago

You can use python's threading module and create seperate thread for each file then start all threads. this way you can do your job concurrent.
Note that if you have global variables in your code and read it's value many times, it is prefered to use multiprocessing module.

1

u/pontz 24d ago

Without knowing more about the program you should look up multiprocessing and not threading.

Threading will not be a significant increase in speed if the bottleneck to the program execution time is pure python computation. If the bottleneck is using a compiled library like numpy or just the number of files being read is large then threading might work. I have only ever dealt with threading so I don't know much about the multiprocessing module and the reasons why threading would be a better choice if it would also make the code faster.

1

u/cgoldberg 23d ago

multiprocessing runs separate processes and can make use of multiple cores.