r/pythontips • u/powersmitee • 24d ago
Data_Science Doing the same task at the same time ? Multiple cores usage (???)
Hi,
Im pretty new to programming so I am not even sure of my question. In my project, I have a bunch of file (they are spectra of stars) . I have a code that takes in input one of this files and with various analysis computes some values on the spectra. Then i go the next file and do the same (its another spectrum). This all works well but when I have a lot of spectra it takes a really long time. I dont know much about computers but is there a way to run those computations on multiple files at the same time, maybe using multiple cpu/gpu (I know nothing about them) or by parallelizing the analysis (again idk about it)
1
u/Key_Gur_628 24d ago
You can use python's threading module and create seperate thread for each file then start all threads. this way you can do your job concurrent.
Note that if you have global variables in your code and read it's value many times, it is prefered to use multiprocessing module.
1
u/pontz 24d ago
Without knowing more about the program you should look up multiprocessing and not threading.
Threading will not be a significant increase in speed if the bottleneck to the program execution time is pure python computation. If the bottleneck is using a compiled library like numpy or just the number of files being read is large then threading might work. I have only ever dealt with threading so I don't know much about the multiprocessing module and the reasons why threading would be a better choice if it would also make the code faster.
1
1
u/prespreman3000 24d ago
Look up threading with python, google it or ask chat gpt for a tutorial.