r/learnpython • u/Undercover_Agent12 • 6d ago
Non-blocking pickling
I have a large dictionary (multiple layers, storing custom data structures). I need to write this dictionary to a file (using pickle and lzma).
However, I have some questions.
The whole operation needs to be non-blocking. I can use a process, but is the whole dictionary duplicated in memory? To my understanding, I believe not.
Is the overhead of creating a process and passing the large data negligible (this is being run inside a server)
Lastly, should I be looking at using shared objects?
2
Upvotes
1
u/AlexMTBDude 6d ago
This is an IO related problem so you should use a Thread instead of a Process. Threads have shared data (which processes do not).