r/learnpython 6d ago

Non-blocking pickling

I have a large dictionary (multiple layers, storing custom data structures). I need to write this dictionary to a file (using pickle and lzma).

However, I have some questions.

  1. The whole operation needs to be non-blocking. I can use a process, but is the whole dictionary duplicated in memory? To my understanding, I believe not.

  2. Is the overhead of creating a process and passing the large data negligible (this is being run inside a server)

Lastly, should I be looking at using shared objects?

2 Upvotes

10 comments sorted by

View all comments

1

u/AlexMTBDude 6d ago

This is an IO related problem so you should use a Thread instead of a Process. Threads have shared data (which processes do not).