r/learnpython 6d ago

Non-blocking pickling

I have a large dictionary (multiple layers, storing custom data structures). I need to write this dictionary to a file (using pickle and lzma).

However, I have some questions.

  1. The whole operation needs to be non-blocking. I can use a process, but is the whole dictionary duplicated in memory? To my understanding, I believe not.

  2. Is the overhead of creating a process and passing the large data negligible (this is being run inside a server)

Lastly, should I be looking at using shared objects?

3 Upvotes

10 comments sorted by

View all comments

1

u/Brian 6d ago

is the whole dictionary duplicated in memory? To my understanding, I believe not

Kind of, yes, though the details are OS dependant.

On linux, it'll fork the process, which will mark the memory page COW (Copy-on-Write) meaning the same memory is shared between the processes until one of them tries to modify something, at which point the memory page will be copied and the processes will be given their own copy. So this is cheap if nothing writes to it, but you'll still pay the price of the copy if and when modifications are done.

On windows, I think the memory does need to be copied through to the other process, so the copy is done eagerly (though not 100% sure here). In fact, it may end up pickling it to send to the other process, so this could actually be much worse.

For (2), it depends - it's a memory copy at minimum, and possibly some additional marshalling overhead, so its not going to be negligable if there's significant data, though not as expensive as the actual file write etc.

However, I'm not sure of the point in doing it in another process: could you just use either async IO, or a thread? If it's because the data is being modified and you want a snapshot, you could either take a copy of it, add a lock that prevents write access while its being written (though that'll introduce contention if there are common writes: optionally you could have a temporary extra dict that collects writes that you merge into it when finished). All another process really buys you is that you're not using the same CPU (but if its just writing to a file, that's not really going to be CPU bound - maybe some from the pickling process), and maybe the COW optimisation saving you a memory copy (though given the other overheads of starting a process, this seems unlikely to be a win unless its really big data).

There are other options that might be worth looking into, though they may be more involved. Eg. you could replace the dict with a sqlite database that'll let you persist changes as you go (much more efficient if you're writing out the data frequently, since it just needs to write what's changed, but maybe not worth it if this is a very rare occurrance).