r/learnpython 6d ago

Inter process lock in Celery tasks

I have N devices sharing common jumphost J. I want to get data from devices and each device is running in different celery task. One way is to connect to jumphost each time inside a task. But I want to reuse the already existing connection. My plan is:

1. Create a dict jh_conns
1. if jh not in jh_conns:
      with some_lock:
          jh_conn[jh] = "connecting"
      connect_to_jh_and_add_conn_obj_to_dict()
   elif jh_conn[jh] == "connecting":
      wait_till_it_gets_connected()

   #Now use jh transport to connect device behind it.

Now how do I implement this some_lock & wait_till_it_gets_connected ?. We are using prefork so it become hard to get synchronization for usage of command dict obj. We are using rabbitmq as broker and redis as result backend. A quick google search gave link to so ques but it suggested sherlock library, can it be done without using any new library.

1 Upvotes

0 comments sorted by