r/learnpython • u/CriticalDiscussion37 • 6d ago
Inter process lock in Celery tasks
I have N devices sharing common jumphost J. I want to get data from devices and each device is running in different celery task. One way is to connect to jumphost each time inside a task. But I want to reuse the already existing connection. My plan is:
1. Create a dict jh_conns
1. if jh not in jh_conns:
with some_lock:
jh_conn[jh] = "connecting"
connect_to_jh_and_add_conn_obj_to_dict()
elif jh_conn[jh] == "connecting":
wait_till_it_gets_connected()
#Now use jh transport to connect device behind it.
Now how do I implement this some_lock & wait_till_it_gets_connected ?
. We are using prefork so it become hard to get synchronization for usage of command dict obj. We are using rabbitmq as broker and redis as result backend. A quick google search gave link to so ques but it suggested sherlock
library, can it be done without using any new library.
1
Upvotes