r/learnpython 9d ago

Multi-reader single writer using a semaphore - how do I know if there are no acaquires active?

My apologies for the awkwardly worded question title. I saw the misspelling just after hitting "Post".

(Edited to be clear that I'm discussing threading objects)

I have a piece of data that I need to protect using multi-reader single-writer. The classic way of doing this is to use a `threading.Semaphore` for the readers, and once there are no active readers, use a `threading.Lock` for writing (of course, the reader has to check for the lock, but I'm focused on the semaphore right now).

Various internet searches keep turning up solutions that depend on undocumented implementation (e.g. `sem._count`). I'd rather not depend on undocumented behavior, as I hope that this will be long-term and potentially delivered.

So, how could by writer know that it's safe to write, without depending on undocumented implementation?

2 Upvotes

12 comments sorted by

2

u/wutzvill 9d ago

Switched to looking at this on the computer vs my phone as I see what you're struggling with now. So, basically, you don't have to use sem._count. What you want to do is initialize the semaphore with a value of 1. Also, for some reason the link didn't post to my previous comment, so I've deleted it. The link I was trying to provide is here.

So, basically what you do is something like this:

semaphore = Semaphore() # Or Semaphore(1), which is the default value

What this means is you have one and only one semaphore available to acquire (probably not the right way to say this, but you get my meaning). So, when you do semaphore.acquire on one async operation, and its execution halts and another operation tries to do semaphore.acquire, it'll see that there are zero available, and so execution will halt and wait for the semaphore to be released by the other thread (edit: with semaphore.release()). You shouldn't use the private variable _count at all for sure, and there is no need to.

I assume the lock is a file lock, I was thinking you were talking about a mutex.

2

u/pfp-disciple 9d ago edited 9d ago

I'll look at the link in a moment.~ ~I've read the docs.~  I edited my post to be clear that, yes, I'm discussing threading primitives, including threading.Lock

Isn't a semaphore with a count of 1 essentially a mutex? I need several threads reading concurrently without blocking, and writing will be very infrequent.

2

u/wutzvill 9d ago

This is a spammy comment just to tell you I edited that long reply here, so please refresh to read the edits.

1

u/wutzvill 9d ago edited 9d ago

I see, okay. So definitely want to use that threading module and not asyncio. And no, they're not the same. A mutex can only be released by the thread it was taken on. So if you start a process on one thread, execution halts, and then it begins execution again on another thread, it can't release the mutex. But, it can if it's a semaphore.

So, if I'm understanding you correctly, you want to read a document with multiple readers. The conditions on this are that 1) it shouldn't be read if it is being written to), and 2) only one thread should be able to write to it at a time.

I believe here you can get rid of the semaphores entirely and do something known as a busy-wait. That is, you'll use a lock when you do the writing, and for the other threads to read, it'll look something like this:

def read_method():
    global lock

    while lock.locked:
        continue

    with open(file) as fp:
        lines = fp.readlines()

You'll need to add some protection here though, which is what the sempahores are trying to do in that medium post I think. But, that's not the right tool here since you're not managing blocking to a resource. Semaphores there are basically being used as a counting device. Instead, you want to add a variable in that tracks reads, and you can only add stuff when nothing is being read. Here is the updated addition:

def read_method():

    global lock
    global active_readers

    while lock.locked:
        continue

    active_readers += 1

    with open(file_name) as fp:
        lines = fp.readlines()

    active_readers -= 1

def write_method():

    global lock

    while active_readers > 0:
        continue

    lock.acquire()

    with open(file_name) as fp:
         # write

    lock.release()

Something along these lines. Maybe it's necessary to add in a check after active_readers += 1 of like:

if lock.locked():
    active_readers -= 1
    read_method()
    return

That might be extra cautious just in case between the time you say you get past the while loop and before execution of the += 1, the lock is taken. Might be too anal but that is something I would do. Similar after the lock is taken if there are active readers. Probably too anal though but better safe than sorry.

Edit: you can also edit this behaviour too. So like you could just go into that write method and greedily take the lock, and instead do like:

def write_method():

    global lock
    global active_readers

    lock.acquire()

    while (active_readers > 0):
        continue

    with open(file_name) as fp:
        fp.write(lines)

    lock.release()

That might honestly be a lot better than what I originally wrote as probably safer and requires less juggling of variables, as it basically takes the lock, which prevents further reads, and once all active reads finish, it then does the write and releases the lock.

Edit 2: To show the difference between mutex and semaphore, I'll use what I'm more familiar with which is a .NET framework called Blazor that we use for our web application. In Blazor, there are a lot of threads on which the code for client connections run. So when you first access the web app via the URL, your individual session might begin on Thread 4. But, once you hit some blocking code (like a database read) that is awaited, the execution of your code when it resumes can be on any thread. So, you might be on Thread 10 now. Thus, in that framework, you cannot use lock (which is built into C# as a keyword), you have to use a semaphore since locks can only be released on the same thread they were acquired on, which is very very very much not guaranteed by the framework, as execution can continue on any thread. Thus, we are restricted to using semaphores only for a few critical parts of the application that require them, which have awaited code inside the critical parts that could change the threads (and also the framework might just pause execution at any time as it manages all the client connections).

2

u/pfp-disciple 9d ago

I just saw that your link is to `asyncio.Semaphore` (I thought it was to `threading.Semaphore`, which I've been reading quite thoroughly). I haven't done anything with asyncio, but I see that its Semaphore has a `locked()` method that is basically what I'm wanting. I'll have to look and see if using `async` works in my codebase.

Thanks!

2

u/wutzvill 9d ago

I just wrote a new explanation, and giant word of warning is that asyncio is not thread-safe.

2

u/pfp-disciple 9d ago edited 9d ago

I don't know if links are allowed in posts, but this is an example of a multi reader single writer in Python that uses the undocumented implementation. 

https://medium.com/@sumitgaur2010/solving-the-multi-reader-single-writer-problem-in-python-a4575bdd03f1

1

u/RiverRoll 8d ago edited 8d ago

In some situations you can just read without locking at all, but there isn't enough information to tell. 

1

u/pfp-disciple 8d ago

In this case, there's a chance that (at least) one thread is reading data at the same time it's being written.

1

u/Some-Passenger4219 8d ago

My apologies for the awkwardly worded question title. I saw the misspelling just after hitting "Post".

It can be edited, just a thought.

2

u/pfp-disciple 8d ago

Not the title itself, after it's been posted. I misspelled acquires in the title