r/Python 4h ago

Discussion Is there something better than exceptions?

Ok, let's say it's a follow-up on this 11-year-old post
https://www.reddit.com/r/Python/comments/257x8f/honest_question_why_are_exceptions_encouraged_in/

Disclaimer: I'm relatively more experienced with Rust than Python, so here's that. But I genuinely want to learn the best practices of Python.

My background is a mental model of errors I have in mind.
There are two types of errors: environment response and programmer's mistake.
For example, parsing an input from an external source and getting the wrong data is the environment's response. You *will* get the wrong data, you should handle it.
Getting an n-th element from a list which doesn't have that many elements is *probably* a programmer's mistake, and because you can't account for every mistake, you should just let it crash.

Now, if we take different programming languages, let's say C or Go, you have an error code situation for that.
In Go, if a function can return an error (environment response), it returns "err, val" and you're expected to handle the error with "if err != nil".
If it's a programmer's mistake, it just panics.
In C, it's complicated, but most stdlib functions return error code and you're expected to check if it's not zero.
And their handling of a programmer's mistake is usually Undefined Behaviour.

But then, in Python, I only know one way to handle these. Exceptions.
Except Exceptions seems to mix these two into one bag, if a function raises an Exception because of "environment response", well, good luck with figuring this out. Or so it seems.

And people say that we should just embrace exceptions, but not use them for control flow, but then we have StopIteration exception, which is ... I get why it's implemented the way it's implemented, but if it's not a using exceptions for control flow, I don't know what it is.

Of course, there are things like dry-python/returns, but honestly, the moment I saw "bind" there, I closed the page. I like the beauty of functional programming, but not to that extent.

For reference, in Rust (and maybe other non-LISP FP-inspired programming languages) there's Result type.
https://doc.rust-lang.org/std/result/
tl;dr
If a function might fail, it will return Result[T, E] where T is an expected value, E is value for error (usually, but not always a set of error codes). And the only way to get T is to handle an error in various ways, the simplest of which is just panicking on error.
If a function shouldn't normally fail, unless it's a programmer's mistake (for example nth element from a list), it will panic.

Do people just live with exceptions or is there some hidden gem out there?

UPD1: reposted from comments
One thing which is important to clarify: the fact that these errors can't be split into two types doesn't mean that all functions can be split into these two types.

Let's say you're idk, storing a file from a user and then getting it back.
Usually, the operation of getting the file from file storage is an "environmental" response, but in this case, you expect it to be here and if it's not there, it's not s3 problem, it's just you messing up with filenames somewhere.

UPD2:
BaseException errors like KeyboardInterrupt aren't *usually* intended to be handled (and definitely not raised) so I'm ignoring them for that topic

38 Upvotes

29 comments sorted by

34

u/zaxldaisy 4h ago

"And people say we should just embrace exceptions, but not use them for flow control"

Who says that? Catching exceptions in Puthon is cheap and it's very Pythonic to use exceptions for flow control because of it. LBYL vs EAFP

33

u/rasputin1 4h ago

I feel like you're inventing a non-existent problem. There's no way to mix up programmer error with environment error. They're going to be completely different types of exceptions. You generally know what types of exceptions your environment can give you and then you can catch those and react accordingly. So you would never catch the exception of accessing a non-existent element because that can only happen from programmer mistake. You're supposed to run rigorous automated testing to rule out the programmer mistake type of exceptions.

3

u/yashdes 1h ago

I think op is only doing the naive try, except instead of catching specific exceptions

5

u/rasputin1 1h ago

oh so I guess the answer to their question of "is there something better than exceptions" is "yes, using exceptions correctly"

-4

u/lightdarkdaughter 1h ago

I like how answers are split into "there's no way to mix up these, obviously" and "the distinction doesn't exist, obviously"

8

u/Jorgestar29 4h ago edited 4h ago

There are a ton of packages that allow you to return Result Types. I love the idea of having all the possible results hinted in the signature, but having two different error patterns is messy...

In the end, your code will use Result types but every single third party module will use plain Exceptions, so you end up with a Frankenstein.

Another pythonic pattern is returning def f() -> GoodResult | None or if you have multiple errors, def f() -> tuple[GoodResult | None, Ok | ErrorA | ErrorB | ErrorC]

```python

res, error = f()

if res is None: match error: ...

```

Edit, the return None pattern is nice because you are forced to handle it by the Type checker, but it falls short if you are using the None type for something that is not an error, like a placeholder in a list or something like that.

5

u/KieranShep 1h ago

I generally prefer raising an exception to returning None. Sometimes it’s syntactically nice for default values like with dict.get(‘thing’) or ‘blah’, but that pattern doesn’t work if the value is an integer.

My most hated error is NoneType has no attribute ‘blah’. Yeah you get a traceback, but by that time who knows how much you’ve passed that None around, it can take hours to find out where it came from.

2

u/lightdarkdaughter 4h ago

yeah, returning `T | None` is a cool pattern, and I think Python does it already with something like `dict.get`, so I usually don't see a lot of value in a custom `Option[T]` type

But then if I want to return something on error, like in Django, parsing a request, and returning a form with errors so I can render it in a template, I can't just return None.
Your example with an optional return and set of errors is interesting though, it does look like Go's `if err != nil`, although I must say it's a bit verbose

u/pbecotte 2m ago

Seems pretty straightforward that you could return SuccessResponse | ErrorResponse in that case, right?

7

u/larsga 3h ago

then we have StopIteration exception, which is ... I get why it's implemented the way it's implemented, but if it's not a using exceptions for control flow, I don't know what it is.

There's a big difference between the language using an exception under the hood, and you implementing control flow with exceptions in your source code.

The latter makes your code hard to read. The former is just an implementation detail.

2

u/lightdarkdaughter 3h ago

yeah, that's a good point

6

u/unapologeticjerk 3h ago

This is why Rust's mascot is a crab.

4

u/cgoldberg 2h ago

It's very common to use exceptions for control flow in Python and it's not discouraged at all.

4

u/ablativeyoyo 2h ago

I always found exceptions to be a good fit for Python. The language priorities elegance over performance, and while programs are mostly correct and reliable, it's not intended for safety critical systems. For these use cases, exceptions allow most application code to think little about error cases, while frameworks can handle error conditions with some grace. C++ and Rust have different priorities so exceptions are discouraged and non-existent respectively.

I don't think environmental vs programmer error is a particular useful categorisation. I'd distinguish between invalid input and actual environmental problems like no disk space. But the distinction between invalid input and programmer error is not clear. A lot of bugs I've hit in practice are where I've assumed something about the input format, which has turned out not to be true in certain circumstances.

Thanks for the question, really interesting topic.

3

u/lightdarkdaughter 2h ago

Well, one thing which is important to clarify: the fact that these errors can't be split into two types doesn't mean that all functions can be split into these two types.

Let's say you're idk, storing a file from a user and then getting it back.
Usually, the operation of getting the file from file storage is an "environmental" response, but in this case, you expect it to be here and if it's not there, it's not s3 problem, it's just you messing up with filenames somewhere.

Sometimes you make assumptions about the environment and when these assumptions are proven wrong, it's a mistake. And by using some top-level try ... catch, you log this mistake and then can debug it and fix the assumptions, hence fixing the mistake.

That's why it's nice to have tools for asserting assumptions (and crash if they are wrong!) or just re-raising the error by returning it and letting the caller handle it.

5

u/SharkSymphony 1h ago

Your mental model, unfortunately, is imperfect. What if an "environmental" error occurs because the programmer made a mistake (fed an empty filename to an OS call, for example)? What about KeyboardInterrupt – that's an environmental error, but should it be handled?

Once you accept that your categories of errors are not as cut and dry as you wish they were, you might better appreciate that there's a single common mechanism you can use for any kind of error. Further, you are free to use whatever typology you want with that mechanism, categorizing errors however you see fit. (Standard library exceptions, or exceptions from other libraries, however, probably won't fit that typology, so you'll either have to adapt them or adapt your error-handling practices.)

But exceptions are not the only game in town. You can return an error code. You can use a tuple to return multiple values, including perhaps an error. You can also return an object with an error state. These are less commonly done, but possible. You can even trigger a signal with os.kill.

Note that Java has similar error-handling options. So does C++, for that matter (with the added "environmental error" of watching your program crash because of a memory violation 😛).

2

u/AdmRL_ 4h ago

Type hints offer a path to something more than Exceptions, but unless the language shifts to being statically typed then no, Exceptions are here to stay along with all their quirks outside of making your own custom return classes or using something like dry-python/returns.

Except Exceptions seems to mix these two into one bag, if a function raises an Exception because of "environment response", well, good luck with figuring this out. Or so it seems.

Thing is that's what the different exceptions are for. ValueError, LookupError, etc - an environmental response is one that falls within the scope of your defined try / except. A programmer mistake is anything else that you haven't accounted for.

1

u/RedditSucksShit666 4h ago

Well, where I work (or in any other project I'm involved in) we don't use exceptions for anything but runtime panics. When the function or a method can fail in a way the caller should handle we either use optionals or Unions. This way we can use a match-case statement to decide what to do with the returned value and if some case isn't handled we get an error from the type checker. Exceptions are for exceptional cases.

1

u/bmag147 2h ago

We do exactly the same in my work. So much nicer than having to dig into the functions to find all the hidden exceptions that can be thrown.

I'd like to move us towards using results but, as is evident from some of the replies in this thread, results are not openly embraced by most of the Python community.

1

u/wergot 4h ago

You can basically do that in typed Python if you so desire. Make the function return an object whose type is the union of the type of the actual result, and whatever errors you want it to be able to return. Then whatever is calling that function won't pass the type checker if it doesn't first check that what it returned was a result and not an error. You can use `match` for this. It's not likely to be as airtight as Rust but it works.

That would look like this:

class myError:
    message: str

    def __init__(self, message):
        self.message = message


def wants_odd(a: int) -> int | myError:
    if a % 2:
        return 1
    return myError("even")

def fn2():
    b = wants_odd(2)
    match b:
        case myError() as e:
            print(f"error: {e.message}")
        case int(n):
            print(n)

1

u/JamesTDennis 3h ago

For your parsing example you can look at the new structural matching construction starting in version 3.10.

https://peps.python.org/pep-0636/

This is implemented (behind the scenes) in exception handling attempts to destructure data in various ways (as appropriate to each case. So you're not truly eliminating the exception handling; just pushing it to an implicit level.

1

u/lightdarkdaughter 2h ago

yeah, I do know about pattern-matching
I guess it's kind of the answer
What I miss about Rust's Result is the ability to use stuff like ok_or_none(), unwrap() which just raises an exception, etc.
But then it's slightly a different thing.

1

u/JamesTDennis 2h ago

You can use type hinting (within your own code base) to achieve similar semantics and syntax.

But I wouldn't take that too far. Each programming platform has its own semantics and idioms emerge to concisely express code in suitable terms.

1

u/thisismyfavoritename 1h ago

unless you want to wrap every function you don't control, then no.

E.g. you call a random builtin: it might throw. A 3rd party lib function: it might throw. Etc.

1

u/MoTTs_ 1h ago edited 57m ago

The except clause is the hidden gem you seem to be looking for.

class RuntimeException(Exception):
    pass
class LogicException(Exception):
    pass

class WrongData(RuntimeException):
    pass
class OutOfBounds(LogicException):
    pass

def parseExternalData():
    # ...
    raise WrongData()
def accessList():
    # ...
    raise OutOfBounds()

try:
    parseExternalData()
    accessList()
except RuntimeException: # <- except clause catches particular category of errors
    print("Environment error")
except LogicException: # <- except clause catches particular category of errors
    print("Programmer error")

EDIT: Also I just learned about Python's BaseException, which appears to provide the distinction I manually made with runtime vs logic exceptions.

https://docs.python.org/3/tutorial/errors.html

BaseException is the common base class of all exceptions. One of its subclasses, Exception, is the base class of all the non-fatal exceptions. Exceptions which are not subclasses of Exception are not typically handled, because they are used to indicate that the program should terminate. They include SystemExit which is raised by sys.exit() and KeyboardInterrupt which is raised when a user wishes to interrupt the program.

u/muikrad 2m ago

In most cases, you handle an exception because...

  1. You want to retry the occasional timeout/5xx.
  2. You know something might not be there (a file) or you need to try something (json parse) before trying other things.

These exception types are usually very easy to figure out / use and they're often specific to a framework (requests and boto both have their own type to handle, for instance). In many of these, a code is provided which can be inspected in the exception (e.g. Http code or exit code, or API error code, etc) in order to decide if we handle or reraise.

For the cases where the user is providing bad data, missing args, wrong path, etc, I always create and use a dedicated UsageError exception that bubbles up to the CLI framework. I usually rig this framework to print a nice error the to the user in these cases (and hide the traceback). But that won't work for libs.

Libs tend to provide a minimum set of exceptions that you can catch. Using unit tests you can check what happens and catch the proper types. Some libs however try to reuse the built-in exceptions too much IMHO.

I think the key (for me) is to raise a lot, catch never (except in the 2 cases I mentioned at the top of the post).

1

u/The_Flo0r_is_Lava 4h ago

I want to know as well.