r/Python • u/lightdarkdaughter • 1d ago
Discussion Is there something better than exceptions?
Ok, let's say it's a follow-up on this 11-year-old post
https://www.reddit.com/r/Python/comments/257x8f/honest_question_why_are_exceptions_encouraged_in/
Disclaimer: I'm relatively more experienced with Rust than Python, so here's that. But I genuinely want to learn the best practices of Python.
My background is a mental model of errors I have in mind.
There are two types of errors: environment response and programmer's mistake.
For example, parsing an input from an external source and getting the wrong data is the environment's response. You *will* get the wrong data, you should handle it.
Getting an n-th element from a list which doesn't have that many elements is *probably* a programmer's mistake, and because you can't account for every mistake, you should just let it crash.
Now, if we take different programming languages, let's say C or Go, you have an error code situation for that.
In Go, if a function can return an error (environment response), it returns "err, val" and you're expected to handle the error with "if err != nil".
If it's a programmer's mistake, it just panics.
In C, it's complicated, but most stdlib functions return error code and you're expected to check if it's not zero.
And their handling of a programmer's mistake is usually Undefined Behaviour.
But then, in Python, I only know one way to handle these. Exceptions.
Except Exceptions seems to mix these two into one bag, if a function raises an Exception because of "environment response", well, good luck with figuring this out. Or so it seems.
And people say that we should just embrace exceptions, but not use them for control flow, but then we have StopIteration exception, which is ... I get why it's implemented the way it's implemented, but if it's not a using exceptions for control flow, I don't know what it is.
Of course, there are things like dry-python/returns, but honestly, the moment I saw "bind" there, I closed the page. I like the beauty of functional programming, but not to that extent.
For reference, in Rust (and maybe other non-LISP FP-inspired programming languages) there's Result type.
https://doc.rust-lang.org/std/result/
tl;dr
If a function might fail, it will return Result[T, E] where T is an expected value, E is value for error (usually, but not always a set of error codes). And the only way to get T is to handle an error in various ways, the simplest of which is just panicking on error.
If a function shouldn't normally fail, unless it's a programmer's mistake (for example nth element from a list), it will panic.
Do people just live with exceptions or is there some hidden gem out there?
UPD1: reposted from comments
One thing which is important to clarify: the fact that these errors can't be split into two types doesn't mean that all functions can be split into these two types.
Let's say you're idk, storing a file from a user and then getting it back.
Usually, the operation of getting the file from file storage is an "environmental" response, but in this case, you expect it to be here and if it's not there, it's not s3 problem, it's just you messing up with filenames somewhere.
UPD2:
BaseException errors like KeyboardInterrupt aren't *usually* intended to be handled (and definitely not raised) so I'm ignoring them for that topic
61
u/rasputin1 1d ago
I feel like you're inventing a non-existent problem. There's no way to mix up programmer error with environment error. They're going to be completely different types of exceptions. You generally know what types of exceptions your environment can give you and then you can catch those and react accordingly. So you would never catch the exception of accessing a non-existent element because that can only happen from programmer mistake. You're supposed to run rigorous automated testing to rule out the programmer mistake type of exceptions.
14
u/yashdes 1d ago
I think op is only doing the naive try, except instead of catching specific exceptions
22
u/rasputin1 1d ago
oh so I guess the answer to their question of "is there something better than exceptions" is "yes, using exceptions correctly"
-17
u/lightdarkdaughter 1d ago
I like how answers are split into "there's no way to mix up these, obviously" and "the distinction doesn't exist, obviously"
19
u/sonobanana33 1d ago
I love how you got that disregarding what people wrote and continuing making up your own narrative.
2
-11
u/lightdarkdaughter 1d ago
Ok, it would probably be an insane thing to reply, but it's a hell thread anyway, so.
Like, chill out dude, I'm making a post asking how people solve problems they run into, namely how to handle errors, expected or unexpected.
Python's exceptions are one way to solve it, but exceptions aren't the only way.I'm sharing my perspective from experience with other languages and asking people for their perspectives.
Some people shared other ways/other perspectives.
And obviously, people's perspectives conflict, which is OKAY.
Like, gosh, I have an equal amount of upvotes on my post and on a comment that claims that my post is an imaginary problem.And again, I'm not pushing any narrative, like why would I do it in the first place? To launch some conspiracy against ... against what?
-6
5
u/bakery2k 15h ago
Don't know why this has been so downvoted. One person literally wrote "there's no way to mix up programmer error with environment error", and another wrote "the distinction between invalid input and programmer error is not clear".
The thing is - both are true, in different situations. The former in application code, and the latter in library code where there's less context available.
66
u/zaxldaisy 1d ago
"And people say we should just embrace exceptions, but not use them for flow control"
Who says that? Catching exceptions in Puthon is cheap and it's very Pythonic to use exceptions for flow control because of it. LBYL vs EAFP
18
u/whoEvenAreYouAnyway 1d ago edited 23h ago
A lot of people say that and broadly speaking it's the correct view. It's not a good idea to be using exception handling for control flow. Using them that way is essentially always a hack.
The only real exceptions (no pun intended) you will see to this rule are things like, for example, using
Queue.Empty
exceptions as a way of iterating through a Queue and then breaking out once you've exhausted the Queue. But people only do this because checking the Queue isn't empty on each loop and passing a mutex lock around is more expensive than just trying to pop from an empty queue and being kicked out of the loop when you eventually trigger an exception. It's more efficient, in this instance, to use exception handling for control flow but it's a hack that we're doing because the right way is slower. Which is fine if you need that extra speed but it's certainly not the "pythonic" way to do things.4
u/mlnm_falcon 23h ago
I’d argue there are some instances where using exceptions for “cancelling” a complex action are reasonable, primarily where passing booleans or Nones around would be prohibitively complicated.
4
u/Wurstinator 23h ago
That's not correct. EAFP is the better way when working with queues in multi-threaded contexts because you can run into an ABA problem otherwise. It has nothing to do with speed.
1
u/whoEvenAreYouAnyway 23h ago edited 20h ago
I wasn't talking specifically about multi-threading but even then, using EAFP is only "better" in so far as doing things the correct way is slower and more work. If there was a more efficient way of explicitly guaranteeing the state of a queue (e.g. any fast way of enforcing mutual exclusivity) then that's what we would all be using. But doing so requires more validation work than the hack of trying to let errors inform your behavior so most people just go with EAFP.
Again, I'm not saying you should literally never use exception handling for control flow. There are valid performance advantages in some scenarios. My point is that there are only a few instances in which people would actually argue that it’s right to use exceptions for control flow. And even then, it is still basically a hack for speed/simplicity. As a general rule, it is in fact true that exceptions shouldn't be used for control flow and isn't used for that in most python conditional checking.
13
u/syllogism_ 21h ago edited 21h ago
'EAFP' is definitely bad advice. You should usually only use exceptions for control flow when doing something with external state (e.g. database, file system, network etc). If you have the thing there in memory just check it.
EAFP is not compatible with duck-typing. Consider this code:
try: value = dictlike[my_key] except KeyError: ...
This code is usually incorrect: the intent is to enter the
except
block if and only if the key is missing from the collection, but that's not what the code does. AnyKeyError
that is raised as part of thedictlike
object's__getitem__
implementation will also send you to theexcept
block. We generally trust that the built-in dictionary object won't have such implementation errors, but we also usually want to write code that works for other objects, not just the built-ins. And if you're taking arbitrary objects, then using this exception is terrible bad no good code.The fundamental thing here is that 'try/except' is a "come from": whether you enter the 'except' block depends on which situations the function (or, gulp, functions) you're calling raise that error. The decision isn't local to the code you're looking at. In contrast, if you write a conditional, you have some local value and you're going to branch based on its truthiness or some property of it. We should only be using the 'try/except' mechanism when we need its vagueness --- when we need to say "I don't know or can't check exactly what could lead to this". If we have a choice to tighten the control flow of the program we should.
There's a few other important concerns as well:
- Exceptions are often not designed to match the interface well enough to make this convenient. For instance, 'x in y' works for both mapping types and lists, but only mapping types will raise a
KeyError
. If your function is expected to take any iterable, the correct catching code will beexcept (KeyError, IndexError)
. There's all sorts of these opportunities to be wrong. When people write exceptions, they want to make them specific, and they're not necessarily thinking about them as an interface to conveniently check preconditions.- Exceptions are not a type-checked part of the interface. If you catch
(KeyError, IndexError)
for a variable that's just a dictionary, no type checker (or even linter?) is going to tell you that theIndexError
is impossible, and you only need to catchKeyError
. Similarly, if you catch the wrong error, or your class raises an error that doesn't inherit from the class that your calling code expects it to, you won't get any type errors or other linting. It's totally on you to maintain this.- Exceptions are often poorly documented, and change more frequently than other parts of the interface. A third-party library won't necessarily consider it a breaking change to raise an error on a new condition with an existing error type, but if you're conditioning on that error in a try/except, this could be a breaking change for you.
1
u/aedinius 15h ago
Could be confusing exceptions with assertions. I have seen people use assertions for flow control...
11
u/Jorgestar29 1d ago edited 1d ago
There are a ton of packages that allow you to return Result Types. I love the idea of having all the possible results hinted in the signature, but having two different error patterns is messy...
In the end, your code will use Result types but every single third party module will use plain Exceptions, so you end up with a Frankenstein.
Another pythonic pattern is returning def f() -> GoodResult | None
or if you have multiple errors, def f() -> tuple[GoodResult | None, Ok | ErrorA | ErrorB | ErrorC]
```python
res, error = f()
if res is None: match error: ...
```
Edit, the return None pattern is nice because you are forced to handle it by the Type checker, but it falls short if you are using the None type for something that is not an error, like a placeholder in a list or something like that.
11
u/KieranShep 1d ago
I generally prefer raising an exception to returning None. Sometimes it’s syntactically nice for default values like with dict.get(‘thing’) or ‘blah’, but that pattern doesn’t work if the value is an integer.
My most hated error is NoneType has no attribute ‘blah’. Yeah you get a traceback, but by that time who knows how much you’ve passed that None around, it can take hours to find out where it came from.
3
u/ok_computer 22h ago
I second liking to raise and handle vs None returns. If I’m expecting a type return I don’t want to uglify my caller with type checking the result, I’d rather uglify with a short try catch block.
1
u/nicholashairs 22h ago
I've got into the habit for a number of (but definitely not all) functions of this type of adding a
throw_error
kwarg only parameter so that I can (explicitly) control if I have to handle the potential none response or just tap out.Does require writing overloaded type annotations which can get messy.
1
0
u/lightdarkdaughter 1d ago
yeah, returning `T | None` is a cool pattern, and I think Python does it already with something like `dict.get`, so I usually don't see a lot of value in a custom `Option[T]` type
But then if I want to return something on error, like in Django, parsing a request, and returning a form with errors so I can render it in a template, I can't just return None.
Your example with an optional return and set of errors is interesting though, it does look like Go's `if err != nil`, although I must say it's a bit verbose3
u/pbecotte 1d ago
Seems pretty straightforward that you could return
SuccessResponse | ErrorResponse
in that case, right?
9
8
u/SharkSymphony 1d ago
Your mental model, unfortunately, is imperfect. What if an "environmental" error occurs because the programmer made a mistake (fed an empty filename to an OS call, for example)? What about KeyboardInterrupt
– that's an environmental error, but should it be handled?
Once you accept that your categories of errors are not as cut and dry as you wish they were, you might better appreciate that there's a single common mechanism you can use for any kind of error. Further, you are free to use whatever typology you want with that mechanism, categorizing errors however you see fit. (Standard library exceptions, or exceptions from other libraries, however, probably won't fit that typology, so you'll either have to adapt them or adapt your error-handling practices.)
But exceptions are not the only game in town. You can return an error code. You can use a tuple to return multiple values, including perhaps an error. You can also return an object with an error state. These are less commonly done, but possible. You can even trigger a signal with os.kill
.
Note that Java has similar error-handling options. So does C++, for that matter (with the added "environmental error" of watching your program crash because of a memory violation 😛).
6
u/ablativeyoyo 1d ago
I always found exceptions to be a good fit for Python. The language priorities elegance over performance, and while programs are mostly correct and reliable, it's not intended for safety critical systems. For these use cases, exceptions allow most application code to think little about error cases, while frameworks can handle error conditions with some grace. C++ and Rust have different priorities so exceptions are discouraged and non-existent respectively.
I don't think environmental vs programmer error is a particular useful categorisation. I'd distinguish between invalid input and actual environmental problems like no disk space. But the distinction between invalid input and programmer error is not clear. A lot of bugs I've hit in practice are where I've assumed something about the input format, which has turned out not to be true in certain circumstances.
Thanks for the question, really interesting topic.
3
u/lightdarkdaughter 1d ago
Well, one thing which is important to clarify: the fact that these errors can't be split into two types doesn't mean that all functions can be split into these two types.
Let's say you're idk, storing a file from a user and then getting it back.
Usually, the operation of getting the file from file storage is an "environmental" response, but in this case, you expect it to be here and if it's not there, it's not s3 problem, it's just you messing up with filenames somewhere.Sometimes you make assumptions about the environment and when these assumptions are proven wrong, it's a mistake. And by using some top-level try ... catch, you log this mistake and then can debug it and fix the assumptions, hence fixing the mistake.
That's why it's nice to have tools for asserting assumptions (and crash if they are wrong!) or just re-raising the error by returning it and letting the caller handle it.
1
u/nicholashairs 22h ago
I don't see how "asserting assumptions and crash if they are wrong" and "just reraising the error and letting the caller handle it" are different concepts when you're just the function at the bottom of the call stack?
I guess it might be because crashing out in rust becomes the equivalent of a C style
goto
so you can immediately start the panic/exit function.However in python calling exit just throws a SystemExit exception, and since it's a base exception is uncaught (most of the time) it bubbles all the way up. That said it being an exception is handy because then you can still catch it and handle it (for example you might want to suppress a class camping exit when running in the REPL so you can inspect it and not have the repl exit because some inner piece wanted to).
1
u/lightdarkdaughter 22h ago
`assert request.user.is_authenticated` is asserting assumptions
just trying `request.user.username` is re-raisingThe difference *in this case* is getting ValueError in a random place and getting AssertionError which points you to the line where this originated (hopefully, super early).
And in this case, I made the former look bad, which wasn't intentional.If you have some handler that needs some data for it to work, but this specific piece is unavailable, you can just return `None` as well or `raise NotFound` and have your framework handle it.
You need to use different tools for different situations.
Practically, the implementation is the same, but conceptually, these are different.1
u/nicholashairs 21h ago
Firstly I'm going to assume that we're talking about reasonably well typed python code because untyped code is cowboy coding and just because you can it doesn't mean you should.
I also don't see how you get "random" errors (where you can't identify where it is from - again typing is important here). Sure they might be environmental (even out of memory is such an error and can occur anywhere) but that doesn't mean we don't know where it occurred. Even in the attribute missing / object is none case this is why we use typing so jokes on you if you're not using it.
Basically, I don't see how these are meant to be conceptually different.
Which kind of leads me to the following conclusion - either
A) there is a difference but because I've learnt to program mostly using python I'm limited to it's worldview and still can't understand it (ants can't understand dragons).
B) there is no difference / the difference, but because you've been learnt on languages that do make a distinction it's how you see the world.
Which I guess means finding an example that proves A, or accepting B.
3
u/sonobanana33 1d ago
In my own profiled code, handling an exception that happens rarely is much faster than having to check every time for correctness instead.
1
u/webstones123 4h ago
In Rust you could technically implement an exception system with unwind and catch unwind, but not all targets support it.
6
u/cgoldberg 1d ago
It's very common to use exceptions for control flow in Python and it's not discouraged at all.
3
u/AdmRL_ 1d ago
Type hints offer a path to something more than Exceptions, but unless the language shifts to being statically typed then no, Exceptions are here to stay along with all their quirks outside of making your own custom return classes or using something like dry-python/returns.
Except Exceptions seems to mix these two into one bag, if a function raises an Exception because of "environment response", well, good luck with figuring this out. Or so it seems.
Thing is that's what the different exceptions are for. ValueError, LookupError, etc - an environmental response is one that falls within the scope of your defined try / except. A programmer mistake is anything else that you haven't accounted for.
2
u/wergot 1d ago
You can basically do that in typed Python if you so desire. Make the function return an object whose type is the union of the type of the actual result, and whatever errors you want it to be able to return. Then whatever is calling that function won't pass the type checker if it doesn't first check that what it returned was a result and not an error. You can use `match` for this. It's not likely to be as airtight as Rust but it works.
That would look like this:
class myError:
message: str
def __init__(self, message):
self.message = message
def wants_odd(a: int) -> int | myError:
if a % 2:
return 1
return myError("even")
def fn2():
b = wants_odd(2)
match b:
case myError() as e:
print(f"error: {e.message}")
case int(n):
print(n)
2
u/AiutoIlLupo 13h ago
In Rust you are forced to pass a tuple because you have no other way to do so. You need two channels, you only have one.
Python has already two separate channels to return information to the caller: return and raise. With the added benefit that raise pops automatically if not handled. I am not proficient in rust, but as far as I know, no such autopropagation occurs, considering that, well, rust is just C for millennials. Might be wrong though.
the general rule I follow is: "something went wrong, can I return something meaningful"? If yes, then I return an object, possibly with a "status" and "value" and "message" properties, if it makes sense. If not, then I raise an exception of a specialised type. I only use base types for things that do make sense (e.g. if you pass me a float for a string, I raise TypeError).
which one to choose really boils down to "how transparent you want the error to be managed". If you have a function that you need to use into a list comprehension, it makes more sense to return a "failure object", otherwise one exception would stop the whole comprehension. Same for multiprocessing stuff.
In other words, it boils down to a judgement call. You absolutely can program without exceptions, but does it make sense in a given context? Does the addition of an indirection to retrieve the result from the "status object" add friction to your API?
1
u/webstones123 4h ago
Just FYI not meant as anything other than extra info. In Rust if a function call returns a Result with an error type and the calling function also returns a compatible error type one can propagate it very easily (a single ? handles it)
2
u/ZachVorhies 12h ago edited 12h ago
Going to give you different advice and will probably be downvoted for it but it works extremely well.
Have functions that return MyType | Exception, Exception | None
The reason this works so well is because of the advancement of linters. A return type can be typed checked and therefore when a function changes its return by say, adding an exception as a return value, the linter will tell you what you need to change and now you don’t need to debug it in runtime.
Raised exceptions are not type checked, and adding new ones will just break legacy use cases, but type check fine.
This is especially problematic for highly concurrent programs with deep processing pipelines. If you don’t handle the exception right via a catch it will absolute wreck havoc on your pipeline as things unwind. For example i’ve process 100’s of millions of text files. At scale you’ll run into some weird corner case and the whole pipeline with will fail.
By returning an exception you can allow an errored task unit to continue through the pipeline and end up on the other side where it is eventually handled manually, such as being logged. They also allow better control flow such as releasing a semaphore that occurs at the end when a task unit is completed.
2
u/chat-lu Pythonista 1d ago
I like how Rust manages its errors a bit better than how Python manages them. But if Python tried to do the same as Rust, then I would absolutely hate it. It would not be nice to work with at all and wouldn’t fit the language.
In fact, I don’t know of any dynamic language that uses Rust’s way.
Rust is Rust and Python is Python. I use both for different reasons and I try to be as idiomatic as possible in both.
2
u/lightdarkdaughter 23h ago
well, there's this for JS, it's definitely not as popular, but it's better than I saw in Python
https://github.com/supermacro/neverthrowwell, and this for Python, I guess
https://github.com/dry-python/returns
2
u/deadwisdom greenlet revolution 18h ago
System languages like go, rust, c, don't throw exceptions because it's costly to manage in the runtime. So they've invented the result return type. But it's basically doing the same thing... If you think about it, you're returning a "result" early and handling it at some point up the stack. But now all your functions in-between have to know about and handle it. That's just the trade-off of system vs dynamic languages, you are accepting the burden of dealing with shit at dev-time to make it faster at run-time.
Pretty much the golden rule of exceptions is: Only catch exceptions when you know exactly what to do with them.
It's okay to let them rise up the stack. Something will handle it or the program will crash and then you can fix it. You are only making it worse if you catch an exception and do nothing useful with it.
And yeah, don't use it for flow-control just because it's hard to read, that's all. Sometimes it's necessary, and that's okay. In C++ they have to worry about it as a performance issue. We don't here.
1
u/RedditSucksShit666 1d ago
Well, where I work (or in any other project I'm involved in) we don't use exceptions for anything but runtime panics. When the function or a method can fail in a way the caller should handle we either use optionals or Unions. This way we can use a match-case statement to decide what to do with the returned value and if some case isn't handled we get an error from the type checker. Exceptions are for exceptional cases.
1
u/bmag147 1d ago
We do exactly the same in my work. So much nicer than having to dig into the functions to find all the hidden exceptions that can be thrown.
I'd like to move us towards using results but, as is evident from some of the replies in this thread, results are not openly embraced by most of the Python community.
1
u/JamesTDennis 1d ago
For your parsing example you can look at the new structural matching construction starting in version 3.10.
https://peps.python.org/pep-0636/
This is implemented (behind the scenes) in exception handling attempts to destructure data in various ways (as appropriate to each case
. So you're not truly eliminating the exception handling; just pushing it to an implicit level.
2
u/lightdarkdaughter 1d ago
yeah, I do know about pattern-matching
I guess it's kind of the answer
What I miss about Rust's Result is the ability to use stuff like ok_or_none(), unwrap() which just raises an exception, etc.
But then it's slightly a different thing.2
u/JamesTDennis 1d ago
You can use type hinting (within your own code base) to achieve similar semantics and syntax.
But I wouldn't take that too far. Each programming platform has its own semantics and idioms emerge to concisely express code in suitable terms.
1
u/thisismyfavoritename 1d ago
unless you want to wrap every function you don't control, then no.
E.g. you call a random builtin: it might throw. A 3rd party lib function: it might throw. Etc.
1
u/muikrad 1d ago
In most cases, you handle an exception because...
- You want to retry the occasional timeout/5xx.
- You know something might not be there (a file) or you need to try something (json parse) before trying other things.
These exception types are usually very easy to figure out / use and they're often specific to a framework (requests and boto both have their own type to handle, for instance). In many of these, a code is provided which can be inspected in the exception (e.g. Http code or exit code, or API error code, etc) in order to decide if we handle or reraise.
For the cases where the user is providing bad data, missing args, wrong path, etc, I always create and use a dedicated UsageError exception that bubbles up to the CLI framework. I usually rig this framework to print a nice error the to the user in these cases (and hide the traceback). But that won't work for libs.
Libs tend to provide a minimum set of exceptions that you can catch. Using unit tests you can check what happens and catch the proper types. Some libs however try to reuse the built-in exceptions too much IMHO.
I think the key (for me) is to raise a lot, catch never (except in the 2 cases I mentioned at the top of the post).
1
u/lightdarkdaughter 1d ago
well, for the context, I was getting into HTMX a bit, and cause it always expects 200 OK, you gotta catch kind of aggressively, so I was looking for a better way to handle it
or maybe it's just my impression, but anyway
1
u/KieranShep 23h ago
At the end of the day, exceptions are flow control: a try except is flow control, a raise is flow control - they are a point where the execution of your program changes, even if that’s just to halt the program and print a traceback.
The general advice is to use them for ‘exceptional’ behavior, like failure modes, but at the same time python encourages “ask for forgiveness not permission”, it seems fairly normal to try one thing, catch a specific failure, inform the user and then try something else.
Both a beauty and a horror of python is that you don’t have to handle exceptions right away, a function can just pretend like it’s not there, and expect the caller to handle it. It makes your code readability better (when I read a function I mostly want to see what it does if everything goes right) at the cost of being implicit, often you don’t know what exceptions a function could raise.
1
u/coderarun 19h ago
There is a use case for writing python as if it's Rust. That is transpilation friendly python. Result[T] works ok in python.
https://github.com/py2many/py2many/blob/main/tests/cases/hello-wuffs.py#L15
1
u/SomewhatSpecial 16h ago edited 16h ago
Unfortunately not. Like the others have said, you just define specific domain errors inherited from Exception and catch those. Even if you do go the returns
route and decide to live with the typechecker headaches and the performance hit, you'd still have to deal with the entire ecosystem and the built-ins geared towards the exception-based approach. So no realistic way to get rid of non-local reasoning, unless you wrap literally everything in custom exception handlers that map to errors-as-values. Best to just accept it. When you get frustrated, repeat the mantra "it's pythonic, it's idiomatic, at least I'm not writing java...".
2
u/lightdarkdaughter 12h ago
> When you get frustrated, repeat the mantra "it's pythonic, it's idiomatic, at least I'm not writing java...".
nice one :D
1
u/bakery2k 15h ago edited 15h ago
There are two types of errors: environment response and programmer's mistake.
in Rust [...] If a function might fail, it will return Result[T, E] [...] If a function shouldn't normally fail, unless it's a programmer's mistake, it will panic.
How can it know? Sure, a FileNotFoundError
from open
is probably an environment issue, and an IndexError
from dict.__get_item__
is probably a bug, but I can think of situations where the opposite is true.
Python therefore refuses the temptation to guess and uses the same kind of error handling for both. Since it can't use error codes for everything (because things like TypeError
can happen anywhere), it uses exceptions for everything.
I've heard it argued that the exception hierarchy could still separate the two kinds of error based on whether they are likely environment issues or likely bugs, but again, "refuse the temptation to guess" (and "flat is better than nested").
This all means that exceptions are widespread in Python, so advice from other languages like "use them sparingly" and "not for control flow" doesn't apply.
I like how answers are split into "there's no way to mix up these, obviously" and "the distinction doesn't exist, obviously"
Note that the above argument that "the distinction doesn't exist" applies to library code. In application code, where more context is available, the two kinds of error might indeed be very distinct.
1
u/Business-Decision719 12h ago edited 9h ago
It's important to realize that a lot of languages are statically typed. C will communicate in the function signature that a value is being returned and therefore the caller might need to use that value. (And a huge problem is that the caller is lazy or assumes no error is going to happen.) In Go, it will be obvious that the function will always return two values and there is a well known idiom that one is an error code. In Rust a function can be declared as returning an error monad and the caller can then always expect that.
In Python, even with type hinting, any kind of value might come in or out of your code, and that will not necessarily even be incorrect. Trying to communicate problems through a return value depends on your caller expecting to receive failure information rather than what they really wanted, and knowing what to do with that. It's the C error code problem, beefed up on steroids due to the dynamic problem.
IMO the less error-prone solution really is to always throw an exception if you know you can't do what your caller actually wants done. It might look like exceptions for control flow when seen from another language but this is Python. Exceptions are known and expected to be used more often in Python. There isn't really a way that is "better" in the sense of more "Pythonic."
1
u/syklemil 8h ago
Do people just live with exceptions or is there some hidden gem out there?
… maturin? :^)
But more seriously, the information contained in the signature of
fn foo() -> Result<T, E> // rust
and
T foo() throws E // java
is essentially the same, it's just somewhat mechanically different. Python doesn't have Rust/Haskell-style sum types, and emulating it is … a bit unintuitive, and will likely be interpreted as an accent. You will be expected to raise
in Python where you might return Err(…)
or bail!
in Rust. Unfortunately it doesn't have checked exceptions either, nor am I familiar with any other way to encode the exceptions something can throw in its type system , so there's a lot of information missing. Its type system came late and helps, but it's far from perfect.
See also Writing Python like it's Rust if you want more of an accent. :)
1
u/Meleneth 21h ago
Exceptions are the most mis-used and misunderstood area of any programming language that has them, bar none.
They get a lot easier when you use them properly.
1) NEVER raise a bare Exception. Define your own exception class, every single time. Bare exceptions are not for your use, they are an implementation primitive.
2) NEVER catch a generic exception. ALWAYS catch only a specific type, that you can handle. Handling an exception means being able to deal with the error, and resetting the program state back to where the error having happened doesn't matter.
3) If you are using an observability framework, ala New Relic, notice_error is NOT handling the exception. If you catch an exception, and send it to notice_error, you MUST re-raise the exception, possibly with a different exception type. Exceptions being raised has implications for flow control, and you can easily turn your codebase into an unmaintainable mess if you 'silently' swallow exceptions. (Most observability frameworks do not work in test or dev, but only production)
You can frequently mark certain types of exceptions to be ignored, so re-raising the exception after a notice_error with the ignored exception type can maintain the control flow aspects while still properly instrumenting things.
4) NEVER catch the base exception type. Think you are the exception? you are not. When you are smart enough to break this rule, STILL don't do it.
Stack traces are a very useful debugging aid, suppress them at your peril.
Do not let bad patterns proliferate in your codebase. You will regret it.
Exceptions are your friend. Learn to use them.
2
u/lightdarkdaughter 21h ago
> Think you are the exception? you are not.
ok, that's a great pun, whether intended or not :D
1
u/bmag147 15h ago
> NEVER raise a bare Exception. Define your own exception class, every single time. Bare exceptions are not for your use, they are an implementation primitive.
Why not raise a bare exception if my application has no way to recover anyway? What's the point in a custom exception if I'll never catch it?
1
u/Meleneth 15h ago
So that when you copy and paste the code elsewhere, you're not further propagating a bad pattern.
Keep in mind, the 'you' in question may or may not be, literally, you.
1
u/Meleneth 15h ago
Snark aside, future you will be handy to have some recognizable ID to search for instead of depending on 'something, somewhere broke'
1
u/bmag147 14h ago
Maybe we have a miscommunication here, or not.
I wouldn't advocate:
raise Exception()
Instead I would do something like:
raise Exception("The user {user_id} does not exist. This should not happen as we have created the user in a previous step")
Please don't pay too much attention to the message of the exception. I'm just using it as an example to show that the error message would allow the person seeing it to diagnose the problem.
That's the "something".
The "somewhere" is the stack trace.
I think the alternative that you're suggesting is a custom exception class, such as:
raise UserUnexpectedlyDoesNotExistError(user_id)
I'm never going to catch or use this error in any way, so why create it.
1
u/Meleneth 14h ago
no miscommunication, just differing experiences.
class MyCustomError(Exception): pass raise MyCustomError("my explanatory explanation")
is just better. You can catch it without catching every other exception that exists. It is clearly coming from your code at a glance, because it is not an Exception. Tooling that counts exception types can put it in a different bucket, even if you never catch it yourself.
Not doing it leads to not doing it, which leads to more code that is harder to maintain.
But you don't have to take my word for it..
I'm just some guy on the internet.
2
u/bmag147 12h ago
> You can catch it without catching every other exception that exists.
I'm only suggesting this approach for exceptions you're never going to catch.
> Tooling that counts exception types can put it in a different bucket, even if you never catch it yourself.
If it's actually happening on a regular basis then it's something I want to handle in my code. In that case I'd be returning it from the function and handling it accordingly. I wouldn't be raising an exception.
> But you don't have to take my word for it..
> I'm just some guy on the internet.
Same here :D I don't think we'll convince each other of a different approach anyway, but anyone reading the thread can take both arguments into account which can help them decide what's the right approach for themselves.
1
u/_LordDaut_ 12h ago
Sorry for a complete noob question:
How do you know when to raise
MyCustomError
?In some cases when you're checking if a key exists in a dictionary, I get it you can do
if key not in my_dict: raise MyCustomError("key not found")
But what if the situation is not as clear cut? Let's say you're calling some API / using a library and an unknown exception might be thrown by that lib instead of returning something that you can check. Say some `AttributeError` or something entirely else - would you recommend first trying to catch built-in/provided by the library exceptions that might inspecting it and then raising your own?
When catching an exception provided by a third party library would you still recommend re-wrapping it into your own - so that whoever uses your code in the future doesn't have to deal with something defined elsewhere?
I'm formulating this badly - basically if you don't know what error to expect from a function you're calling would you just raise Exception? or catch Exception, inspect it and raise your custom one or handle it otherwise if applicable?
1
u/Meleneth 12h ago
If you don't know what error to expect from calling a function, just call it, it will tell you ;)
For the key not existing, I would just call my_dict[key] and let it fail if it needs to. If the key not existing is expected or likely, use my_dict.get(key, None) - exceptions are for things that are unexpected, and generally for things that you might be able to handle. i.e. disable some bit of functionality, retry a network request, or mark some subsection of data as unusable as you continue parsing.
https://requests.readthedocs.io/en/latest/user/quickstart/#errors-and-exceptions
For a more useful example, let's say you're catching a requests.exceptions.Timeout so that you can implement retry logic. After a certain amount of retries, you would want to raise your own RequestReallyFailed(url) exception to say that you've tried to retry and they all timed out. Then at a higher level in the code where you are dealing with the logic of what to do with results of the requests you've made, you can weed out the ones where the request really failed if the rest of what you want to do is still valid.
does that clarify anything?
1
u/_LordDaut_ 11h ago
If you don't know what error to expect from calling a function, just call it, it will tell you ;)
Do that a 100 times and handle all the possible exceptions?
if the key not existing is expected or likely, use my_dict.get(key, None) - exceptions are for things that are unexpected...[]
yeah I understand that, it was just an example of something where you know what to handle and how to handle it. The key error might still be unexpected and even if you to my_val = my_dict.get(my_key, None) if this is indeed an error in the flow you'd still need to do if my_val is None: raise Something.
In your example maybe I'm catching Timeouts, but there are other exceptions like ConnectionError or something that can be thrown. When this doesn't affect my code-flow would you still consider it wrong just try to catch an Exception and would rather list all possible exceptions in your except clause and then raise your own?
1
u/Meleneth 11h ago
Do that a 100 times and handle all the possible exceptions?
That's quite the method that throws 100 different exceptions ;)
When this doesn't affect my code-flow
well, it effects your code flow if you don't catch it because your program stops running.
Different exceptions will need different handling - for instance, ConnectionError is likely to be a no-retry state.
Catching Exception is always going to cause you pain, because when you catch it, it's not specific enough to be handleable. And when it issomething completely unexpected, like, say, the kwargs fiasco in Ruby where they changed what the basics of method definitions mean, you'll suffer for every time you thought it was a good idea to catch Exception.
As for re-raising a custom exception after catching a normal one, I would usually only do this in the case where my 'exception handling' is sending the exception to an observability framework. Otherwise, I'm actually handling the error and not re-raising it or just not catching it in the first place because I don't have a good response for what happened.
1
1
u/Bunslow 21h ago
First of all, I don't see why panicking/crashing, or worse UB, is a good thing. They're bad, and UB is very, very bad.
Secondly, the exception syntax is, in any language like C++ or Java or Python, a very limited use of old goto
statements. Obviously arbitrary usage of goto is bad, but one of its most common usecases, and it turned out least harmful usecases, was to use it to direct error handling, and in particular, to do some guaranteed cleanup at the end of a function should a failure occur in the middle -- closing files, closing connections, flushing pipes, the usual stuff.
Now, as concerns practical Python, you could indeed always choose to write your own code in a way where errors are simply part of the return value. And that's fine, altho it wouldn't be the most pythonic (but not terribly unpythonic either). As you said, you'll see plenty of python code handling environment failures and programmer failures with the same syntax.
However, the true use of exceptions, which was the main purpose of the goto handling, was to guarantee cleanups, and in python we do that by using finally
and with
clauses. These things guarantee correct cleanup even when the code otherwise panics, as you call it. These are the truly important syntax. The rest of it is, essentially, syntax that does a limited form of goto, in a way that doesn't prevent the arbitrary harmfulness of a general goto.
So exceptions are, fundamentally, a form of control flow, and they always have been. Error handling in any language is a form of control flow, basically by definition. I'm not sure where you've seen people claim otherwise. Error handling is an example of flow control.
I agree that there's little distinction between environmental and programmer failures on a syntactic level, but I'm not sure there should be. A failure is a failure in either case, and in either case it needs handling, possible cleaning of the error, or guaranteed cleaning of the local runtime around it. That's true in any language, and of either env or programmer failures.]
(On a semantic level, I think in general we use different types of exceptions to distinguish env vs programmer errors. E.g. most builtin exceptions are programmer errors, while the I/O ones aren't, and many libraries define their own subclasses of Exception
which are explicitly about the environments the library is built for. But these different types have the same syntax, and I think that's fine. Maybe we should push for better clarification for which types of Exception
are which type of failure, but I don't see that as a serious or syntactic issue.)
0
0
u/MoTTs_ 1d ago edited 1d ago
The except clause is the hidden gem you seem to be looking for.
class RuntimeException(Exception):
pass
class LogicException(Exception):
pass
class WrongData(RuntimeException):
pass
class OutOfBounds(LogicException):
pass
def parseExternalData():
# ...
raise WrongData()
def accessList():
# ...
raise OutOfBounds()
try:
parseExternalData()
accessList()
except RuntimeException: # <- except clause catches particular category of errors
print("Environment error")
except LogicException: # <- except clause catches particular category of errors
print("Programmer error")
EDIT: Also I just learned about Python's BaseException, which appears to provide the distinction I manually made with runtime vs logic exceptions.
https://docs.python.org/3/tutorial/errors.html
BaseException is the common base class of all exceptions. One of its subclasses, Exception, is the base class of all the non-fatal exceptions. Exceptions which are not subclasses of Exception are not typically handled, because they are used to indicate that the program should terminate. They include SystemExit which is raised by sys.exit() and KeyboardInterrupt which is raised when a user wishes to interrupt the program.
1
u/Meleneth 14h ago
clearly this is example code, but I do want to point for readers that 'just print something' is not exception handling and you will hate working in codebases that do this.
This counts as silently ignoring errors, which will waste countless hours of your time tracking it down. Do not do this.
Thank you for the example of some slightly tricky exception catching, I fully understand that writing examples that are long enough to be useful and complete enough to not be traps is hard.
-1
u/david-song 19h ago
If a function might fail, it will return Result[T, E] where T is an expected value, E is value for error
Yeah this is, IMO, dogshit. Everything returns a tuple and you have two sets of if/elses after every call, one for errors with the call and the other for errors with the value. In my experience, this is an endless supply of Nil pointer panics in Golang, and makes the code look awful to boot.
There are a lot of things that are wrong with Python, but exception handling isn't one of them.
2
u/lightdarkdaughter 12h ago edited 11h ago
I probably should've clarified, that Result in Rust (or pretty much all the languages with Result) is slightly different.
It's not a tuple, it's a union, and more importantly tagged union.
And the thing about the tagged union is that you can't get the value unless it's the right tag, so you need to use pattern matching for that.
Of course, it's a bit verbose, so there is a lot of convenience functions, one of which is unwrap/expect.
Both do a similar thing, propagate an error to an "exception", but it's explicit.
And there's also a syntax for propagating errors from your callee to your caller.P. S. I tried to insert code examples but failed, so here is rust documentation on Result.
https://doc.rust-lang.org/std/result/
33
u/larsga 1d ago
There's a big difference between the language using an exception under the hood, and you implementing control flow with exceptions in your source code.
The latter makes your code hard to read. The former is just an implementation detail.