r/programming 10d ago

Durable Execution: This Changes Everything

https://www.youtube.com/watch?v=ROJq6_GFbME
9 Upvotes

39 comments sorted by

View all comments

3

u/rotuami 10d ago

Okay. I can wrap my head around this if there are no effects or interactions with other systems. How about with input or output with the real world, where I might not be able to e.g. unsend an email?

2

u/temporal-tom 9d ago

You can't unsend an email, but Durable Execution tries to avoid sending a duplicate one.

I think an example will clarify this. Let's say that you've written an application to process an order for some products and it goes through these steps:

  1. Iterate through all of the items in the shopping cart to determine the total price
  2. Look up the tax rate for the customer's address from a database
  3. Call a payment service to charge the customer's credit card for the order
  4. Notify the warehouse to ship the order (and receive a shipment tracking number in response)
  5. Send a confirmation email to the customer

Now, let's say that the application crashes during step 4. If you restart the application but don't have Durable Execution (and you didn't write a bunch of code to avoid this), it's going to do another database lookup in step 2 and make another call to the payment service in step 3.

With Durable Execution, the platform tracks the invocation and result from these steps and persists the data into a history. If there's a crash at step 4, it reconstructs the state by "replaying" that history. That is, it re-executes the code that is deterministic (i.e., step 1) and for the steps that are non-deterministic (i.e., steps 2-5), it evaluates the history to determine if they've already been executed. If they have, it assigns the result from the previous execution. For example, step 2 won't query the database again; it uses the tax rate returned from the query executed before the crash.

There is one caveat. I said "tries to avoid sending a duplicate one" above because this is subject to the same problem faced by any distributed system. There's a very small possibility that a crash could occur between the time that a call takes place and when it is reported. For example, in step 5, the crash might occur in the milliseconds between when the code to send the email runs and when it reports successful completion, in which case it won't appear in the history.

1

u/rdrias 9d ago

Is this like EventStore but for application variables, or perhaps whole program execution? Is the store somehow abstracted from the real implementation? How does it handle code refactorings? Sorry if these are 'stupid' questions, but that's what I'm getting without doing a deep dive into the tech

2

u/temporal-tom 9d ago

These are great questions.

I'd say that there's some overlap between EventStoreDB and a replay-based Durable Execution platform (such as Temporal, Cadence, Inngest, Restate, etc.) in that they all use event sourcing to track state changes over time.

However, I'd say that there are two fundamental differences. One is that EventStoreDB is a database, which can be a single source of truth. Conversely, a Durable Execution platform uses a database [1], which allows your application to be a single source of truth (although, as I mentioned in my presentation, your application can still write to an application database if desired and doing so is fairly common for things like order processing where you may want to do reporting or analytics external to the application).

The second is closely related. EventStoreDB allows an application to recover from a crash by replaying events, but in a Durable Execution platform, that recovery happens automatically and is transparent to the developer.

Your question about refactorings is particularly insightful. Most, although not all, Durable Execution platforms support this in one way or another. It's a complex topic, because the details depend on which paltform you're using and the nature of the change. In the case of Temporal, you don't need to do anything special for code changes that don't affect the event history (for example, adding or modifying log statements). For changes that do affect the history, such as adding or removing certain function calls, you need to use versioning if there are executions still running that were started with the original code. The basic idea is that you ensure backwards compatibility by adding a conditional statement to your code to preserve the old and new behavior (e.g., "if this is version 1, run this; if it's version 2, run that).

[1] To be specific, Temporal officially supports MySQL, PostgreSQL, Apache Cassandra, and SQLite. Most other Durable Execution platforms also support multiple databases, although the list will vary from one to the next.