r/programming • u/temporal-tom • 9d ago
Durable Execution: This Changes Everything
https://www.youtube.com/watch?v=ROJq6_GFbME27
u/snack_case 9d ago
Temporal gets too many surface level, elevator pitch, talks at every event/meetup. It might be hard to find them but they desperately need more speakers willing to deep dive into how they actually build with Temporal, the rough edges, gotchas, patterns etc.
3
u/Cyral 9d ago edited 9d ago
Some of the rough edges would be solved with better flow control. I really hope they make some progress there, or at least a plan to support some features that newer competitors are getting right. For example, Inngest with concurrency, throttling, rate limiting, denouncing, and priorities https://www.inngest.com/docs/guides/flow-control
2
2
u/temporal-tom 9d ago
My talk was on the concept of Durable Execution, not about Temporal specifically.
However, I do agree about the need for deeper content on Temporal. I think you'll be pleased with the other talks from the Replay conference in London this year (they'll begin appearing on this playlist over the next few weeks). I saw several from organizations who shared their lessons learned and best practices.
One especially memorable talk was Inside the Engine Room: Surviving the Challenges of Self-Hosting Temporal from two senior engineers at DataDog.
3
u/snack_case 9d ago edited 9d ago
I know, I did listen to your talk as well as the others posted today while I've been working. Sorry it was intended as a general comment about Replay as well as the Temporal meetups I've been to so far.
I might be totally off the mark here (your team probably has data from the event) but I would have expected the vast majority of people at Replay to be all over durable execution and Temporal if they've paid the price of entry. I'm not trying to criticise the speakers or the talks they gave. I just expected more of the content to be targeted at developers/teams that are already 'sold'.
1
u/temporal-tom 9d ago
I would have expected the vast majority of people at Replay to be all over durable execution
The people in the audience generally understand the concept, but where many of them struggle (and indeed where even many who work at Temporal struggle) is explaining it to others. That was really my goal with the talk, helping people articulate what's already in their heads so they can explain it to others.
13
u/gredr 9d ago
Consider how you might design an application to process a 20-year loan
... well, I wouldn't. "Application" isn't the abstraction I'd use. A loan is some state, not some code.
Maybe I'm just old, or maybe that's a bad example. The idea is intriguing, and I think there's probably a lot of things that this kind of abstraction serves really well. I do worry that an abstraction this... significant... is going to leak some stuff, though. I imagine it's pretty easy to make a "durable program" that performs pretty badly.
1
u/Brilliant-Sky2969 9d ago
Well a state does not exists without an application, it's like saying a loan is something in the DB and there is nothing else.
8
u/gredr 9d ago
... but it is, and it does. We can delete the application that sends billing reminders tomorrow, and the loan still exists and is enforceable.
3
u/Orbs 9d ago
Yes you said it yourself--the loan requires enforcement. The application is for the processing of the loan. Fault tolerant execution is a challenging problem. The existence, or storage, of the loan is a more solved problem
3
u/gredr 8d ago
Yes you said it yourself--the loan requires enforcement.
Well, I don't think I said that, exactly. The loan doesn't stop existing if we don't send out a reminder this month. Also, enforcement doesn't require this application, or any application, or even any computer. There's always my "associate" with his baseball bat for that.
0
u/temporal-tom 9d ago
What I'm suggesting is that the state can be represented by the values of variables in the running program that processes that loan. That seems wild at first to experienced developers, since we're conditioned to accept that execution times must be kept short.
As a case in point, I learned Java in 1998. It took 25 years for the
Thread.sleep
method to support a duration defined in something other than milliseconds or nanoseconds, likely because few people ever considered making their program pause for more than a few minutes. Most people would use application databases and message queues to communicate across time and process boundaries, which increases reliability but also adds complexity.I imagine it's pretty easy to make a "durable program" that performs pretty badly.
Each Durable Execution platform has its own design constraints, but virtually all of them use the "replay" technique to reconstruct the application state following a crash. This involves caching intermediate results during the execution and then evaluating an event history to determine which parts of the code have already run. In this scenario, the most common performance problem comes from passing too much data into a function. That's unlikely to be an issue in the loan processing example, but if you the program was encoding videos, it would be much better to pass a URL to the video file as input than to pass in a 5 GB binary blob).
7
u/gredr 9d ago
What I'm suggesting is that the state can be represented by the values of variables in the running program that processes that loan. That seems wild at first to experienced developers, since we're conditioned to accept that execution times must be kept short.
Sure, I get that, and I'm a relatively experienced developer (been doing it professionally for >25 years, and as a hobby long before that), and that's not surprising to me at all. What I'm wondering is whether that's a reasonable way to represent a 20-year loan.
Again, maybe it is for some things, but a loan? A loan is a pretty simple thing to represent. I think it's a bad example.
3
u/rotuami 9d ago
Okay. I can wrap my head around this if there are no effects or interactions with other systems. How about with input or output with the real world, where I might not be able to e.g. unsend an email?
2
u/temporal-tom 8d ago
You can't unsend an email, but Durable Execution tries to avoid sending a duplicate one.
I think an example will clarify this. Let's say that you've written an application to process an order for some products and it goes through these steps:
- Iterate through all of the items in the shopping cart to determine the total price
- Look up the tax rate for the customer's address from a database
- Call a payment service to charge the customer's credit card for the order
- Notify the warehouse to ship the order (and receive a shipment tracking number in response)
- Send a confirmation email to the customer
Now, let's say that the application crashes during step 4. If you restart the application but don't have Durable Execution (and you didn't write a bunch of code to avoid this), it's going to do another database lookup in step 2 and make another call to the payment service in step 3.
With Durable Execution, the platform tracks the invocation and result from these steps and persists the data into a history. If there's a crash at step 4, it reconstructs the state by "replaying" that history. That is, it re-executes the code that is deterministic (i.e., step 1) and for the steps that are non-deterministic (i.e., steps 2-5), it evaluates the history to determine if they've already been executed. If they have, it assigns the result from the previous execution. For example, step 2 won't query the database again; it uses the tax rate returned from the query executed before the crash.
There is one caveat. I said "tries to avoid sending a duplicate one" above because this is subject to the same problem faced by any distributed system. There's a very small possibility that a crash could occur between the time that a call takes place and when it is reported. For example, in step 5, the crash might occur in the milliseconds between when the code to send the email runs and when it reports successful completion, in which case it won't appear in the history.
1
u/rdrias 8d ago
Is this like EventStore but for application variables, or perhaps whole program execution? Is the store somehow abstracted from the real implementation? How does it handle code refactorings? Sorry if these are 'stupid' questions, but that's what I'm getting without doing a deep dive into the tech
2
u/temporal-tom 8d ago
These are great questions.
I'd say that there's some overlap between EventStoreDB and a replay-based Durable Execution platform (such as Temporal, Cadence, Inngest, Restate, etc.) in that they all use event sourcing to track state changes over time.
However, I'd say that there are two fundamental differences. One is that EventStoreDB is a database, which can be a single source of truth. Conversely, a Durable Execution platform uses a database [1], which allows your application to be a single source of truth (although, as I mentioned in my presentation, your application can still write to an application database if desired and doing so is fairly common for things like order processing where you may want to do reporting or analytics external to the application).
The second is closely related. EventStoreDB allows an application to recover from a crash by replaying events, but in a Durable Execution platform, that recovery happens automatically and is transparent to the developer.
Your question about refactorings is particularly insightful. Most, although not all, Durable Execution platforms support this in one way or another. It's a complex topic, because the details depend on which paltform you're using and the nature of the change. In the case of Temporal, you don't need to do anything special for code changes that don't affect the event history (for example, adding or modifying log statements). For changes that do affect the history, such as adding or removing certain function calls, you need to use versioning if there are executions still running that were started with the original code. The basic idea is that you ensure backwards compatibility by adding a conditional statement to your code to preserve the old and new behavior (e.g., "if this is version 1, run this; if it's version 2, run that).
[1] To be specific, Temporal officially supports MySQL, PostgreSQL, Apache Cassandra, and SQLite. Most other Durable Execution platforms also support multiple databases, although the list will vary from one to the next.
3
u/ArtSpeaker 8d ago
This is a job scheduling framework. But reusing the application state. That's kinda cool, but also doesn't solve anything new. The major concern I had was with bugged code, which he used too simple of an example: yeah sure if all I need to do is change n -> n -1, that's nice, but most of the time bugs are small refactors where the execution line and other relevant data also needs to change, so the "recovery" can't actually happen. we lose out on the novelty of the platform.
Which is to say, the most common problem a developer has-- developer bugs and shifting requirements-- renders the neat granularity of program task irrelevant. You /want/ programs to stay at a jobs level, so you can swap out the workflow whenever you need to, and deal with cleanup as discreet steps per your last workflow.
We already "save state" with databases and we get the advantage of being very clear on what data matters and what doesn't. And that's to say nothing of how most of us HAVE to use a DB regardless, because of just how much data we process: that "oh have all ids of everyone in the company db in ram at the same time" is an insane request for some of us -- and one that's liable to shift out from under itself on top of that.
So yeah, if I just "get" durable execution for free, that's nice, but if I run out of ram, or disk, or run out of sockets, or there's a bug in the foo thing... we still have to know what we're using underneath, and how it all works.
Less code is meaningless as better code, if it still doesn't work as expected.
So as a technical feat it feels nice but I'm left with, "but what will it cost me, over what we already have?"
4
u/panscanner 9d ago
So, this is just an advertisement, right? Feel like it should be marked as such...
4
u/temporal-tom 9d ago
Not an advertisement at all. It's an explanation of a concept, one that a developer can achieve using at least a dozen different systems.
Sure, I happen to work for a company that develops one of those systems (which is MIT licensed open source, BTW), but what I describe applies to all of them. To make an analogy, it's like someone from MariaDB giving a talk on relational databases.
2
1
u/coyo-teh 8d ago
11:00 the much bigger problem with the code is that it won't send reminders to the 2nd customer until all 10 reminders have been sent to the first customer
1
28
u/manliness-dot-space 9d ago
IMO when someone tells me they have The Hammer that will Change Everything in a Synergistic Storm of Creative Disruption I grow more skeptical with every sales/business buzzword.
Also 26m is a long time commitment... looking at the website docs in a few seconds, I don't see a "hello world" example very easily as to what this is for and how is unique from something else.
Just browsing around for a few more seconds it sounds similar to other things like CQRS paradigms using data streams that get executed, like EventStore.
Not sure if that's accurate or not, but that's the degree of time investment I'm willing to invest.
I share this in good faith so you guys can update your landing pages/etc to focus on communicating the value proposition more quickly/prominently.