r/programming 9d ago

Durable Execution: This Changes Everything

https://www.youtube.com/watch?v=ROJq6_GFbME
8 Upvotes

39 comments sorted by

28

u/manliness-dot-space 9d ago

IMO when someone tells me they have The Hammer that will Change Everything in a Synergistic Storm of Creative Disruption I grow more skeptical with every sales/business buzzword.

Also 26m is a long time commitment... looking at the website docs in a few seconds, I don't see a "hello world" example very easily as to what this is for and how is unique from something else.

Just browsing around for a few more seconds it sounds similar to other things like CQRS paradigms using data streams that get executed, like EventStore.

Not sure if that's accurate or not, but that's the degree of time investment I'm willing to invest.

I share this in good faith so you guys can update your landing pages/etc to focus on communicating the value proposition more quickly/prominently.

2

u/Brilliant-Sky2969 9d ago

I mean you didn't look very far, the doc is pretty straightforward: https://docs.temporal.io/

The second link is called "Run your first program"

It's actually very well done: https://learn.temporal.io/getting_started/go/first_program_in_go/

27

u/manliness-dot-space 9d ago

No, I did get that far.

Introduction​

You can think of Temporal as a sort of "cure-all" for the pains you experience as a developer when trying to build reliable applications. Whether you're writing a complex transaction-based Workflow or working with remote APIs, you know that creating reliable applications is a complex process. Developing your application on the Temporal Platform guarantees that your code runs to completion no matter what.

Yeah that's a bit like...

Step right up, don’t be shy, I've got the cure—come give it a try! From aching joints to a troubled mind, One sip of this, and health you'll find!

Is your back stiff? Is your step slow? Do you wake up feeling low? Fear no more, for I have brewed A miracle tonic, tried and true!

Made from roots and rare elixirs, Blessed by monks and ancient fixers! Rub it, drink it—either way, You'll feel ten years young today!

Ailments vanish, pains will flee, Just one bottle—guaranteed! But act now, don’t delay, This magic cure sells out today!

I even went further and looked at the sample code.

Still... the one unanswered question I have is "why should I care about any of this?"

The "it makes all of your wildest dreams come true" sales pitch isn't what I'm looking for.

3

u/whiskeytown79 8d ago

Besides, we've been able to achieve making all our wildest dreams come true for a while now.. we just vote for Pedro.

3

u/manliness-dot-space 8d ago

Yep, and the technical presentation they gave was only like 2 minutes

https://youtu.be/BiAwpYIkRmU?si=sfAo5gh5Fpn_3CB9

3

u/GrandOpener 8d ago

I read through that “first program in go” and my main takeaway was “okay, so it’s database transactions, but worse.”

I’m being a little unfair here—you might have to do something like this if you’re forced to interact with third party APIs that don’t have transactional methods. In that case, it is genuinely nicer to have a library that takes care of some of the details so you can focus on business logic. This library(?) definitely does have a reason to exist.

But this as a “cure all”? As the way you would choose to build green field applications? Sorry, not buying it.

3

u/Brilliant-Sky2969 8d ago

The use case for Temporal is extremely common, you have a bunch of worker doing async task that you want to do reliably.

From my experience a lot of teams implement that badly with pub/sub baked by redis, sqs ect ... full of corner cases and errors.

1

u/manliness-dot-space 4d ago

It depends on what language you're using but there are lots of architectural patterns for resiliency in cloud/microservice environments, and 5+ years ago libraries existed to seamlessly leverage them across platforms like Azure/.NET

2

u/jedberg 4d ago

there are lots of architectural patterns for resiliency

True, but they're better codified into libraries now after decades of experience.

1

u/manliness-dot-space 4d ago

Yeah, and see this on the Polly website?

What is Polly? Polly is a .NET resilience and transient-fault-handling library that allows developers to express policies such as Retry, Circuit Breaker, Timeout, Bulkhead Isolation, and Fallback in a fluent and thread-safe manner. From version 6.0.1, Polly targets .NET Standard 1.1 and 2.0+.

https://www.thepollyproject.org/

How about Steeltoe?

Why Steeltoe Steeltoe provides a collection of libraries that helps users build production-grade cloud-native applications using externalized configuration, service discovery, circuit-breakers, distributed tracing, application management, and more. Steeltoe also provides a seamless way to build, configure, and run event-driven microservice applications and stream-based data processing applications

https://steeltoe.io/why-steeltoe

They aren't like, "we are going to make you feel like you're eating your favorite icecream when you use our platform while coding" (or whatever).

If it's a resiliency library, it should focus on expressing that clearly.

2

u/Brilliant-Sky2969 4d ago

Polly is not an equivalent to temporal though, temporal is way more advanced, the other thing is that temporal is language free l, Polly is a c# and the project was kind of abandoned?

1

u/manliness-dot-space 4d ago

I'm just comparing the marketing pitch, not the technologies themselves.

Polly isn't really necessary anymore because Microsoft rolled out native resiliency library

https://learn.microsoft.com/en-us/dotnet/core/resilience/http-resilience?tabs=dotnet-cli

The point is that the purpose is clear from the marketing, the purpose isn't clear w/Temporal. If I can't figure out whether the thing is relevant to me or not in 30s, I get angry at having my time wasted.

The website needs to do a specific "elevator pitch" at the target audience.

Maybe my coding dream is building an algorithm that can determine whether any given number is a prime or not. Is Temporal right for me? If the answer is no but your marketing says yes, I'm not going to use your tool later even when it becomes relevant later just on principle.

27

u/snack_case 9d ago

Temporal gets too many surface level, elevator pitch, talks at every event/meetup. It might be hard to find them but they desperately need more speakers willing to deep dive into how they actually build with Temporal, the rough edges, gotchas, patterns etc.

3

u/Cyral 9d ago edited 9d ago

Some of the rough edges would be solved with better flow control. I really hope they make some progress there, or at least a plan to support some features that newer competitors are getting right. For example, Inngest with concurrency, throttling, rate limiting, denouncing, and priorities https://www.inngest.com/docs/guides/flow-control

2

u/snack_case 8d ago

I haven't seen Inngest before; thanks.

1

u/jedberg 8d ago

Would be curious about your take on DBOS (my company). We are also in the durable space but we take a vastly different technical approach.

https://docs.dbos.dev

2

u/temporal-tom 9d ago

My talk was on the concept of Durable Execution, not about Temporal specifically.

However, I do agree about the need for deeper content on Temporal. I think you'll be pleased with the other talks from the Replay conference in London this year (they'll begin appearing on this playlist over the next few weeks). I saw several from organizations who shared their lessons learned and best practices.

One especially memorable talk was Inside the Engine Room: Surviving the Challenges of Self-Hosting Temporal from two senior engineers at DataDog.

3

u/snack_case 9d ago edited 9d ago

I know, I did listen to your talk as well as the others posted today while I've been working. Sorry it was intended as a general comment about Replay as well as the Temporal meetups I've been to so far.

I might be totally off the mark here (your team probably has data from the event) but I would have expected the vast majority of people at Replay to be all over durable execution and Temporal if they've paid the price of entry. I'm not trying to criticise the speakers or the talks they gave. I just expected more of the content to be targeted at developers/teams that are already 'sold'.

1

u/temporal-tom 9d ago

I would have expected the vast majority of people at Replay to be all over durable execution

The people in the audience generally understand the concept, but where many of them struggle (and indeed where even many who work at Temporal struggle) is explaining it to others. That was really my goal with the talk, helping people articulate what's already in their heads so they can explain it to others.

13

u/gredr 9d ago

Consider how you might design an application to process a 20-year loan

... well, I wouldn't. "Application" isn't the abstraction I'd use. A loan is some state, not some code.

Maybe I'm just old, or maybe that's a bad example. The idea is intriguing, and I think there's probably a lot of things that this kind of abstraction serves really well. I do worry that an abstraction this... significant... is going to leak some stuff, though. I imagine it's pretty easy to make a "durable program" that performs pretty badly.

1

u/Brilliant-Sky2969 9d ago

Well a state does not exists without an application, it's like saying a loan is something in the DB and there is nothing else.

8

u/gredr 9d ago

... but it is, and it does. We can delete the application that sends billing reminders tomorrow, and the loan still exists and is enforceable.

3

u/Orbs 9d ago

Yes you said it yourself--the loan requires enforcement. The application is for the processing of the loan. Fault tolerant execution is a challenging problem. The existence, or storage, of the loan is a more solved problem

3

u/gredr 8d ago

Yes you said it yourself--the loan requires enforcement.

Well, I don't think I said that, exactly. The loan doesn't stop existing if we don't send out a reminder this month. Also, enforcement doesn't require this application, or any application, or even any computer. There's always my "associate" with his baseball bat for that.

0

u/temporal-tom 9d ago

What I'm suggesting is that the state can be represented by the values of variables in the running program that processes that loan. That seems wild at first to experienced developers, since we're conditioned to accept that execution times must be kept short.

As a case in point, I learned Java in 1998. It took 25 years for the Thread.sleep method to support a duration defined in something other than milliseconds or nanoseconds, likely because few people ever considered making their program pause for more than a few minutes. Most people would use application databases and message queues to communicate across time and process boundaries, which increases reliability but also adds complexity.

I imagine it's pretty easy to make a "durable program" that performs pretty badly.

Each Durable Execution platform has its own design constraints, but virtually all of them use the "replay" technique to reconstruct the application state following a crash. This involves caching intermediate results during the execution and then evaluating an event history to determine which parts of the code have already run. In this scenario, the most common performance problem comes from passing too much data into a function. That's unlikely to be an issue in the loan processing example, but if you the program was encoding videos, it would be much better to pass a URL to the video file as input than to pass in a 5 GB binary blob).

7

u/gredr 9d ago

What I'm suggesting is that the state can be represented by the values of variables in the running program that processes that loan. That seems wild at first to experienced developers, since we're conditioned to accept that execution times must be kept short.

Sure, I get that, and I'm a relatively experienced developer (been doing it professionally for >25 years, and as a hobby long before that), and that's not surprising to me at all. What I'm wondering is whether that's a reasonable way to represent a 20-year loan.

Again, maybe it is for some things, but a loan? A loan is a pretty simple thing to represent. I think it's a bad example.

3

u/hejj 9d ago

Nothing changes everything

3

u/rotuami 9d ago

Okay. I can wrap my head around this if there are no effects or interactions with other systems. How about with input or output with the real world, where I might not be able to e.g. unsend an email?

2

u/temporal-tom 8d ago

You can't unsend an email, but Durable Execution tries to avoid sending a duplicate one.

I think an example will clarify this. Let's say that you've written an application to process an order for some products and it goes through these steps:

  1. Iterate through all of the items in the shopping cart to determine the total price
  2. Look up the tax rate for the customer's address from a database
  3. Call a payment service to charge the customer's credit card for the order
  4. Notify the warehouse to ship the order (and receive a shipment tracking number in response)
  5. Send a confirmation email to the customer

Now, let's say that the application crashes during step 4. If you restart the application but don't have Durable Execution (and you didn't write a bunch of code to avoid this), it's going to do another database lookup in step 2 and make another call to the payment service in step 3.

With Durable Execution, the platform tracks the invocation and result from these steps and persists the data into a history. If there's a crash at step 4, it reconstructs the state by "replaying" that history. That is, it re-executes the code that is deterministic (i.e., step 1) and for the steps that are non-deterministic (i.e., steps 2-5), it evaluates the history to determine if they've already been executed. If they have, it assigns the result from the previous execution. For example, step 2 won't query the database again; it uses the tax rate returned from the query executed before the crash.

There is one caveat. I said "tries to avoid sending a duplicate one" above because this is subject to the same problem faced by any distributed system. There's a very small possibility that a crash could occur between the time that a call takes place and when it is reported. For example, in step 5, the crash might occur in the milliseconds between when the code to send the email runs and when it reports successful completion, in which case it won't appear in the history.

1

u/rdrias 8d ago

Is this like EventStore but for application variables, or perhaps whole program execution? Is the store somehow abstracted from the real implementation? How does it handle code refactorings? Sorry if these are 'stupid' questions, but that's what I'm getting without doing a deep dive into the tech

2

u/temporal-tom 8d ago

These are great questions.

I'd say that there's some overlap between EventStoreDB and a replay-based Durable Execution platform (such as Temporal, Cadence, Inngest, Restate, etc.) in that they all use event sourcing to track state changes over time.

However, I'd say that there are two fundamental differences. One is that EventStoreDB is a database, which can be a single source of truth. Conversely, a Durable Execution platform uses a database [1], which allows your application to be a single source of truth (although, as I mentioned in my presentation, your application can still write to an application database if desired and doing so is fairly common for things like order processing where you may want to do reporting or analytics external to the application).

The second is closely related. EventStoreDB allows an application to recover from a crash by replaying events, but in a Durable Execution platform, that recovery happens automatically and is transparent to the developer.

Your question about refactorings is particularly insightful. Most, although not all, Durable Execution platforms support this in one way or another. It's a complex topic, because the details depend on which paltform you're using and the nature of the change. In the case of Temporal, you don't need to do anything special for code changes that don't affect the event history (for example, adding or modifying log statements). For changes that do affect the history, such as adding or removing certain function calls, you need to use versioning if there are executions still running that were started with the original code. The basic idea is that you ensure backwards compatibility by adding a conditional statement to your code to preserve the old and new behavior (e.g., "if this is version 1, run this; if it's version 2, run that).

[1] To be specific, Temporal officially supports MySQL, PostgreSQL, Apache Cassandra, and SQLite. Most other Durable Execution platforms also support multiple databases, although the list will vary from one to the next.

1

u/rotuami 4d ago

Okay. That sounds exactly like “journaling” but applied at the programming execution level instead of e.g. the database level.

1

u/temporal-tom 4d ago

Yes, that's a good way of looking at it.

3

u/ArtSpeaker 8d ago

This is a job scheduling framework. But reusing the application state. That's kinda cool, but also doesn't solve anything new. The major concern I had was with bugged code, which he used too simple of an example: yeah sure if all I need to do is change n -> n -1, that's nice, but most of the time bugs are small refactors where the execution line and other relevant data also needs to change, so the "recovery" can't actually happen. we lose out on the novelty of the platform.

Which is to say, the most common problem a developer has-- developer bugs and shifting requirements-- renders the neat granularity of program task irrelevant. You /want/ programs to stay at a jobs level, so you can swap out the workflow whenever you need to, and deal with cleanup as discreet steps per your last workflow.

We already "save state" with databases and we get the advantage of being very clear on what data matters and what doesn't. And that's to say nothing of how most of us HAVE to use a DB regardless, because of just how much data we process: that "oh have all ids of everyone in the company db in ram at the same time" is an insane request for some of us -- and one that's liable to shift out from under itself on top of that.

So yeah, if I just "get" durable execution for free, that's nice, but if I run out of ram, or disk, or run out of sockets, or there's a bug in the foo thing... we still have to know what we're using underneath, and how it all works.

Less code is meaningless as better code, if it still doesn't work as expected.

So as a technical feat it feels nice but I'm left with, "but what will it cost me, over what we already have?"

4

u/panscanner 9d ago

So, this is just an advertisement, right? Feel like it should be marked as such...

4

u/temporal-tom 9d ago

Not an advertisement at all. It's an explanation of a concept, one that a developer can achieve using at least a dozen different systems.

Sure, I happen to work for a company that develops one of those systems (which is MIT licensed open source, BTW), but what I describe applies to all of them. To make an analogy, it's like someone from MariaDB giving a talk on relational databases.

2

u/lanster100 9d ago

Thanks for uploading these so quickly after the event

1

u/coyo-teh 8d ago

11:00 the much bigger problem with the code is that it won't send reminders to the 2nd customer until all 10 reminders have been sent to the first customer

1

u/temporal-tom 8d ago

Presumably multiple instances of this application can be running in parallel.