you can (and people already) model steps in any arbitrarily large workflow and have those results be processed in a modular fashion and have whatever process that begins this workflow check the state of the necessary preconditions prior to taking any action and thus go to the currently needed step, or retry ones that failed, and so forth.
Short answer: we need to be able to redeploy and bounce the orchestrator without worrying about what stage each running VM on our platform is in.
JP, the dev that built this out for us, talks a bit about the design rationale (search for "Cadence") here:
https://fly.io/blog/the-exit-interview-jp/
The library itself is open:
- Checking the status of a task (queued, pending, failed, cancelled, completed) - Cancelling a queued task (or pending task if the execution environment supports it) - Re-prioritizing queued tasks - Searching for tasks based off an attribute (e.g. tag)
You really do need a database for this.
It is a queue if we squint really hard, but it allows random access and reordering. Do we have durable structures of this kind?
I can’t imagine how to shoehorn this into Kafka or SQS.
Yup. Being able to write imperative code that automatically resumes where it left off is very valuable. It's best to represent durable turing completeness using modern approaches of authoring such logic - programming languages. Being able to loop, try/catch, apply advanced conditional logic, etc in a crash-proof algorithm that can run for weeks/months/years and is introspectable has a lot of value over just using queues.
Durable execution is all just queues and task processing and event sourcing under the hood though.
I recently started doing something very similar on Postgres [1] and I'm greatly enjoying using it. I think the total solution I ended up with is under 3000 lines of code for both the SQL and the TypeScript SDK combined, and it's much easier to use and to operate than many of the solutions on the market today.
Hope would you say it compares with pgqueuer?
(Disclosure: I work on DBOS [1]) The author's point about the friction from explicit step wrappers is fair, as we don't use bytecode generation today, but we're actively exploring it to improve DX.
There is value in such a wrapper/call at invocation time instead of using the proxy pattern. Specifically, it makes it very clear to both the code author and code reader that this is not a normal method invocation. This is important because it is very common to perform normal method invocations and the caller needs to author code knowing the difference. Java developers, perhaps more than most, likely prefer such invocation explicitness over a JVM agent doing byte code manip.
There is also another reason for preferring a wrapped-like approach - providing options. If you need to provide options (say timeout info) from the call site, it is hard to do if your call is limited to the signature of the implementation and options will have to be provided in a different place.
As stated in the post, I like how the proxy approach largely avoids any API dependency. I'd also argue that Java developers actually are very familiar with this kind of implicit enrichment of behaviors and execution semantics (e.g. transaction management is weaved into applications that way in Spring or Quarkus applications).
But there's also limits to this in regards to flexibility. For example, if you wanted to delay a method for a dynamically determined period of time, rather than for a fixed time, the annotation-based approach would fall short.
Also, since Temporal activity invocations are (often) remote, many times a user may only have the definition/contract of the "step" (aka activity in Temporal parlance) without a body. Finally, many times users _start_ the "step", not just _execute_ it, which means it needs to return a promise/future/task. Sure this can be wrapped in a suspended virtual thread, but it makes reasoning about things like cancellation harder, and from a client-not-workflow POV, it makes it harder to reattach to an invocation in a type-safe way to, say, wait for the result of something started elsewhere.
We did the same proxying approach for TypeScript, but we saw as we got to Python, .NET, and Ruby that being able to _reference_ a "step" while also providing options and having many overloads/approaches of invoking that step has benefits.
If you have any long-running operation that could be interrupted mid-run by any network fluke (or the termination of the VM running your program, or your program being OOMed, or some issue with some third party service that your app talks to, etc), and you don’t want to restart the whole thing from scratch, you could benefit from these systems. The alternative is having engineers manually try to repair the state and restart execution in just the right place and that scales very badly.
I have an application that needs to stand up a bunch of cloud infrastructure (a “workspace” in which users can do research) on the press of a button, and I want to make sure that the right infrastructure exists even if some deployment attempt is interrupted or if the upstream definition of a workspace changes. Every month there are dozens of network flukes or 5XX errors from remote endpoints that would otherwise leave these workspaces in a broken state and in need of manual repair. Instead, the system heals itself whenever the fault clears and I basically never have to look at the system (I periodically check the error logs, however, to confirm that the system is actually recovering from faults—I worry that the system has caught fire and there’s actually some bug in the alerting system that is keeping things quiet).
> I've written service tree management tools that do that sort of thing on a single host but not any kind of distributed system.
That’s essentially what Kubernetes is—a distributed process manager (assuming process management is what you are describing by “service tree”).
The style presented in this blog post doesn't suffer from those downsides. It's all done with local databases and pure language libraries, and is completely transparent to the user.
In the end I'm left wondering what the net benefit is over say an actor framework that more directly maps to the notion of long-lived state with occasional activity and is easier to test.
All that said some of the vendors have raised hundreds of millions of dollars so someone must believe in the idea.
I wrote a durable system that recovers from all sorts of errors (mostly network faults) without writing much error handling code. It just retries automatically, and importantly the happy path and the error path are exactly the same, so I don’t have to worry that my error path has much less execution than my happy path.
> but the part of that transaction that failed was “charging the customer” - did it fail before or after the charge went through?
In all cases, whether the happy path or the error path, the first thing you do is compare the desired state (“there exists a transaction exists charging the customer $5”) with the actual state (“has the customer been charged $5?”) and that determines whether you (re)issue the transaction or just update your internal state.
> once you’ve built sufficient atomicity into your system to handle the actual failure cases - the benefits of taking on the complexity of a DE system are substantially lower than the marketing pitch
I probably agree with this. The main value is probably not in the framework but rather in the larger architecture that it encourages—separating things out into idempotent functions that can be safely retried. I could maybe be persuaded otherwise, but most of my “durable execution” patterns seem to be more of a “controller pattern” (in the sense of a Kubernetes controller, running a reconciling control loop) and it just happens that any distributed, durable controller platform includes a durable execution subsystem.
For example, stripe lets you include an idempotency key with your request. If you try to make a charge again with the same key, it ignores you. A DE framework like DBOS will automatically generate the idempotency key for you.
But you're correct, if you can't make the operation idempotent, then you have to handle that yourself.
DBOS is tied to Postgres, right? That wouldn't scale anywhere near where we need either.
Sadly there aren't many shortcuts in this space and pretending there are seems a bit hip at the moment. In the end, mostly everyone who can afford to solve such problems are gonna end up writing their own systems for this.
I would challenge that assumption. We have 50 years of experience scaling Postgres. It can scale pretty far, and then you can shard it for even more. Or you can use one of the new flavors of Postgres compatible database that has unlimited horizontal scaling.
> In the end, mostly everyone who can afford to solve such problems are gonna end up writing their own systems for this.
Hard disagree (granted, I'm the CEO of one of the companies selling a solution in this space). If done right with a good DX and lightweight enough, ideally everyone will use DE by default, and will use one of the frameworks provided. Most likely one of the new style frameworks that you see in this blog post and that DBOS uses, that don't use an external coordinator and black box binary with a shim.
DBOS uses in process coordination with a pure language library, which makes it far more performant with a lot less hardware. It's not an apples to apples comparison.
I do still think there is sufficient amount of boilerplate to potentially justify some engine like this.
I think this is a gross misrepresentation of what durable executions are. DEs were never expected to magically eliminate the need to handle errors. What DEs do is provide an high-level abstraction of the same pattern that is recurrent on all workflow engines, and they provide a simpler way for developers to implement rollback and compensation steps when workflows fail.
If you are designing and implementing a transaction with a DE, you still need to design and implement a transaction. DEs simplify much of the logic, but you still need to design and implement a transaction. There is no silver bullet.
> Even the canonical examples everyone uses - so you’re using a DE engine to restart a sales transaction, but the part of that transaction that failed was “charging the customer” - did it fail before or after the charge went through? (...)
That's immaterial to the discussion on DEs. You, as a software engineer, still need to design and implement a transaction. DEs greatly simplify your job, but you still need to analyze failure modes and perform the necessary compensation steps.
> All of these problems are tractable, but once you’ve solved them - once you’ve built sufficient atomicity into your system to handle the actual failure cases - the benefits of taking on the complexity of a DE system are substantially lower than the marketing pitch.
I completely disagree, but you do you. Some durable execution engines greatly simplify tracking state and implementing activities and rollback logic. Some cloud providers even provide services that allow you to implement long-running workflows with function-as-a-service components that provide out-of-the-box support for manual approvals. If you feel you are better off rolling your own support, good for you. Meanwhile, everyone around you is delivering the same value with much less work.
I still cannot figure out how this is any different than launching a workflow in something like Airflow. Is the novel thing here that it can be done using the same DB you already have running?
The embedded database approach here is interesting though - low latency, no network calls, perfect for single-agent workflows. TPF assumed massive concurrent load across distributed terminals. Different problems, similar durability patterns.
Does anyone really do this?
The people I worked with were not specifically HN audience. Rather in the Java bubble in Germany-Austria-Switzerland which is also surprisingly a small world. If BPMN is not really needed, then I would also not use it nowadays. It increases complexity, and who knows if it makes project communication better at all.
Update: On the Camunda website there are 60 case-studies of customers/clients using BPMN, https://camunda.com/case-studies/. One of them has the teaser: "The 10th largest US Bank created an omnichannel onboarding platform that handles 12m process instances per year across 100 workflows". Now I have something to read for this Sunday evening.
Not really sure because "Lately, there has been a lot of excitement around Durable Execution (DE) engines" is also news to me, and AFAICT it seems to be yet-another-reinvention CS is famous for.
fiddlerwoaroof•2mo ago
andersmurphy•2mo ago
Janet lang lets you serialize coroutines which is fun. Make this sort of stuff trivial.
gunnarmorling•2mo ago
> You could think of [Durable Execution] as a persistent implementation of the memoization pattern, or a persistent form of continuations.
smitty1e•2mo ago
rileymichael•2mo ago