I'm building Durable Execution without event replay.
I've been a fan of Durable Execution for years and have seen it have huge impact on projects, but I've also seen it confuse developers and create unusual development habits to avoid issues that are unique to this paradigm, particularly making code changes safely to workflows that are in-flight. This project is an experimental prototype to avoid all of those issues and present a durable runtime for code that runs simply and sequentially, like any other code.
It's early for the project, the docs and UX are rough, but the bones are solid and you can play with it. I'm sharing it now to gather early feedback on the approach and trade-offs involved. Let me know what you think.
mnorth•1mo ago
I'll add that there a couple key architectural choices in here aimed at problems I care about other than replay complexity:
1) This kind of technology IMO should be as easy to use as a task queue, so I've built it with Postgres as the only dependency. Most of the implementation is in Rust and invoked via native interfaces from the host language, which will allow the project to scale to support multiple languages without rewriting it for each.
2) I've seen teams struggle to understand when durable execution should replace simple background tasks. Reasonably, they would prefer not to maintain and monitor both a task queue framework and a platform for durable execution. But durable workflows are too heavy an abstraction for most background jobs, so you end up needing both. As I see it, durable execution is effectively just an orchestration layer for background tasks. My ideal would be that a platform could allow you to queue both simple tasks and complex workflows, and unify the scheduling capabilities and observability of both. So I'm doing that.
mnorth•1mo ago
I've been a fan of Durable Execution for years and have seen it have huge impact on projects, but I've also seen it confuse developers and create unusual development habits to avoid issues that are unique to this paradigm, particularly making code changes safely to workflows that are in-flight. This project is an experimental prototype to avoid all of those issues and present a durable runtime for code that runs simply and sequentially, like any other code.
It's early for the project, the docs and UX are rough, but the bones are solid and you can play with it. I'm sharing it now to gather early feedback on the approach and trade-offs involved. Let me know what you think.
mnorth•1mo ago
1) This kind of technology IMO should be as easy to use as a task queue, so I've built it with Postgres as the only dependency. Most of the implementation is in Rust and invoked via native interfaces from the host language, which will allow the project to scale to support multiple languages without rewriting it for each.
2) I've seen teams struggle to understand when durable execution should replace simple background tasks. Reasonably, they would prefer not to maintain and monitor both a task queue framework and a platform for durable execution. But durable workflows are too heavy an abstraction for most background jobs, so you end up needing both. As I see it, durable execution is effectively just an orchestration layer for background tasks. My ideal would be that a platform could allow you to queue both simple tasks and complex workflows, and unify the scheduling capabilities and observability of both. So I'm doing that.