frontpage.
newsnewestaskshowjobs

Made with ♥ by @iamnishanth

Open Source @Github

fp.

Open in hackernews

Show HN: Obelisk – a WASM-based deterministic workflow engine

https://obeli.sk/
103•tomasol•6mo ago
A lightweight engine for durable execution / deterministic workflows I built with Rust, wasmtime and the WASM Component Model. Its main use is running reliable, long-running workflows that can automatically resume after failures. Looking for feedback on the approach and potential use cases!

Comments

emgeee•6mo ago
This is a pretty cool idea but I'm trying to think of the advantage of WASM vs other execution engines.

It seems to me one of the main use-cases for WASM is to execute lambdas, which are often short-lived (like 500ms timeout limits). Maybe this could have a place in embedded systems?

tomasol•6mo ago
The biggest motivator for me is that WASM sandbox provides true deterministic execution. Contrary to engines like temporal, using hashmaps is 100% deterministic here. Attempting to spawn a thread is a compile error. It also performs well - the bottleneck is in the write throughput of sqlite. Last but not least - all the interfaces between workflows and activities are type safe, described in a WIT schema.
jcmfernandes•6mo ago
Somewhat similar to Golem - https://github.com/golemcloud/golem - correct?

So, I like this idea, I really do. At the same time, in the short-term, WASM is relatively messy and, in my opinion, immature (as an ecosystem) for prime time. But with that out of the way (it will eventually come), you'll have to tell people that they can't use any code that relies on threads, so they better know if any of the libraries they use does it. How do you foresee navigating this? Runtime errors suck, especially in this context, as fixing them requires either live patching code or migrating execution logs to new code versions.

tomasol•6mo ago
Yeah, looks like Golem went similar route - using WASM Component Model and wasmtime.

There is always this chicken and egg problem on a new platform, but I am hoping that LLMs can solve it partially - the activities are just HTTP clients with no complex logic.

Regarding the restrictions required for determinism, they only apply to workflows, not activities. Workflows should be describing just the business logic. All the complexities of retries, failure recovery, replay after server crash etc. are handled by the runtime. The WASM sandbox makes it impossible to introduce non-determinism - it would cause a compile error so no need for runtime checks.

jcmfernandes•6mo ago
I understand what you mean by being able to fully sandbox things and guarantee determinism, a must for the workflows and not the activities (using temporal lingo).

When you say that the runtime handles, for example, retries, doesn't that require me to depend on your HTTP client component? Or do I also need to compile activities to WASM and have obelisk running them because they are essentially background jobs (that is, you have workers pulling)?

Finally, do you see the component's interface as the right layer for capturing IO? I'm imagining people attempting to run managed code (Java, python, ruby, etc.). The VMs can do thousands of syscallls before they start executing they user's code. Logging them one by one seems crazy, but I also don't see an alternative.

EDIT:

I RTFM and found the answers to my first two questions in the README :)

tomasol•6mo ago
> do I also need to compile activities to WASM

Yes, currently all activities must conform to the WASI 0.2 standard. This is the simplest for deployment, as you only need the obelisk executable, toml config file. The webhooks, workflows and activities pulled from a OCI registry on startup.

To support native code I plan to add external activities as well, with an interface similar to what Netflix Conductor uses for its workers.

> Finally, do you see the component's interface as the right layer for capturing IO?

An activity must encapsulate something much higher level than a single IO operation. So something like "Configure BGP on a router", "Start a VM" etc. It needs to be able to handle retries and thus be idempotent.

Regarding performance, a workflow execution can call 500-700 child executions serially, or around 1400 child executions concurrently per second.

jcmfernandes•6mo ago
> An activity must encapsulate something much higher level than a single IO operation. So something like "Configure BGP on a router", "Start a VM" etc. It needs to be able to handle retries and thus be idempotent.

I was referring to the workflows, that is, writing the workflows in managed languages, not the activities.

Out of curiosity, are you working full-time on this? I'm working part-time on the same problem, looking to go full time soon, and it's interesting to see how the same ideas are popping up somewhat independently across different projects :) let me know if you're interested in chatting!

tomasol•6mo ago
> I was referring to the workflows, that is, writing the workflows in managed languages, not the activities.

Ah understood. I have no plans supporting native workflow executors.

> Out of curiosity, are you working full-time on this?

Yes, currently I work on it full time.

> I'm working part-time on the same problem, looking to go full time soon, and it's interesting to see how the same ideas are popping up somewhat independently across different projects :)

Nice website! I also see the ideas of determinism, replayability etc more and more.

> let me know if you're interested in chatting!

Sure, my email is visible in the Git commit history.

AlotOfReading•6mo ago
WASM isn't quite deterministic. An easy example is NaN propagation, which can be nondeterministic in certain circumstances. Obelisk itself seems to allow nondeterminism via the sleep() function. Just create a race condition among a join set. I imagine that might even get easier once the TODO to implement sleep jitter is completed.

It's certainly close enough that calling it deterministic isn't misleading (though I'd stop short of "true determinism"), but there's still sharp edges here with things like hashmaps (e.g. by recompiling: https://dev.to/gnunicorn/hunting-down-a-non-determinism-bug-...).

genuine_smiles•6mo ago
> An easy example is NaN propagation, which can be nondeterministic in certain circumstances.

Which circumstances?

xmcqdpt2•6mo ago
See for example

https://github.com/WebAssembly/design/issues/1463

In general, if NaN1 and NaN2 are different (there are 23 bits in a NaN that can be set to an arbitrary value, the NaN payload) then combining them isn’t deterministic. NaN1 + NaN2 might produce NaN1 or NaN2 depending on the processor model, instructions etc. IEEE754 only says that the result must be one of the two input NaNs (or at least it did last time I checked the standard.)

In practice, NaN payloads are seldomly used so it doesn’t matter much. NaN canonicalization involves transforming all NaNs to specific NaN values, and AFAIK it’s expensive because technically you need to check for NaN all the time.

tomasol•6mo ago
Thanks for bringing that up. Regarding the NaN canonicalization, there is a flag for it in wasmtime [1], I should probably make sure it is turned on.

Although I don't expect to be an issue practically speaking, Obelisk checks that the replay is deterministic and fails the workflow when an unexpected event is triggered. It should be also be possible to add an automatic replay of each finished execution to verify the determinism e.g. while testing.

[1] https://docs.rs/wasmtime/latest/wasmtime/struct.Config.html#...

Edit: Enabling the flags here: https://github.com/obeli-sk/obelisk/pull/67

tomasol•6mo ago
> Just create a race condition among a join set.

All responses and completed delays are stored in a table with an auto-incremented id, so the `-await-next` will always resolve to the same value.

As you mention, putting a persistent sleep and a child execution into the same join set is not yet implemented.

AlotOfReading•6mo ago
I get that, the nondeterminism would come from the completion order of the join set. If the children sleep appropriately, they'll race to be inserted after completing, and order of the result set will depend on the specifics of the implementation. It's possible this could happen deterministically, but probably not reasonably.
tomasol•6mo ago
Sorry for the late reply.

The actual order in which child workflows finish and their results hit the persistence layer is indeed nondeterministic in real-time execution. Trying to force deterministic completion order would likely be complex and defeat the purpose of parallelism, as you noted.

However, this external nondeterminism is outside the scope of the workflow execution's determinism required for replay.

When the workflow replays, it doesn't re-run the race. It consumes events from the log. The `-await-next` operation during replay simply reads the next recorded result, based on the fixed order. Since the log provides the same sequence of results every time, the workflow's internal logic proceeds identically, making the same decisions based on that recorded history.

Determinism is maintained within the replay context by reading the persisted, ordered outcomes of those nondeterministic races.

disintegrator•6mo ago
Really nice project. What’s the reasoning behind the AGPL licensing. My understanding is that it will hurt adoption unless you’re planning to offer paid licensing options? Either way it’s a really nice project and I’m keen to try it out. I’ve found it tricky to get a WASM/WASI setup where I can at least my http requests (probably my own skill issue).
tomasol•6mo ago
Thanks for the kind words. In an ideal world I would like to offer a cloud version that would be monetized. There are a few examples on how to do HTTP requests, I have a demo repository [1] with GraphQL and regular JSON-over-HTTP activities. I do agree that the ecosystem is not mature yet, but I was able to generate HTTP activities using LLM on a single shot.

1: https://github.com/obeli-sk/demo-stargazers

SvenL•6mo ago
One issue I had many time with workflow engines was updates. I have a workflow and it has already running instances. 2 scenarios:

Can I update the workflow while it has running instances without interfering the running instances?

Can I update a running instance with a new version of the workflow to patch some flaw? If no, can I replay an updated version of a workflow with the log of an old workflow version?

tomasol•6mo ago
Great questions. If you are fixing a bug in a workflow, which has running executions, there are two scenarios:

Either the fix does not break the determinism, meaning the the execution did not hit the fix yet. In this case the execution can be replayed and continue on the patched WASM component.

Otherwise, the execution replay causes "Non determinism detected" error. In this case you need to handle the situation manually. Since the execution log is in a sqlite file, you can select all execution affected by the bug and perform a custom cleanup. Also you can create a "forked" execution just by copying the execution log + child responses into a new execution, however there is no API for it yet.

> Can I update the workflow while it has running instances without interfering the running instances?

If you mean keep the in-progress executions on the old version of the code, you can do that by introducing a new version in the WIT file and/or change the new function name.

halamadrid•6mo ago
We are using a workflow engine called Unmeshed - which has what you are asking about. Workflow definitions can be updated without running interfering with running instances and if you choose to you can patch updates on to running workflows. And you can also rerun workflows with the same input from an older execution.
Philpax•6mo ago
I believe https://flawless.dev/ is another implementation with a very similar technology stack. I'd love to know how you compare and what the key differences are!
tomasol•6mo ago
Indeed. I cannot compare the implementations as flawless is not open source. However on a high level they both share the same philosophy.

I believe the biggest difference is that Obelisk relies on the WASM Component Model:

Obelisk aims to avoid vendor lock-in. It is possible to write activities, workflows and webhooks with no obelisk SDK. Activities and webhooks are WASI 0.2 components that can be run on any compatible runtime like wasmtime e.g. for testing. This should also help with the adoption as any runtime will need a ton of integrations.

jusonchan81•6mo ago
I’m not sure there is much risk in vendor lock in. Look at Temporal. If you use the SDK you are probably locked in for life.
euroderf•6mo ago
Would it be possible to see an example using Go ? Admittedly, docu for Go in the Component Model is pretty uneven.
tomasol•6mo ago
I can do that. Please create an issue if you want to be notified about it.
chaosprint•6mo ago
I have a similar idea but for audio effect and music production:

https://github.com/wasm-audio/wasm-audio-examples

For workflow, how do you think of async in wit at this moment?

tomasol•6mo ago
The structured concurrency paradigm in workflows is stricter to what I'm reading here [1]. The whole execution model is different from the Task [2], as workflows are transparently unloaded and replayed.

Obelisk has a concept called join sets [3], where child executions are submitted and awaited. In the future I plan on adding cancellation and allow custom cleanup functions.

[1] https://github.com/WebAssembly/component-model/blob/main/des...

[2] https://github.com/WebAssembly/component-model/blob/main/des...

[3] https://obeli.sk/docs/latest/concepts/workflows/join-sets/

Show HN: Dlog – Journaling and AI coach that learns what drives wellbeing (Mac)

https://dlog.pro/
21•dr-j•7h ago•8 comments

Show HN: Erdos – open-source, AI data science IDE

https://www.lotas.ai/erdos
51•jorgeoguerra•8h ago•25 comments

Show HN: Git Auto Commit (GAC) – LLM-powered Git commit command line tool

https://github.com/cellwebb/gac
42•merge-conflict•7h ago•29 comments

Show HN: JSON Query

https://jsonquerylang.org/
85•wofo•8h ago•55 comments

Show HN: Write Go code in JavaScript files

https://www.npmjs.com/package/vite-plugin-use-golang
139•yar-kravtsov•19h ago•40 comments

Show HN: MyraOS – My 32-bit operating system in C and ASM (Hack Club project)

https://github.com/dvir-biton/MyraOS
230•dvirbt•1d ago•53 comments

Show HN: Easily visualize torch, Jax, tf, NumPy, etc. tensors

https://github.com/hardik-vala/tensordiagram
2•HardikVala•3h ago•0 comments

Show HN: Action Engine — An API/Agent Buildkit Putting Flexibility First

https://actionengine.dev/
2•helenapankov•4h ago•2 comments

Show HN: nblm - Rust CLI/Python SDK for NotebookLM Enterprise automation

https://github.com/K-dash/nblm-rs
8•K-dash•9h ago•0 comments

Show HN: Helium Browser for Android with extensions support, based on Vanadium

https://github.com/jqssun/android-helium-browser
61•jqssun•1d ago•27 comments

Show HN: Vetr.is – Privacy-First Cloud in Iceland

https://vetr.is
3•falkensmaze66•5h ago•0 comments

Show HN: LinkPatrol – Free merchant-agnostic tool to find broken affiliate links

https://www.linkpatrol.in
2•deepu_bansal87•7h ago•1 comments

Show HN: Relai-SDK – simulate → evaluate → optimize AI agents

https://github.com/relai-ai/relai-sdk
4•sfeizi•8h ago•0 comments

Show HN: Shadcn/UI theme editor – Design and share Shadcn themes

https://shadcnthemer.com
132•miketromba•2d ago•40 comments

Show HN: Omnia OS, the Most Efficient Email Client Without AI

https://omniaos.co/
5•jmchugh9•8h ago•0 comments

Show HN: OpenSkills - Run Claude Skills Locally Using Any LLM

https://github.com/BandarLabs/open-skills
5•mkagenius•9h ago•0 comments

Show HN: Whatdidido – CLI to summarize your work from Jira/Linear

https://github.com/oliviersm199/whatdidido
5•olism•9h ago•0 comments

Show HN: ChatHawk – Stop Copy-Pasting the Same Question Across Every AI Model

https://chathawk.co
4•chadlad101•10h ago•0 comments

Show HN: Create-LLM – Train your own LLM in 60 seconds

https://github.com/theaniketgiri/create-llm
49•theaniketgiri•1d ago•35 comments

Show HN: Learn Basic Chess Movements

https://www.minichessgames.com/?hn
11•patrickdavey•1d ago•0 comments

Show HN: Pinpam, TPM2-backed pin authentication for Linux

https://github.com/RazeLighter777/pinpam
3•razighter777•10h ago•0 comments

Show HN: MNML – Android Launcher (Open Testing Release)

4•tuxxness•5h ago•2 comments

Show HN: A minimalist, no-clout social network with chronological feed

2•cornfieldlabs•10h ago•0 comments

Show HN: Diagram as code tool with draggable customizations

https://github.com/RohanAdwankar/oxdraw
250•RohanAdwankar•2d ago•59 comments

Show HN: LLM Rescuer – Fixing the billion dollar mistake in Ruby

https://github.com/barodeur/llm_rescuer
92•barodeur•3d ago•14 comments

Show HN: MCP Agent Mail, Like Gmail for Coding Agents

https://github.com/Dicklesworthstone/mcp_agent_mail
3•eigenvalue•12h ago•0 comments

Show HN: Write JavaScript code that Runs on RUST hits 1M req/s

https://shyam20001.github.io/rsjs/
5•StellaMary•14h ago•1 comments

Show HN: The Legal Embedding Benchmark (MLEB)

https://huggingface.co/blog/isaacus/introducing-mleb
5•ubutler•1d ago•0 comments

Show HN: Chonky – a neural text semantic chunking goes multilingual

https://huggingface.co/mirth/chonky_mmbert_small_multilingual_1
42•hessdalenlight•2d ago•5 comments

Show HN: MacOS Live Screensaver – A screensaver that plays live video streams

https://github.com/hauxir/macos-live-screensaver
64•hauxir•6d ago•42 comments