Most Raft/Paxos libraries treat the state machine as a black box. That’s fine until the state machine does real-world things like charging a card, sending an email or firing a webhook.
Then the leader crashes after the side effect but before commit, and you get duplicates. Every time. All the “exactly once” hand-waving collapses the moment a process dies mid-flight.
This repo is my attempt to fix that by design: side effects live in the replicated log, not behind it.
how it works, roughly:
Side effects are written as pending entries in replicated state. Only the current leader executes them, under a fencing token.
Leadership is fenced durably so a zombie leader can’t wake up and re-run old effects.
The state machine runs with a deterministic RNG seed + block time from the log, so replay is bit-for-bit or it halts.
The WAL is strict: CRC + hash-chained. Corruption stops the system instead of guessing.
Trade-offs are explicit:
Effects are at-least-once. Exactly-once requires stable IDs and sink-side dedup.
CP over AP. Safety over availability.
If you’ve ever been paged because a leader died at exactly the wrong microsecond, you know why I built this.
Repo: https://github.com/abokhalill/chr2
Happy to answer any questions about chr2 architectural internals or design trade offs.