Diom includes implementations for common backend primitives such as cache, key-value, idempotency, rate-limiting, queues, and streams, with more on the way.
While building Svix, we had to reimplement the same backend primitives that everyone have to reimplement. We also constantly felt the tension between building something custom on top of existing infra (like Redis and Postgres) and adding more dedicated services (like RabbitMQ and Kafka) which we would then need to configure, operate, back up, and maintain. This was even worse for us because Svix is open-source, so additional infrastructure meant additional burden on our customers.
Six months ago we finally decided to build Diom, and focus on developer experience and ease of operation. It's open source, self-contained, and manages its own storage using fjall (a fast LSM-tree-based storage similar to RocksDB). It requires no external runtime dependencies (no redis/postgres/kafka/etc), and supports running as a single node or a highly-available Raft based cluster.
The goal of Diam is to provide developers with the backend primitives they need without having to write custom code on top of Redis, RabbitMQ, Kafka, or even need to run them at all. It currently supports cache, key-value, idempotency, rate-limiting, queues, and streams. We also plan on adding auth-tokens, distributed settings, feature flags, and other common components; as well as adding more functionality to existing components.
Diom favors ease of operation over scale, so it doesn't match Kafka-level throughput or very high QPS like Redis and Dragonfly. However, most products and developers don't process multiple terabytes and billions of events per second anyway. That said, Diom can still hit high performance for its target use-cases as it implements higher-level primitives rather than basic operations. Additionally, because the primitives live in the same process as the storage, there are fewer network round-trips, which keeps latency low.
It uses HTTP/2 with msgpack as the wire protocol (works fine from browsers), and ships a CLI and SDKs for Python, TypeScript, Rust, Go, and Java, with more on the way.
We have Svix fully ported to Diom and continuously running tests and simulated workloads in one of our staging environments. GA (general availability) is planned for later this year, once we've moved Svix production workloads over.
Repo (MIT licensed): https://github.com/svix/diom
Docs: https://docs.diom.com
Live playground: https://diom.com/playground
I'm excited to finally share Diom, and would love to hear what everyone thinks, and what other components you would like us to build! Would also love help figuring out what to call this. We currently say "component platform," but I'm not a fan of the name.