Hey HN community, Rohan this end, I am one of the founders of Datazip the company behind OLake. We have building OLake for more than a few years, and you may have seen it here couple times. In one line, OLake is a open source tool which replicates Databases (Postgres, MySQL, MSSQL, MongoDB, oracle and DB2), S3 and Kafka (it does support upserts for Kafka too).
We built it because we faced this problem first hand, stitching up 3 tools - Debezium, Kafka and Spark streaming just to get the data in Iceberg was not exactly easy and needed an entire team of 2 just to make sure its running smoothly over things like schema changes, databases restarts or just infra failures.
It writes through Glue, Hive, JDBC or REST catalogs, without Spark or a Kafka cluster in between, and has a web UI you start with docker compose or use Helm for scale.
The repo link is attached with the title. Happy to answer anything, especially if its about your experience working similar workloads!
rohankhameshra•23m ago
We built it because we faced this problem first hand, stitching up 3 tools - Debezium, Kafka and Spark streaming just to get the data in Iceberg was not exactly easy and needed an entire team of 2 just to make sure its running smoothly over things like schema changes, databases restarts or just infra failures.
It writes through Glue, Hive, JDBC or REST catalogs, without Spark or a Kafka cluster in between, and has a web UI you start with docker compose or use Helm for scale.
The repo link is attached with the title. Happy to answer anything, especially if its about your experience working similar workloads!