It's great to see my name mentioned here, cheered me up as I'm not having a good weekend. It feels like a lifetime ago since I got to work on this and parquet-rs.
Yeah, I agree with the conclusion on Kafka here. Kafka is a very resource-heavy application that's worth replacing with leaner options.
Oxbow seems very interesting, I presume it's designed to forward data that trickles into some S3 bucket as parquet data?
raverbashing•2mo ago
> The big argument against kafka-delta-ingest was Apache Kafka. If an organization has Kafka for other reasons, then kafka-delta-ingest can be a useful “sidecar” process to persist data flowing through Kafka. If however the organization is running Kafka just for ingestion, there are cheaper options available. As the organization evolved, the other consumers of Kafka drifted away, driving the value proposition of kafka-delta-ingest lower and lower.
Sounds like how most of Kafka stories end.
rmnclmnt•2mo ago
If only for the initiative driving the work behind delta-rs, I’d say it was well worth it!
redwood•2mo ago
What about WarpStream? Isn't that purpose built for lower Kafka ingest costs?
nevi-me•2mo ago
Yeah, I agree with the conclusion on Kafka here. Kafka is a very resource-heavy application that's worth replacing with leaner options.
Oxbow seems very interesting, I presume it's designed to forward data that trickles into some S3 bucket as parquet data?