A stream has cumulative acknowledgement, i.e I have read up to X offset on partition Y, if I restart unexpectedly please redeliver all messages since X. This means that if any message on Y is failing you can't update the committed offset X without a) dropping it into the ether or b) writing it to retry topic. b) sounds like a solution but it's really just kicking the can down the road because you face the same choice there until it ends up in a dead-letter topic that you send stuff that can't be automatically dealt with. In the literature this is called head of line blocking.

Queues are completely different. Usually instead of having a bunch of partitions with exclusive consumers you want a work-stealing approach that has consumers rip whatever work items they can get and stay as well fed as possible and be able to deal with failing items by Nack'ing them and sending them back to the queue. In order to facilitate this though the queue needs to implement the ability to selectively Ack(nowledge) messages and keep track of which messages haven't been successfully consumed.

This is easy with a traditional queuing system because they usually don't offer any ordering guarantees (or if they do they are per key or something and pretty loose) and they store the set of messages "yet to be delivered" rather than "all messages in order" like a streaming system does. This makes it trivial to acknowledge a message has been processed (delete it) or nack it (remove the processing lock, start a redelivery timer for it). Naturally though this means the ability to re-consume already acknowledged messages pretty much doesn't exist in most queue systems as they are long-gone once they have been successfully processed.

Mixing the two is the magic of Pulsar. It has the underlying stream storage approach, with it coming ordering properties and a whole bunch of stuff that is good for scaling and reliability but layers on a queue based consumption API by storing subscription state durably on the cluster i.e it tracks which individual messages have been Ack'd rather than offsets like Kafka consumer groups or similar APIs.

Building this yourself on Kafka/Redpanda is possible but it's extremely difficult to do correctly and you need to be very careful about how you store the subscription state (usually on a set of compacted topics on the cluster). I say this because I took this path in the past and I don't recommend it for anyone that isn't sufficiently brave. :)

But because other people use complex networking stacks in Kubernetes, that means for a bunch of good reasons, that apparently means it can’t be used without that stack? If you don’t need that functionality, and you have different requirements, then why would you implement a k8s stack that includes all that functionality? That would be foolish.

Everyone ships with kube-proxy because it’s the simplest way to provide an unopinionated network stack that will give you a fully functional k8s cluster with expected bells and whistles like simple service discovery. But most people quickly replace it with a CNI for various reasons.

My understanding is that your application is all about managing high throughput, direct to application TCP connections. So I have no idea why you’re talking about the need for fully fledged service discovery, or long lived TCP connections pinned to a single pod (I mean, that just the nature of TCP, got nothing to do with k8s or any of its network stacks).

> You can design a lean Kubernetes network path with minimal abstractions, but: You lose things like dynamic service discovery, load balancing, and readiness-based traffic shifting. You must manage more infra manually (e.g., configure iptables or direct routing rules yourself). You’re fighting against the grain of what Kubernetes is designed to do.

If you’re building a stack without k8s, then you don’t get any of those feature either, so not sure what the fuss is about. Doesn’t mean that’s k8s core competency as a workload schedular, and its general framework for managing stateful and stateless resources doesn’t provide a lot of value. You just need to learn how to take advantage of it, rather than following some cookie cutter setup for a stack that doesn’t meet your needs.