Kafka for communication between microservices, and MQTT (VerneMQ) for IOT devices

Solid queue in rails

Sidekiq, Sidekiq, Sidekiq (or just Postgres if Im dealing with something trivial)

NATS

SQS. For Ruby apps I use Shoryuken with SQS.

Surprised no body is mentioning ActiveMQ!

I have so far gotten by plenty well writing my own queue systems to fit the needs of the consuming application. Normally the only place where I need queue systems is in distributed systems with rapid fire transmissions to ensure messages hit the network in time sequence order. The additional benefit is that network traffic is saved in order when the current network socket fails so that nothing is lost but time.

I played with most message queues and I go with RabbitMQ in production.

Mostly because it has been very reliable for years in production at a previous company, and doesn’t require babysitting. Its recent versions also has new features that make it is a descent alternative to Kafka if you don’t need to scale to the moon.

And the logo is a rabbit.

A cron job did thd work.

For large applications in a service-oriented architecture, I leverage Kafka 100% of the time. With Confluent Cloud and Amazon MSK, infra is relatively trivial to maintain. There's really no reason to use anything else for this.

For smaller projects of "job queues," I tend to use Amazon SQS or RabbitMQ.

But just for clarity, Kafka is not really a message queue -- it's a persistent structured log that can be used as a message queue. More specifically, you can replay messages by resetting the offset. In a queue, the idea is once you pop an item off the queue, it's no longer in the queue and therefore is gone once it's consumed, but with Kafka, you're leaving the message where it is and moving an offset instead. This means, for example, that you can have many many clients read from the same topic without issue.

SQS and other MQs don't have that persistence -- once you consume the message and ack, the message disappears and you can't "replay it" via the queue system. You have to re-submit the message to process it. This means you can really only have one client per topic, because once the message is consumed, it's no longer available to anyone else.

There are pros and cons to either mechanism, and there's significant overlap in the usage of the two systems, but they are designed to serve different purposes.

The analogy I tend to use is that Kafka is like reading a book. You read a page, you turn the page. But if you get confused, you can flip back and reread a previous page. An MQ like RabbitMQ or Sidekiq is more like the line at the grocery store: once the customer pays, they walk out and they're gone. You can't go back and re-process their cart.

Again, pros and cons to both approaches.

"What didn't work out?" -- I've learned in my career that, in general, I really like replayability, so Kafka is typically my first choice, unless I know that re-creating the messages are trivial, in which case I am more inclined to lean toward RabbitMQ or SQS. I've been bitten several times by MQs where I can't easily recreate the queue, and I lose critical messages.

"Where did you regret adding complexity?" -- Again, smaller systems that are just "job queues" (versus service-to-service async communication) don't need a whole lot of complexity. So I've learned that if it's a small system, go with an MQ first (any of them are fine), and go with Kafka only if you start scaling beyond a single simple system.

"And if you stuck with a DB-based queue -- did it scale?" -- I've done this in the past. It scales until it doesn't. Given my experience with MQs and Kafka, I feel it's a trivial amount of work to set up an MQ/Kafka, and I don't get anything extra by using a DB-based queue. I personally would avoid these, unless you have a compelling reason to use it (eg, your DB isn't huge, and you can save money).

UUCP

Does Google Cloud Tasks count?

Pulsar

No one ever seems to use it, but for AMPQ I like Beanstalkd. It’s fast, stable and has not failed me under high RPS.

Kafka is fairly different from the rest of these — it’s persistent and designed for high read throughput to multiple simultaneous clients at the same time, as some other commenters have pointed out.

We wanted replayability and multiple clients on the same topic, so we evaluated Kafka, but we determined it was too operationally complex for our needs. Persistence was also unnecessary as the data stream already had a separate archiving system and existing clients only needed about 24hr max of context. AWS Kinesis ended up being simpler for our needs and I have nothing but good things to say about it for the most part. Streaming client support in Elixir was not as good as Kafka but writing our own adapter wasn’t too hard.

SQS

NATS.io because I'm using Go, and I can just embed it for one server [0], one binary to deploy with Systemd, but able to split it out when scaling the MVP.

[0] https://www.inkmi.com/blog/how-i-made-inkmi-selfhealing

I am using Beanstalkd, it is small and fast and you just apt-get it on Debian.

However, I have noticed that oftentimes devs are using queues where Workflow Engines would be a better fit.

If your message processing time is in tens of seconds – talk to your local Workflow Engine professional (:

Postgres. Doing ~ 70k messages/second average. Nothing huge but don’t need anything dedicated yet.

For my extremely specialized case, I use a SQLite database as a message queue. It absolutely wouldn't scale, but it doesn't need to. It works extremely well for what I need it to do.

using zeebe/Camunda at work. The system gives you a way of designing and partitioning message-based workflows. It has a very thorough design.

The US Federal Reserve uses IBM MQ for the FedNow interbank settlement service that went live last year.

Architecture info: https://explore.fednow.org/resources/technical-overview-guid...

I've used Qless for several years;

For those unfamiliar, it's a Lua library that gets executed in Redis using one of the various language bindings (which are essentially wrappers around calling the Lua methods).

With our multi-node redis setup it seems to be quite reliable.

Maybe start by explaining what you want to use it for?

I prefer pulsar. Elegant modular design and fully open source ecosystem.

Performance is at least as good as Kafka.

For simpler workload, beanstalkd could be a good fit, either.

SQS is great if you're already on AWS - it works and gets out of your way.

Kafka is a great tool with lots of very useful properties (not just queues, it can be your primary datastore), but it's not operationally simple. If you're going to use it you should fully commit to building your whole system on it and accept that you will need to invest in ops at least a little. It's not a good fit for a "side" feature on the edge of your system.

Another option to consider: Cloudflare Workers. They have a simple queue but you'll need to patch it with a Worker anyway. This means you can programatically manage the queue through the worker and also it makes it easy to send/receive HTTP requests.

What do people think of Supabase?

I would highlight a distinction between Queues and Streams, as I think this is an important factor in making this choice.

In the case of a queue, you put an item in the queue, and then something removes it later. There is a single flow of items. They are put in. They are taken out.

In the case of a stream, you put an item in the queue, then it can be removed multiple times by any other process that cares to do so. This may be called 'fan out'.

This is an important distinction and really effects how one designs software that uses these systems. Queues work just fine for, say, background jobs. A user signs up, and you put a task in the 'send_registration_email' queue.[1]

However, what if some _other_ system then cares about user sign ups? Well, you have to add another queue, and the user sign-up code needs to be aware of it. For example, a 'add_user_to_crm' queue.

The result here is that choosing a queue early on leads to a tight-coupling of services down the road.

The alternative is to choose streams. In this case, instead of saying what _should_ happen, you say what _did_ happen (past tense). Here you replace 'send_registration_email' and 'add_user_to_crm' with a single stream called 'used_registered'. Each service that cares about this fact is then free to subscribe to that steam and get its own copy of the events (it does so via a 'consumer group', or something of a similar name).

This results in a more loosely coupled system, where you potentially also have access to an event history should you need it (if you configure your broker to keep the events around).

--

This is where Postgresql and SQS tend to fall down. I've yet to hear of an implementation of streams in Postgresql[2]. And SQS is inherently a queue.

I therefore normally reach for Redis Steams, but mostly because it is what I am familiar with.

Note: This line of thinking leads into Domain Driven Design, CQRS, and Event Sourcing. Each of which is interesting and certainly has useful things to offer, although I would advise against simply consuming any of them wholesale.

[1] Although this is my go-to example, I'm actually unconvinced that email sending should be done via a queue. Email is just a sequence of queues anyway.

[2] If you know of one please tell me!