There are plenty of choices between PSQL & Kafka. It's not like you take one step north and you're in the "oh no you better know what you're doing" territory.
Postgres land is a comfy place filled with transactions across all your data at once, one backup solution that you (hopefully) have had running for months or years and has been thoroughly tested, and ACID compliance. You have a single host, probably, which means that you are neither Available nor Partion-tolerant, but at least you are Consistent.
The moment you expand beyond a single database host you now have a distributed system, and woe unto you if you don't understand what that means.
You can work around this, but to the best of my knowledge you can't have consistency (between your existing Postgres database and your separate queue or event log) and simplicity.
LinkedIn have moved onto Northguard... but no GitHub yet
> Managed services make running Kafka a very uneventful experience (pun intended) and should be the first choice
Confluent, you say?
Good idea; this is stated in the bio on my web site, but I've just added the same info again to the end of the post.
It might be worth adding a more direct call-out to posts like this one. Many may not go as far as reading the Bio page. That may be on them technically speaking, but still.
In any case, thank you for writing and sharing your considered opinion!
The author is a an employee for Confluent/Kafka so because his paycheck and equity grant depends on it and CFLT stock price, obviously whatever he writes is going to be heavily slanted in favor of Kafka. This isn't something that is a footnote at the bottom, it should be right up at the front.
I think that shouldn't matter but I still have a lot to disagree with the article.
feels like overengineering has become the standard for some people, and I quite dislike it personally.
Example: Someone writes a software that could use something simple like SQLite, and they switched to Postgres for performance reasons. Now unless what Kafka beings is the core reason they switched to Postgres not pulling in another dependency and adding a nother piece to the puzzle, can be a total legitimate engineering decision. And that renders the "considered harmful" utterly ridiculous.
Use a system like Kafka if you need what it brings (a distributed event streaming platform). If that isn't what you need or a very simple postgres solition suffices, go for that. Maybe you need event streaming but distributing it is overkill. Maybe you just need some sort of queue. Who knows? Not the author of this post.
Indeed that is the point I am trying to make in the article. Postgres oftentimes absolutely is the right tool to use, and oftentimes it's not. The thing I'm advocating to be wary of is "if all you have is a hammer...". This is to what "considered harmful" refers.
Nailed it. I read the original post earlier this week and was very impressed with its technical detail. But the point of the the post was incongruent with the post's title. But the post got way more attention because of that title.
But if you think about the effort it took to write that post, the title was a really good bet on ROI.
> Nailed it.
Worked for the confluent marketing fluff as well.
Take the approach that appeals to you and feel happy about it without big open source telling you "you're holding it wrong!"
ekjhgkejhgk•13h ago
wewewedxfgdf•11h ago
noxs•8h ago