Here's Malcom Gladwell discussing spagetti sauce which feels oddly relevant: https://youtu.be/iIiAAhUeR6Y?si=UJUUiF6H0j6IY3lL
Why do I need to push Postgres to its limits before using a different solution? Throwing a hosted Redis in front of some hot-path API calls is very straightforward and easier to reason about than materialized views or UNLOGGED tables.
You already have a connection string to your database with a password or authn/z with your cloud provider. If this is a "serious" application, you have backups, monitoring, user roles, pgbouncer, partitioning, and other Postgres-specific things to think about. With just a little bit of care, you can make whatever queries you are running fast enough to not need redis.
But ok, you think adding redis is going to solve your performance problem because you can just cache API responses in redis instead of hitting the DB. Maybe, but now you have to think about cache invalidation, eviction behavior, sizing the redis instance, another set of authn/z roles to think about, and of course more cost.
I realize we're speaking past each other, but IME Postgres will work well into the terabyte range and if you can't tune your database setup for performance then reaching for cache is a form of premature optimization.
Requiring HA, partitioning, and replication are good problems to have.
The alternative is spending engineering time on setting all these up for a failed service with like 100 users.
Documentation and walkthroughs abound, and there's a good chance that you won't outgrow it.
But, yes, PostgreSQL is all I ever use for anything that needs to be big. I ported a big old web app that had ScyllaDB, Elastic Search, Redis, and probably some other stuff I've forgotten. It got PostgreSQL+PostGIS (it's a mapping app), that's it. I'm sure there's some situation where it would be worth looking at all that other stuff, but it's ridiculous to build all that complexity in before you even have users.
I really don’t understand why everyone insists that you should use it as a work/message queue.
There are lots of purpose-built bullet proof queuing systems that are simple to setup and administer (or just use SQS).
Your queue is likely to have very different access patterns than the rest of your data, and sticking it in Postgres means you’re probably going to end up setting up partitions or optimizing auto-vacuum on that table way earlier than you probably need to mess around with this things in your scaling.
If your queue has more than a few hundred jobs a day (or you anticipate that like anytime soon), just use a queue.
Because in 99% of cases you don't need a purpose built solution (Even if engineers often think that) at the scale that most people operate in. Nothing is easier to setup and administer than the database you already use.
We are using Postgres as a worker queue in production for many years, with millions of items being processed at any time and it's been perfectly fine. If you have hundreds like in your example...might as well use sqlite.
There's great projects like https://github.com/NikolayS/pgque and https://lucumr.pocoo.org/2026/4/4/absurd-in-production/ that give you even some tooling around that.
Using Postgres gives you transactions and consistency if you have to restore from a backup. Most of the time this doesn't matter (and is a liability) and you can just use some external queue system but sometimes it does matter.
Kafka? No one wants to operate Kafka, if it's a serious contender it's because you need things only it can do. Same with Elasticsearch, it sucks to operate, sucks to build a second stack just for search, so you'd only consider it at the point that Postgres is no longer suitable. Same with Snowflake.
People reach for the things you mentioned wayyyyyy before they should.
Just because you want something queue shaped or search, doesn't mean you should reach for the big, specialized, expensive technology, when Postgres can already support it in your existing infrastructure up until some significant scale you often won't surpass.
A well-tuned Postgres installation on fast hardware and intelligent schema design can scale incredibly far, even if you’re asking it to double as a message bus and full-text search tool.
Like any tool, it works until it doesn't. And when it doesn't it takes a herculean effort to unwind it.
I looked at the first entry and yeah, just say no to moving your business logic into your database. Because change happens...and don't you want that happening in something more plastic than your RDBMS? But it's a great way to bind your solution to Postgres forever.
As an aside, I've used oracle, sybase, informix, mysql, postgresql, rdb, db2, mssql, and a few more that I can't remember. And the idea that pgsql is always the answer is the wrong answer to probably the wrong question.
You don’t need to operate on the entire database. You can backup or roll back individual tables and schemas.
Simplify: move code into database functions
Just Use Postgres for Everything
A relational database is one form of persistent storage. They are great for managing persistent representations of key abstractions and their relationships.
They are not application frameworks nor scalable messaging systems by design.
Besides that, you usually won't need nosql thanks to jsonb, and other special types like in Postgis cover other use cases. SQL is better than dealing with various DSLs like you'd see for timeseries. Then there are things you may want separate tooling for but can also do in Postgres if you want to avoid more infra: pubsub/queues, search w/ pgvector, graph DBs, KV stores, caches.
A lot of the other examples here look ridiculous though, like no I'm not hosting a webserver on Postgres. Database functions should be used sparingly too. Never touched triggers since I normally have a general-purpose language driving DB changes, but could see why someone else might use them.
PostgreSQL is enough - https://news.ycombinator.com/item?id=39273954 - Feb 2024 (316 comments)
As for scale... Just use a larger machine. This works for regular transactional data until you're at something like Amazon scale.
Edit:
Think about this, suppose that you store 1 megabyte of data for each of your customers. So if you have a million customers, it's just 1Tb. And these days, you can have a server with 10Tb RAM delivered overnight. Although you might have to sell your firstborn son (offer applies only to royal families) to fund it.
A lot of sharding/no-sql/... development happened in the late 2000-s when computers were about ~100 times less powerful than now. You _could_ get a system with 10Tb RAM in 2010, but as a specially-designed supercomputer.
For most other problems, Postgres works well, or at least well enough.
A db can be performant, but at a certain point the global locks incremental primary keys create just strangle throughput. What makes a good db design normal form, is almost guaranteed to be inefficient at scale sooner or later. =3
I'm thinking of the problem as using a small amount of text to represent the work that needs to be done and then using a postgres table where some entries are being added as work that needs to be done, and then a worker is pulling the rows of work out of that table, and maybe putting a completion message somewhere in postgres. I'll concede that is more transient data than probably most of the other tables, it might benefit from vacuuming more often. Does the autovacuuming system not figure out it needs to run more often and do it?
Wouldn't the issue would be more overall queries per second, the amount of writes you're already doing, and the general load on the database. We just added some audit tables that are quickly growing to millions of rows, and it seems like Postgres isn't even breaking a sweat. I'm mostly spit balling here and probably glossing over some details.
But, like you said SQS is pretty easy too.
BadBadJellyBean•1h ago