I've built pretty scalable things using nothing but Python, Celery and Postgres (that usually started as asyncio queues and sqlite).
Why?
Because in 1999 when I started using PHP3 to write websites, I couldn't get MySQL to work properly and Postgres was harder but had better documentation.
It's ridiculous spinning up something as "industrial strength" as Postgres for a daft wee blog, just as ridiculous as using a 500bhp Scania V8 for your lawnmower.
Now if you'll excuse me, I have to go and spend ten seconds cutting my lawn.
The tooling in a lot of languages and frameworks expects you to use an ORM, so a lot of the time you will have to put up a fair bit of upfront effort to just use Raw SQL (especially in .NET land).
On top of that ORM makes a lot of things that are super tedious like mapping to models extremely easy. The performance gains of writing SQL is very minor if the ORM is good.
Which is the exact point the article is making. You don't have scale. You don't need to optimize for scale. Just use Postgres on its own, and it'll handle the scale you need fine.
There are things we don’t want to do (talk to costumers, investors, legal, etc.), so instead we do the fun things (fun for engineers).
It’s a convenient arrangement because we can easily convince ourselves and others that we’re actually being productive (we’re not, we’re just spinning wheels).
Unless you actively push yourself to do the uncomfortable work every day, you will always slowly deteriorate and you will run into huge issues in the future that could've been avoided.
And that doesn't just apply to software.
I should get off HN, close the editor where I'm dicking about with HTMX, and actually close some fucking tickets today.
Right after I make another pot of coffee.
...
No. Now. Two tickets, then coffee.
Thank you for the kick up the arse.
Removing it, no matter whether I created it myself, sure, that can be a hard problem.
I've certainly been guilty creating accidental complexity as a form of procrastrination I guess. But building a microservices architecture is not one of these cases.
FWIW, the alternative stack presented here for small web sites/apps seems infinitely more fun. Immediate feedback, easy to create something visible and change things, etc.
Ironically, it could also lead to complexity when in reality, there is (for example) an actual need for a message queue.
But setting up such stuff without a need sounds easier to avoid to me than, for example, overgeneralizing some code to handle more cases than the relevant ones.
When I feel there are customer or company requirements that I can't fulfill properly, but I should, that's a hard problem for me. Or when I feel unable to clarify achievable goals and communicate productively.
But procrastrination via accidental complexity is mostly the opposite of fun to me.
It all comes back when trying to solve real problems and spending work time solving these problems is more fun than working on homemade problems.
Doing work that I am able to complete and achieving tangible results is more fun than getting tangled in a mess of unneeded complexity. I don't see how this is fun for engineers, maybe I'm not an engineer then.
Over-generalization, setting wrong priorities, that I can understand.
But setting up complex infra or a microservices architecture where it's unneeded, that doesn't seem fun to me at all :)
Normally the impetus to overcomplicate ends before devs become experienced enough to be able to even do such complex infra by themselves. It often manifests as complex code only.
Overengineered infra doesn't happen in a vacuum. There is always support from the entire company.
I certainly did for a number of years - I just had the luck that the cool things I happened to pick on in the early/mid 1990s turned out to be quite important (Web '92, Java '94).
Now my views have flipped almost completely the other way - technology as a means of delivering value.
Edit: Other cool technology that I loved like Common Lisp & CLOS, NeWS and PostScript turned out to be less useful...
Interesting term. Probably pretty on-point.
I’ve been shipping (as opposed to just “writing”) software for almost my entire adult life.
In my experience, there’s a lot of “not fun” stuff involved in shipping.
Or is it to satisfy the ideals of some CTO/VPE disconnected from the real world that wants architecture to be done a certain way?
I still remember doing systems design interviews a few years ago when microservices were in vogue, and my routine was probing if they were ok with a simpler monolith or if they wanted to go crazy on cloud-native, serverless and microservices shizzle.
It did backfire once on a cloud infrastructure company that had "microservices" plastered in their marketing, even though the people interviewing me actually hated it. They offered me an IC position (which I told them to fuck off), because they really hated how I did the exercise with microservices.
Before that, it almost backfired when I initially offered a monolith for a (unbeknownst to me) microservice-heavy company. Luckily I managed to read the room and pivot to microservice during the 1h systems design exercise.
EDIT: Point is, people in positions of power have very clear expectations/preferences of what they want, and it's not fun burning political capital to go against those preferences.
In those interviews (and in real work too) people still want you skewing towards certain answers. They wanna see you draw their pet architecture.
And it's the same thing in the workplace.
Also: if there's limited knowledge on the interviewer side, an incorrect answer to a question might throw off a more experienced candidate.
It's no big deal but it becomes more about reading the room and knowing the company/interviewers than being honest in what you would do. People don't want to hear that their pet solution is not the best. Of course you still need to know the tech and explain it all.
For context, my current project is a monolith web app with services being part of the monolith and called with try/catch. I can understand perhaps faster, independent, less risky recovery in the micro services case but don’t quite understand the fault tolerance gain.
It's been a long time since I've done "normal" web development, but I've done a number of high-performance or high-reliability non-web applications, and I think people really underestimate vertical scaling. Even back in the early 2000s when it was slightly hard to get a machine with 128GB of RAM to run some chip design software, doing so was much easier than trying to design a distributed system to handle the problem.
(we had a distributed system of ccache/distcc to handle building the thing instead)
Do people have a good example of microservices they can point us to the source of? By definition it's not one of those things that makes much sense with toy-sized examples. Things like Amazon and Twitter have "micro" services that are very much not micro.
His reasoning was all the big players use it, so we should be too...
It was literally a solution looking for a problem. Which is completely arse backwards.
That diagram is just aws, programming language, database. For some reason hadoop I guess. And riak/openstack as redundant.
It just seems like pretty standard stuff with some seemingly small extra parts because that make me think that someone on the team was familiar with something like ruby, so they used that instead of using java.
"Why is Redis talking to MongoDB" It isn't.
"Why do you even use MongoDB" Because that's the only database there, and nosql schemaless solutions are faster to get started... because you don't have to specify a schema. It's not something I would ever choose, but there is a reason for it.
"Let's talk about scale" Let's not, because other than hadoop, these are all valid solutions for projects that don't prioritize scale. Things like a distributed system aren't just about technology, but also data design that aren't that difficult to do and are useful for reasons other thant performance.
"Your deployment strategy" Honestly, even 15 microservices and 8 databases (assuming that it's really 2 databases across multiple envs) aren't that bad. If they are small and can be put on one single server, they can be reproduced for dev/testing purposes without all the networking cruft that devops can spend their time dealing with.
Sure, they aren't bad. They're horrible.
I was there before 10 years ago. I remember the pain in the ass that was hosting your own web server and own hardware, dealing with networking issues with cisco switches and thinking about getting a ccna. I remember the days of trying to figure out php and ranodm ass modules or how python and wsgi fit together on a slow ass windows machine instead of just spinning up an app and doing network calls using a spa.
Have you guys just forgotten all the enterprise crap that existed? Have you guys forgotten before that how things like compilers (ones you had to pay exorbintant amounts of money for) and different architectures were the headaches?
It's been two steps forward, one steps back, but we're still way better off.
Yes, people bring in k8s because they want to resume build and it goes poorly, but I've also used k8s in my personal setup that was much easier than the poor man's version I had of it.
All of this is just rose-tinted glasses, and people throwing the baby out with the bathwater. Just because some people have bad experiences with microservices because people don't often do them right, people just write them off completely.
I personally don't care for it and if I design something I make it so it avoids that stuff if I can at all help it. But I've come to see that it can have real value.
The thing is though, that then you really need someone to be that very competent ops person. If you're a grug like me, you don't get many shots to be good at something. I probably don't have the years in me to be good at both ops and "pure" programming.
So if you are a startup and you're not some kind of not only very smart but smart, fast and with taste, maybe pick your battles.
If you are great at the ops side, ok, maybe design it from that perspective and hire a bunch of not-made-of-unobtainium regular middle-of-the-road coders to fill in what the microservices and stuff should contain and manage those. This requires money for a regular hiring budget. (Or you are supersmart and productive and "play pretend enterprise" with all roles yourself. But I have never seen such a person.)
Or focus on a tight design which can run without any of that, if you come more from the "I'm making a single program" part of the world.
Tinkering syndrome can strike in any kind of design, so you need personal maturity whatever path you choose.
Clown fiesta.
Once you have a service that has users and costs actual money, while you don’t need to make it a spaghetti of 100 software products, you need a bit of redundancy at each layer — backend, frontend, databases, background jobs — so that you don’t end up in a catastrophic failure mode each time some piece of software decides to barf.
I mean it will happen regardless just from the side effects of complexity. With a simpler system you can at least save on maintenance and overhead.
I totally get the point it makes. I remember many years ago we announced SocketStream at a HackerNews meet-up and it went straight to #1. The traffic was incredible but none of us were DevOps pros so I ended up restarting the Node.js process manually via SSH from a pub in London every time the Node.js process crashed.
If only I'd known about upstart on Ubuntu then I'd have saved some trouble for that night at least.
I think the other thing is worrying about SPOF and knowing how to respond if services go down for any reason (e.g. server runs out of disk space - perhaps log rotation hasn't been setup, or has a hardware failure of some kind, or the data center has an outage - I remember Linode would have a few in their London datacenter that just happened to occur at the worst possible time).
If you're building a side project I can see the appeal of not going overboard and setting up a Kubernetes cluster from the get-go, but when it is things that are more serious and critical (like digital infrastructure for supporting car services like remotely turning on climate controls in a car), then you design the system like your life depends on it.
Consider WhatsApp could do 2M TCP connections on a single server 13 years ago, and Ford sells about 2M cars per year. Basic controls like changing the climate can definitely fit in one TCP packet, and aren't sent frequently, so with some hand-waving, it would be reasonable to expect a single server to handle all remote controls for a manufacturer for all cars from some year model.
Or maybe you could use wifi-direct and bypass the need for a server.
Or a button on the key fob. Perhaps the app can talk to the key fob over NFC or Bluetooth? Local/non-internet controls will probably be more reliable off-grid... can't have a server outage if there are no servers.
I guess my point is if you take a step back, there are often simple, good solutions possible.
As opposed to what? Not doing anything at all and participating in this insanity of complexity?
Really that's going way too far - you do NOT need Redis for caching. Just put it in Postgres. Why go to this much trouble to put people in their place for over engineering then concede "maybe Redis for caching" when this is absolutely something you can do in Postgres. The author clearly cannot stop their own inner desire for overengineering.
The reason startups get to their super kubernetes 6 layers mega AWS powered ultra cached hyper pipelined ultra optimised web queued applicatyion with no users is because "but technology X has support for an eventually consistent in-memory caching layer!!"
What about when we launch and hit the front page of HN how will the site stay up without "an eventually consistent in-memory caching layer"?
lol, In the diagram, Redis is not even talking with MongoDB
The caching abstractions your frameworks have are also likely designed with something like Redis in mind and work with it out of the box. And often you can just start with an in-memory cache and add Redis later, if you need it.
Probably should stop after this line - that was the point of the article. It will work at lower scales. Optimize later when you actually know what to optimize.
It'll give you time to redesign and rebuild so Postgres is fast enough again. Then you can take Redis out, but once you've set it up you may as well keep it running just in case.
I think that redis is a reasonable exception to the rule of ”don’t complicate things” because it’s so simple. Even if you have never used it before, it takes a few minutes to setup and it’s very easy to reason about, unlike mongodb or Kafka or k8s.
In that scenario, the last thing you need is another layer between application and database.
Even in a distributed environment, you can scale pretty far with direct-to-database as you say.
Yes, I also put Redis in that list. You can cache and serve data structure in many other ways, for example replicate the individual features you need in you application instead of going the lazy route and another service to the mix. And don't get me started on Kafka... money thrown in the drain when a stupid grpc/whatever service would do.
Part of being an engineer is also selecting the minimum amount of components for your architecture and not being afraid of implementing something on your own if you only need 1 of 100s features that an existing product require.
Well put!
But I have to defend Tailwind, it's not a massive CSS framework, it just generates CSS utility classes. Only the utility classes you use end up in the output CSS.
However having in the CV any of those items from left side in the deployment strategy is way cooler than mentioning n-tier architecture, RPC (regardless how they are in the wire), any 1990's programming language, and so forth.
A side effect from how hiring works so badly in our industry, it isn't enough to know how master a knife to be a chef, it must be a specific brand of knife, otherwise the chef is not good enough for the kitchen.
For example, in the recent "who's hiring" thread, I saw at least two places where they did that: Duckduckgo (they mention only algorithms and data structures and say "in case you're curious, we use Perl") and Stream (they offer a 10-week intro course to Go if you're not already familiar with it). If I remember correctly, Jane Street also doesn't require prior OCaml experience.
The place where I work (bevuta IT GmbH) also allowed me to learn Clojure on the job (but it certainly helped that I was already an expert in another Lisp dialect).
These hiring practices are a far cry from those old style job postings like "must have 10+ years of experience with Ruby on Rails" when the framework was only 5 years old.
[1] all those examples check that box, but please let's not start a language war over this statement.
[2] for Jane Street I hear they do, DDG pays pretty well especially because it pay the same rate regardless where you are in the world, so it's a top-talent salary for many places outside SV.
And best of all, you don't feel the need to keep chasing after the latest hype just to keep your CV relevant.
Yes, it’s nonsense, stirring up a turbulent slurry of eventually consistent components for the sake of supporting hundreds of users per second, it’s also the nonsense that you’re expected to say, just do it.
Adding guardrails to protect your team from itself mandates some complexity, but just hand-waving that away as unnecessary is a bad answer. At least if you're not working as part of a team.
> Organizations which design systems... are constrained to produce designs which are copies of the communication structures of these organizations.
Make them part of your build first. Tagging a release? Have a documented process (checklist) that says 'run this, do that'. Like how in a Java Maven build you would execute `mvn release:prepare` and `mvn release:perform`, which will execute all tests as well as do the git tagging and anything else that needs doing.
Scale up to a CI pipeline once that works. It is step one for doing that anyway.
And some of these guidelines have grown into satus quo common recipes. Take your starting database for example, the guideline is always "sqlite only for testing, but for production you want Postgres" - it's misleading and absolutely unnecessary. These defaults have also become embedded into PaaS services e.g. the likes of Fly or Scaleway - having a disk attached to a VM instance where you can write data is never a default and usually complicated or expensive to setup. All while there is nothing wrong with a disk that gets backed up - it can support most modern mid sized apps out there before you need block storage and what not.
I just set one build agent up with a tag that both plans required. The simplest thing that could possibly work.
Or, consider redundancy: Your customers likely expect your service to not have an outage. That's a simple requirement, but very hard to get right, especially if you're using a single server that provides your application. Just introducing multiple copies of the app running in parallel comes with changes required in the app (you can't assume replica #1 will handle the first and second request—except if you jump through sticky session hoops, which is a rabbit hole on its own), in your networking (HTTP requests to the domain must be sent to multiple destinations), and your deployment process (artefacts must go to multiple places, restarts need to be choreographed).
Many teams (in my experience) that have a disdain for complex solutions will choose their own, bespoke way of solving these issues one by one, only to end up in a corner of their own making.
I guess what I'm saying is pretty mundane actually—solve the right problem at the right time, but no later.
(except the caching layer. Remember the three hard problems of computer science, of which cache invalidation is one.)
Still hoping for a good "steelman" demonstration of microservices for something that isn't FAANG-sized.
Rather, cache invalidation is the process of determining which cache entries are stale and need to be replaced/removed.
Oh, it absolutely does. You need some way to get your secrets into the application, at build- or at runtime, for one, without compromising security. There's a lot of subtle catches here that can be avoided by picking standard tooling instead of making it yourself, but doing so definitely shapes your architecture.
You can get all that with a monolith server and a Postgres backend.
(Sarcasm)
If your business can afford irregular downtime, by all means, go for it. Otherwise, you'll need to take precautions, and that will invariably make the system more complex than that.
Except it's really a "What over-engineered monstrosity have you built?" in the theme of "choose boring technology"
p.s. MariaDB (MySQL fork) is technically older and more boring than PostgreSQL so they're both equally valid choices. Best choice is ultimately whatever you're most familiar with.
Left unchecked, Claude is very happy to propose "robust, scalable and production ready" solutions - you can try it for yourself. Tell it you want to handle new signups and perform some work like send an email or something (outside the lifecycle of the web request).
That is, implying you need some kind of a background workload and watch it bring in redis, workflow engines, multiple layouts for docker deployment so you can run with and without jobs, obscene amount of environment variables to configure all that, create "fallbacks" and retries and all kinds of things that you will never spend time on during an MVP and even later resist adding just because of the complexity and maintenance they require.
All that while (as in the diagram of the post), there is an Erlang/Elixir app capable of doing all that in memory :).
Especially in an age where you can basically click a menu in GitHub and say "Hey, can I have a CI pipeline please?"
Or at least it's not engaging with the obvious counterargument at all - that: "You may not need the scale now, but you may need it later". For a startup being a unicorn with a bajillion users is the only outcome that actually counts as success. It's the outcome they sell to their investors.
So sure, you can make a unscalable solution that works for the current moment. Most likely you wont need more. But that's only true b/c most startups don't end up unicorns. Most likely is you burn through their VC funding and fold
Okay stack overflow allegedly runs on a toaster, but most products don't fit that mold - and now that they're tied to their toaster it probably severely constrains what SO can do it terms of evolving their service
You're making two assumptions - both wrong:
1) That this is an unscalable solution - A monolith app server backed by Postgres can take you very very far. You can vertically scale by throwing more hardware at it, and you can horizontally scale, by just duplicating your monolith server behind a load-balancer.
2) That you actually know where your bottlenecks will be when you actually hit your target scale. When (if) you go from 1000 users to 10,000,000 users, you WILL be re-designing and re-architecting your solution regardless what you started with because at that point, you're going to have a different team, different use-cases, and therefore a different business.
Your solution is to basically do a re-write when scale becomes a problem. Which is the textbook example of something that sounds good but never works
On the other hand I can't think of a business that failed b/c it failed to scale :)
I recently had to work with SQL again after many years, and was appalled at the incidental complexity and ridiculous limitations. Who in this century would still voluntarily do in-band database commands mixed with user-supplied data? Also, the restrictions on column naming mean that you pretty much have to use some kind of ORM mapping, you can't just store your data. That means an entire layer of code that your application doesn't really need, just to adapt to a non-standard from the 70s.
"just use postgres" is not good advice.
The problem is that people don't like hearing their ideas suck. I do this too, to be fair. So, yes, we spend endless hours architecting what we'd desperately hope will be the next Facebook because hearing "you are definitely not the next Facebook" sucks. But alas, that's what doing startups is: mostly building 1000 not-Facebooks.
The lesson here is that the faster you fail, the faster you can succeed.
In almost any other scenario I feel the author is being intentionally obtuse about much of the reality surrounding technology decisions. An engineer operating a linux box running postgres & redis (or working in an environment with this approach) would become increasingly irrelevant & would certainly earn far less than the engineer operating the other. An engineering department following "complexity is not a virtue" would either struggle to hire or employ engineers considered up-to-date in 2006.
Management & EXCO would also have different incentives, in my limited observations I would say that middle and upper management are incentivised to increase the importance of thier respective verticals either in terms of headcount, budget or tech stack.
Both examples achieve a similar outcome except one is : scalable, fault tolerant, automated and the other is at best a VM at Hetzner that would be swiftly replaced should it have any importance to the org, the main argument here (and in the wild) seems to be "but its haaaard" or "I dont want to keep up with the tech"
KISS has a place and I certainly appreciate it in the software I use and operating systems I prefer but lets take a moment to consider the other folks in the industry who aren't happy to babysit a VM until they retire (or become redundant) before dispensing blanket advice like we are all at a 2018 ted talk . Thanks for coming to my ted talk
Now every system design interview expects you to build some monstrous stack with layers of caching and databases for a hypothetical 1M DAU (daily active users) app.
Mess in the head.
sachahjkl•3h ago