I have worked as a programmer for some years, but I still feel absolutely incapable of designing a highly scalable system and in a way that’s cost effective (without wasting money unnecessarily).
I have read system design books and designing data intensive applications. They are useful but I don’t feel like I can “problem solve” effectively in this domain.
For instance, I got asked in an interview how to scale a web crawler and I froze. Ask me how google can give an AI overview so quickly and I wouldn’t know where to start.
Most of what I do at work is adding new features and fixing bugs. Companies often dictate things like “we’re a Kubernetes, Postgres and redis shop”.
For others that have had to learn these skills without learning from someone at your company, how did you do it? And how do you approach solving these problems?
I understand this is a vague question but I’d appreciate any sort of pointers. Even if you’ll be telling me “the obvious”, that will just help me to trust that advice more (I struggle to trust my own judgement).
Thanks.