Consider that this isn't about not caring what people think, it is about not caring what certain people think.

As someone who certainly is a nerd but doesn't fit the traditional stereotype (good at sports, improvisational noiserock musician on stage, good with people and groups) my number one lesson for a happy life is: Don't surround yourself with people who dislike people who take an interest in obscure things, doesn't matter how obscure or nerdy it is. People who have no hobbies and are not interested in the intricacies of the world that surrounds them are sad, insecure creatures, especially if the only reason they don't do what they like is the fear of what other such people think. If a topic interests you and your first thought is what others might think if you dive into it too deep, that isn't exactly the sign of a happy, fulfilling existence. And by my book we got only one of those.

For example I don't care much what someone who doesn't understand the type of music I make thinks about it. But I deeply care about the opinion of people who like and understand the stuff I aim to do. This way I got to know many good friends.

See, now you get to choose between what the likes of me think about you not/having a weird hobby and what the afraid closet-nerds or boring normies think other people think they can or can't do. There will always be people who think you're weird, you just have to choose which ones. Be a punk and the business people think you're weird. Drive a lamborghini and the punks think you're an asshole. The difference between good weird and bad weird is whether you're aware of your own weirdness.