The best way (anecdotally) for me has been to not understand the subject matter at all, and just work work work on something until it's near completion. At that point understanding the subject matter is much easier, and you can compare it to your own understanding and work. Many times I've since improved my work from subject matter because other people have had great ideas that stood the test of time, and very occasionally my work has created a new frontier without me really understanding it at the time. Still, revolutionary ideas are not always something that can be turned into a business. Funnily, I see many business ideas are just gluing things together, and I don't think anyone is really upset about that.

As you said though, sometimes you have to go the wrong way (knowingly or unknowingly) just to see what's there or if there aren't missed opportunities. Heavily established (sub-)fields can be very rigid and hard to find new ground in.

I think there are two common traps here.

First, not every good idea needs to be a "secret that could change the world." The most valuable ideas often aren't glamorous; they're the ones that solve real, boring problems. If everyone only chased the next big thing, we'd never get the crucial, unsexy stuff built. For example, a new ad-tech algorithm might feel exciting, but a simple internal tool that cuts a developer's daily grunt work by an hour can have a far bigger, more tangible impact. There's value in both.

Second, the best ideas rarely come from isolated genius. You can get stuck in your own head, convinced you've found something new, only to later realize you missed a crucial piece of context. The real magic happens through constant communication. For instance, I once spent weeks on a technical solution for a problem, only to show it to a product manager who, in two minutes, pointed out a much simpler, non-technical way to solve the same user pain (he just told me to skip it). Talking with people, especially those outside your niche, helps you get a reality check, refine your thoughts, and avoid building something no one actually needs. My best ideas weren't born in a flash of solo inspiration, but in messy, back-and-forth conversations.