First the good news: There is a relatively well beaten path to becoming an independent consultant/contractor who gets to pick and chose their clients and projects. You probably won't work on intellectually challenging or stimulating projects 100% of the time, but you can schedule them so that there's usually an interesting contract running in parallel to the ones that reliably pay the bills. There will be dry spells where you have nothing but boring work but as you get better and have a proven history of delivering, that work improves.

Now the bad news: getting there takes a lot of work - most of it unrelated to technology - and requires a certain personality. You need to get out there and network till your ears bleed, get good at sales and marketing, manage the stress of not having a W2 salary and health insurance, learn how to bill clients and convince them to actually pay, maintain relationships with current and past coworkers, and a hundred other yaks you have to shave. It's death by a thousand cuts but if this lifestyle fits your personality, you can achieve a very interesting and varied career.

If you spend your college years and summers building out that network, you can even start this process shortly after graduation but it is very risky. Depending on how the job market looks in a couple years, it might even be the optimal thing to do (it's what I did after the 2008 GFC), but this is not a path for the feint of heart. It is hard and much more stressful than collecting a paycheck. I was personally relieved to get a FTE position after years of consulting.

>I'm getting the impression that the field is over-saturated, with new research being published on a daily basis

The research being published is mostly about optimizations of transformer models.

There is a lot more research in other areas. For example, everyone seemed to forget about MuZero, but that was one of the most important breakthroughs in the last decade, because it was a model that could recursively search.

So if you think about tasks like driving a car, a model that can predict the evolution of a physical system (much like MuZero can predict the evolution of the game board), and make a decision, is going to perform much better than any of the vision based transformer models that are in cars today.

But there are other areas of research as well, namely in hardware with specific instructions for general compute.

If you want a hard problem that's far from predictable with tangible utility, why don't you work on something that helps humans with human problems, but whee you can still use your talents? Some research that could help understand a disease, or a new kind of medical technology, or something like that. I don't know how well it aligns with your salary goals but it might be meaningful.

I feel like you've assembled a group of criteria (some of which are mutually exclusive) and trying to make a job fit all of them. I wish you luck in thst, but it may help you to identify them to understand what to compromise on.

Significance. You don't want to be a cog in a machine. Since it's easy to find significance outside of work, and since almost no work is significant this may be the easiest to recalibrate.

You want job security with high pay. This comes with working at large companies. They're less likely to go under. With some luck you can avoid the regular culls. These jobs require zero creativity (and likely oppose creativity. )

Creativity is risky. Because what you create may have value, or not. If you need to be creative, that's also easy to do outside work. Take up painting, music, pottery, whatever.

You're still young. Understand that most of the work you do over the next 40 years will just be work. It can supply money. It can be made "secure". But dont expect your work to give you significance (find that elsewhere), don't expect your work to satisfy your creative urges (find that elsewhere.)

Life is more than work. Figure out what you want in life and do multiple things to satisfy those goals. Don't place the burden on work satisfying them all. That route typically does not reduce well.

Although it's difficult to monetize, you could (for the time being) work full-time on your open-source projects. I recognize that might not be the easiest path, but if you pull it off (you can do it!) then you have - in my view - the best position in tech.

In summary, install linux, move to a cabin in the woods, become open-source developer.