I hear you, and I struggle with this exact problem when talking about this stuff. The issue, really, is that "silicon valley" is a convenient shorthand for a certain sort of company that is quite popular in the actual silicon valley. It's true that not all companies that happen to be in the actual silicon valley are "silicon valley" companies in this sense, and it's also true that Musk companies are very much "silicon valley" companies in this context. "Silicon Valley" in this sense is referring more to a kind of culture or philosophy than a geographic location.

But how else to refer to it? I tend to go with "SV-style" companies, but that's really clunky.