I agree with this but there is one issue, AFAIK, the languages used do not lend themselves to optimization. And I expect the databases in use have the same issue.
It is almost like you need to put optimizations in the hardware kind of like what IBM does with its mainframes for transaction processing. Instead, AI companies is doing the usual 'race to be there first', ignoring about the consequences of the design.
The ML research community is very focused on scaling. As an example that doesn't (fully) deanonymize me look at how people reacted to things like KAN or MAMBA. Even in HN comments questions about scale are always front and center. Don't get me wrong, scale is an important question, but the context matters. These questions are out of place because they are not giving the new frameworks a chance to scale. As any sane researcher would do, you scale iteratively. To be resource efficient you test your ideas at a small scale and then move up, fixing other problems along the way. It's not even just a matter of resource efficiency, but even for solving the problems. By jumping straight to scale you add a lot of complexity into the mix and make it difficult to decouple the problems. This hyper-fixation on making everything bigger is really hindering us. Yeah, sure there are papers that do make it out (as I even gave examples) but these are still harder for smaller labs to pursue and get through review (review isn't completely blind and we'd be ignorant to ignore the politics that goes on).
EGreg•1mo ago
And just like him, when it comes to AI, I am making a huge exception for my usual principles.
My usual principles are that open-source gift economies benefit the world and break people free from gatekeepers. The World Wide Web liberated people from having to pay Payola to radio stations just to get their song played, from TV, Magazines, Newspapers, etc. It let anyone publish worldwide within a second, and make changes just as easily. It is what led to Facebook, Amazon, Google, LinkedIn, X etc. even existing (walled gardens like AOL would never allow it).
Wikipedia has made everyone forget about Britannica and Encarta. Linux runs most computers in the world. Open protocols like VoIP and packet switching brought marginal costs of personal communication down to zero. And so on and so forth.
But when it comes to AI, we can't have everyone do whatever they want with AI models, for the same reason we can't give everyone nuclear weapons technology. The probability that no one will misuse it becomes infinitesimally small real fast. And it takes just a few people to create a designer virus with a long incubation period, that infects and kills everyone, as just one example. Even in the digital world we are headed towards a dark forest where everything is adversarial, nothing can be trusted, and anyone's reputation, wealth and peace of mind can be destroyed at scale, by swarms of agents. That's coming.
For now, we know where the compute is. We can see it from space, even. We can trace the logistics, and we can make sure that it runs only "safe" models that refuse to do these things. All the stories you read about some provider "stopping" large-scale hacking is because they ran the servers.
So yes, for this one thing, I make a strong exception. I don't want to see proliferation of AI models everywhere. Sadly, though, as long as the world runs on "competition" instead of "cooperation", destruction is inevitable. Because if we don't do it, then China will, etc. etc.
There have been a few times in recent history that humanity successfully came together to ban dangerous things. Chemical weapons ban. Nuclear non-proliferation. Montreal Protocol and CFCs (repair the hole in the ozone layer). We can still do this for AI models running on dark compute pools. But time is running out. Chaos is coming.
wswope•1mo ago
Your train of thought makes sense, but relies on the assumption that people and small groups wouldn’t keep tinkering at scale to do bad things even if we had a united world government trying to stop it.
EGreg•1mo ago
In general, cleaning up a mess is easier when the mess isn't self-preserving and being grown at an exponential scale by swarms of agents running on dark compute.