That being said, I don't think current token-predictors can do that.
The RICE myth and the lactic acid myth will surely be a part of the training material so the AI will realize that there's a fair amount of unjustified conclusions in the bioworld
Why ask ourselves, when we can ask the AI? Here's the start of my conversation with Gemini:
> Me: What is known about fatty acid combustion in the brain?
> Gemini: The Brain's Surprising Use of Fat for Fuel For a long time, the brain was thought to rely almost exclusively on glucose for its immense energy needs. However, a growing body of research reveals that fatty acid combustion, or beta-oxidation, plays a more significant and complex role in brain energy metabolism and overall neurological health than previously understood. While not the brain's primary fuel source under normal conditions, the breakdown of fatty acids is crucial for various functions, particularly within specialized brain cells and under specific physiological states....
It cites a variety of articles going back at least to the 1990s.
So
> would AI have told us this?
Yes and it did
LLMs tell you what you want to hear, sourced from a random sample of data, not what you need to, based on any professional/expert opinion.
gemini.google.com mentions lactate and fat. But it also knows I care about science. I'm not sure how much history is used currently.
But this is kind of silly because if you're a member of the public and ask a scientist what the brain uses as fuel they'll also say glucose. If you've ever been in a conversation with someone who felt the need to tell you *every detail* of everything they know, then you'll understand that that's not how human communication typically works. So if you want something more specific you have to start the conversation in a way that elicits it.
Indeed.
I'm currently finding myself forced to do so with some customer support agents who keep forgetting critical issues on each step. I do not know if the agents are humans or AI, but either way it's not fun to keep repeating all the same details each time.
And in normal cases, do sometimes notice I've given a wrong impression by skipping some background that I didn't realise the other party didn't already have, precisely because it's not natural to share everything.
I used to think software developer would be the final thing that gets automated because we'd be the ones making specific new AI for each task, but these days I think it's more likely to be spy craft that's the "final profession", because there's nothing to learn from except trial and error — all the info (that isn't leaked) is siloed even within any given intelligence organisation, so AI in that field not only don't get (much) public info about the real skills, they also don't even get classified info. What they will get instead is all the fiction written about spies, which will be about as helpful to real spies as Mr. Scott's Guide to the Enterprise is to the design and construction of the Boeing Starliner.
But:
(1) You're right that it's a hard experiment to do now because LLMs can search the web, on the other hand...
(2) LLMs can search the web, which cuts against the author's implicit premise that LLMs can only know what is in their training data. LLMs have access to tools and oracles. But also...
(3) I checked gemma3:12b running in Ollama. It can not search the internet. But it also knew about fatty acid combustion in the brain and recapped the research history on it.
Now obviously I expect gemma3 to be more prone to hallucination since it's a weaker model and doesn't have any safeguards a production model would have. Also it's working entirely from memory.
But I feel comfortable concluding that the author overestimates the novelty of the study and underestimates LLMs. The study rewrites the Bayesian weights on various interpretations of fatty acid combustion results in the brain. It doesn't completely propose and prove an unheard of hypothesis. Even the offline Gemma told me that glucose-only metabolism used to be the dogma etc etc.
But that's also true in general about science. Breakthroughs are always building on things that came before. And the LLM knows everything that came before.
My first thought: if it did, would you believe it?
> Yes and it did
And before today and this thread, if I asked something like it honestly, without already knowing the answer, and an LLM answered like this...
... I'd figure it's just making shit up.
Before AI will be able to "pretty much solve all our outstanding scientific problems Real Soon Now", it needs to be improved some more, but there's a second, underappreciated obstacle: we will need to learn to gradually start taking it more seriously. In particular, novel hypotheses and conclusions drawn from synthesizing existing research will, by their very nature, look like hallucinations to almost everyone, including domain experts.
The point was that LLMs are not well set up to find new insights unless they are already somehow contained in the knowledge they have been trained on. This can mean "contained indirectly" which still makes this useful for scientific purposes.
The fact that the author maybe underestimated the knowledge about the topic of the article already contained within an LLM does not invalidate this point.
So, AI will look online and synthesize the latest relevant blog posts. In Gemini's case, it will use trends to figure out what you are probably asking. And since this post caught traction, suddenly the long tail of related links are gaining traction as well.
But had Derek asked the question before writing the article, his links would not have matched. And his point that it isn't the AI that figured out that something has changed, remains relevant.
OT, I really enjoy his posts. As AI takes over, will we even read blog posts [enough for authors like him to keep writing], or just get the AI cliff notes - until there is no one writing novel stuff?
The author is, to use his phrase, "deeply uninformed" on this point.
LLMs generalize very well and they have successfully pushed the frontier of at least one open problem in every scientific field I've bothered to look up.
I think what Lowe meant was that an LLM could not have come up with this "on its own", if it was only trained on papers supporting the dogma.
So it cannot produce novel insights which would be a requirement if LLMs should "solve science".
How sure are we about this statement, and why? I have been hearing this a lot, and it might be true, but I would like to read some research into this area.
You write as if this is your conclusion but it's really your premise.
> if it was only trained on papers supporting the dogma.
It's not, it's also trained on all the papers the authors of the current study read to make them think they should spend money researching fatty acid combustion in the brain.
> so that information is probably already in the training data.
Run an offline copy of Gemma with training cutoff before this study came out and it will also tell you about fatty acid combustion in the brain, with studies going back to the 60s and taking off around the 2000s or 2010s.
> So let’s ask ourselves: would AI have told us this? Remember, when people say AI they are about 95% saying “machine learning”, so would it really have told us about this after having been trained on years and years of the medical literature telling it that neurons are obligate glucose users and don’t really have a role for triglycerides? Of course not. And this is why I keep saying (and I’m sure not the only one) that we simply don’t know enough to teach the machine learning algorithms yet. Not to mention that some of what we’d be teaching them is just wrong to start with.
But yeah, if Gemini cites papers from the 90s, then either the shift in thinking on this topic was further back than Lowe makes it seem here (and the "new findings" are already established for decades) or the model interpreted the old papers differently than the scientists did back then.
Reminds me of astronomy and also quantum mechanics
strangattractor•6mo ago