Various reasons

Some people just believe there is no innate knowledge or we dont need it if we just scale/learn better (in the direction of Bitter Lesson)

(ML) Academia is also heavily biased against it due to mainly two reasons: - Its harder to publish, since if you learn Task X with innate Knowledge, its not as general, so reviewer can claims its just (feature) engineering - Which hurts acceptance, so people always try to frame their work as general as possible - Historical reasons due to the conflict the symbolic community (which rely heavily on innate knowledge)

The problem with assuming tons of innate knowledge is that it needs to be stored somewhere. DNA certainly contains enough information to determine the development of various different neuron types and which kinds of other neurons they connect to, but it certainly cannot specify weights for every individual synapse, except for animals with very low neuron counts.

So the existence of a sensorimotor feedback loop for a basic behavior is innate (e.g. moving forward to seek food), but the fine-tuning for reliabily executing this behavior while adapting to changing conditions (e.g. moving over difficult terrain with an injured limb after spotting a tasty plant) needs to be learned through interacting with the environment. (Stumbling around eating random stuff to find out what is edible.)

The complexity of the human body surely weighs in at over 1 GB.

I think of DNA analogously to the rules of cellular automata. The entropy of the rules is much less than the entropy of the dynamical system the rules describe.

The body is filled with innate knowledge. The organs all know what to do. The immune system learns to detect intruders (without synapses). Even a single cell organism is capable of complex and fluid goal-oriented behavior, as Michael Levine attests.

I think the assumption that all knowledge exists in the brain, and all knowledge in the brain is encoded by neuronal weights, is probably too simplistic.

Regarding language and vision, I think the cognitive scientists are right: it is better to view these as organs or “modules” suited to a function. Damage Broca’s area and you get Broca’s aphasia. Damage your lung and you get trouble breathing. Neither of these looks like the result of statistical learning from randomly initialized parameters.

> Or it could be that a single neuron in the human brain is worth 10000 neurons in an artificial neural network, and thus, the biggest part of the "secret" of human brain is just that it's hilariously overparameterized.

Being overparameterized alone doesn't explain how fast we learn things compared to deep neural nets though. Quite the opposite actually.

Only 1 GB code maybe, but dependent on the universe as the runtime environment.

Not really - that 1GB is the seed for a procedural generation mechanism that has been finely tuned to its unfolding in an environment over 4 billion years.

DNA is the ultimate demoscene exe

2 more years bro.

Yeah, he was quite vocal in his opinion that they would plateau earlier than they did and that little value would be derived from them because they're just stochastic parrots. Agree with him that they're probably not sufficient for AGI, but, at least in my experience, they're adding a lot of value and they're continuously performing better in a range of tasks that he wasn't expecting them to.

excellent point

> There is a lot of "transformer LLMs are flawed" going around, and a lot of alternative architectures being proposed, or even trained and demonstrated. But so far? There's nothing that would actually outperform transformer LLMs at their strengths. Most alternatives are sidegrades at best.

That's kind of an awkward timing to say that, as alternative to transformers have flourished over the past few weeks (Qwen3-Next, Granite 4).

But IIRC Le Cun's criticism applies to more than just transformers and to next-token predictors as a whole.