I know we just had the carbon announcement which was distinctly confusing, but comparatively I think that this is an exceptionally well-written article. I do believe that Google has some exceptional scientific minds as illustrated by their ability to take highly technical concepts and explain them in an understandable way for the layperson.

As for the technical nature of this, I'm not sure if I've totally understood correctly despite how well it's written, but are they essentially holding back sort of unweighted parameters built in to the model architecture which they can then populate with new context during 'sleep' and then using the techniques outlined in the 'dream' section to reinforce what gets stored in those MoE weights? That to me seems like a solution which only Google is best placed to solve due to their fully integrated AI stack and historic experience with hardware optimisation, but which would be rather hard to do if you are a more resource and compute constrained organisation

  > are they essentially holding back sort of unweighted parameters built in to the model architecture which they can then populate with new context during 'sleep' and then using the techniques outlined in the 'dream' section to reinforce...

  > Input streams from the active phase are stored in short-term memory before entering the sleep cycle, which bifurcates into Consolidation (transferring fragile short-term tokens into expanded long-term parametric memory using low-rank structures and MoE) and Dreaming (autonomous self-improvement via synthetic routing with random experts and RL refinement).

> On-policy distillation: The expanded student model learns token-specific feedback from teacher distributions generated on self-sampled sequences.

The weird thing is to see this based off the KV-Cache in the diagrams.

Completely different from the Databricks RAFT [1].

My team built "Dreamtime" earlier this year [2], which used a different mechanism to distill out learnings across multiple sessions.

This didn't pull from the token layer, but from user nudges on unsatisfactory responses.

This works like a sawtooth wave where we accumulate context, check for dissonances (a user gets to say "It depends" and fix it), compress it back, retire the old one.

The audit requirements are the real problem for customers, where you need to be able to ask "When did this change? and who changed it?" which was hard to answer from the lowest layer of the model unless you can retain lineage to the original sessions.

A model we train can't serve as an accomplice to a user who wants to hide some financial crimes by nudging the agent repeatedly.

[1] - https://arxiv.org/abs/2403.10131

Good, this seems one of the crucial missing pieces towards AGI. Add critical thinking during cleanup, and we get very close to it.

Reminds me of one of my favourite ML papers, DreamCoder: https://dl.acm.org/doi/pdf/10.1145/3453483.3454080

The actual paper: https://arxiv.org/pdf/2606.03979

Who cares. Fix Google search and stop this LLM nonsense that ruins the economy and burns whole rainforests.