"We introduce requential coding, where a teacher model selects training samples drawn from the student's own distribution"

Due to new learning technique the model has achieved better generalization skill without overfitting and memorization. This become possible because of new learning method which made the student model to generate samples for itself. It led to intensive reuse of existing neurons and allowed to encode information in a more dense way

While researchers are calling it a compression, I think it's a retopologization, and Microsoft had tried to do something similar in the past with their Phi model family, which they trained on reduced dictionary and simplified knowledge base first. But it seems like MS' researchers didn't explore this exact way of learning. I believe this should give even better results in the future and this is another small breakthrough moment

Paper: https://arxiv.org/html/2607.11883v1

Repository: https://github.com/shikaiqiu/requential-coding