Confirmatory of Sutskever's view that predicting the next token forces a deep understanding. To effectively predict the next token it needs a good idea of what comes after the next token.

How is this news? isn't it an obvious fact from the Transformer architecture?

> A coding agent solving a software-engineering task spends dozens of steps reasoning

No. That's simple PR hype. Parrotry is not reasoning.

>> Probes trained to predict the outcome of future edits (before they are materialized and written on disk) achieve performance above chance up to roughly 25 steps in advance.

Are these probes effectively run in parallel? The way this reads is more about predicting a future outcome than keeping the current token relevant based on past tokens.

It's been known for several years that LLM activations encode future tokens ahead of time (e.g. https://arxiv.org/abs/2404.00859).

But this has only been shown on simple tasks, so I think this paper is still quite neat. The interesting thing is that they show "future horizon length" varies across models.

In other words, since the next semantic prediction for forecasting the future is built on the training dataset, it's hard for anything truly new to emerge.

Then how do humans create something 'creative'—something that didn't exist before? I think it might be because the process of simplifying the complex system of nature differs between individuals. The data being learned now is all labeled by humans and simplified through human cognition. Within that kind of information, creativity seems hard to emerge.

Ultimately, with data that already contains interpretation, no matter how much you repeat the learning, it just becomes an encyclopedia that only explores within human knowledge, repeating predictions within human interpretation. So I wonder if we actually need a different encoder that interprets raw data—not based on human interpretation.

In reality, what changed Newton's absolute time to Einstein's relativity was a conclusion derived simply from observing the world. Newton's interpretation was supported by a lot of evidence in its time. If an AI studied all the medieval data from Newton's era, could it actually come up with the theory of relativity?

I'm always curious about this. I think AI is already very good at coding and will soon become better than humans. Logical structures are ultimately human interpretations, and reasoning within that framework is something AI can probably do more logically than humans. In other words, once humans create the framework, stacking the logical Jenga blocks within it—AI will be better at that.

But true creativity lies in breaking the framework itself, and I'm skeptical about whether AI can do that. The encoder also seems insufficient. There will likely be limits. I might be trapped in my own biases.

But the limitations of the current approach seem too clear to ignore.

When I look at the approach of these papers, it feels like an argument that adding shadows that imitate the world will eventually make them become the objects themselves.

I used to ask my coding agent to present two alternatives to choose from while implementing a task, and include a tokens required for each alternative to implement. (So that I can choose one which needs less token vs one which needs more rigour depending on task)

Finally there is evidence that the model kinda actually knows the correct token spend on each method.

It makes intuitive sense. How else could you write a 500-line script top-to-bottom with no backspace key and no arrow keys and get all the imports etc. right upfront?