Nah. I can write a whole program using 0 tokens, I can’t write a whole program with 0 lines of code.

> What’s the going rate for tokens in terms of dollars?

It depends on the provider/model, usually pricing is calculated as $/million tokens with input/output tokens having different per token pricing (output tends to be more expensive than input). Some models also charge more per token if the context size is above a threshold. Cached operations may also reduce the price per token.

OpenRouter has a good overview over provider and models, https://openrouter.ai/models

The math on what people are actually paying is hard to evaluate. Ime, most companies rather buy a subscription than give their developers API keys (as it makes spending predictable).

max context sizes are probably going to go up, but smaller contexts will always be cheaper/more-efficient than larger ones

> Who is the target for this?

Agents. Going to be more tools and software targeted for consumption by agents

It’s interesting but I think it’s measuring the wrong thing. Abstraction is a fundamental principle in software. As a human, I’ve worked with classes and modules far larger than what fits in my head, just because I’m only fitting the function signatures and purpose into my head, and not the implementation details. In practice I find Claude really good at extracting useful information in a human-like way from a codebase. It doesn’t usually stuff the entire codebase into its context window.

I’m not so sure an increasingly large context window will be seen as a critical enabler (as it was viewed 6 months ago), after watching how amazingly effective subagents and tool calls are at tackling parts of the problem and surfacing the just the relevant bits for the task at hand. And if increasing the context window isn’t the current bottleneck, effort will be put elsewhere.

Gemini 1.5 Announced the 1 million token context window in 2024. I admire this view of being forward looking towards new technologies, specially when we see the history of how bad people can be at predictions just by looking at history HN posts/comments.

If we look at back 2 years, companies weren't investing into training their LLMs so heavily on code. Any code they got their hands on was what was in the LLMs training corpus, it's well known that the most recent improvements in LLM productivity occurred after they spent millions on different labs to produce more coding datasets for them.

So while LLMs have gotten a lot better at not needing the entire codebase in context at once, because their weights are already so well tuned to development environments they can better infer and index things as needed. However, I fail to see how the context window limitation would no longer be an issue since it's a fundamental part of the real world. Would we get better and more efficient ways of splitting and indexing context windows? Surely. Will that reduce our fear of soiling our contexts with bad prompt response cycles? Probably not...

value/size