I have been thinking along these lines myself. Most of the time, if we need to calculate things, we'd use a calculator or some code. We wouldn't do it in our head, unless it's rough or small enough. But that's what we ask LLMs to do!

I believe we juggle 7 (plus or minus 2) things in our short term memory. Maybe short term memory could be a tool!

We also don't have the knowledge of the entire internet in our heads, but meanwhile we can still be more effective at strategy/reasoning/planning. Maybe a much smaller model could be used if the only thing it had to do is use tools and have a basic grasp on a language.

Let’s not forget that every round trip with the LLM costs latency (and extra input tokens). We now have parallel tool calls which sometimes works in some models[1]. But it’s great because now a model can say “write these 3 files then read these 2 files” before the time-to-first token latency is incurred once more (not to mention input token cost).

I think LLMs will indirectly move towards being fuzzy VMs that output tokens much like VM instructions so they can prepare multiple conditional branches of tool calling, load/unload useful subprograms, etc. It might not be expressed exactly like that, but I think given how LLMs today are very poor at reusing things in their context window, we will naturally add features that take us in this direction. Also see frameworks like CodeAct[2] etc.

[1] This can be converted to a single tool call with many arguments instead, which you’ll see providers do in their internal tools, but it’s just messier.

[2] https://machinelearning.apple.com/research/codeact