I think LLMs will indirectly move towards being fuzzy VMs that output tokens much like VM instructions so they can prepare multiple conditional branches of tool calling, load/unload useful subprograms, etc. It might not be expressed exactly like that, but I think given how LLMs today are very poor at reusing things in their context window, we will naturally add features that take us in this direction. Also see frameworks like CodeAct[2] etc.
[1] This can be converted to a single tool call with many arguments instead, which you’ll see providers do in their internal tools, but it’s just messier.
There's an open source tool being developed that is sort of along these lines: https://github.com/raestrada/storycraftr
But:
- it expects the user to be the orchestrator, rather than running fully unattended in a loop, and
- it expects the LLM to output a whole chapter at a time, rather than doing surgical edits: https://github.com/raestrada/storycraftr/blob/b0d80204c93ff1...
(It does use a vector store to help the model get context from the rest of the book, so it doesn't assume everything is in context.)
I’d like to apply what is being suggested in this post, but it doesn’t make sense to me to have to give an LLM access to a text editor just to write a novel. Isn’t there a better way?
Tool calling is great, but LLMs are - and should be used as - more than just tool callers. I mean, some tools will have to be other LLMs doing what they’re good at, like writing a novel, summarizing, brainstorming ideas, or explaining complex topics. Tools are useful, but the stuff LLMs actually do is also useful. The basic premise that LLMs should never output anything beyond tools and arguments is leaving most of the value of LLMs on the table.
FWIW, I've done this and it works incredibly well. It's essentially integrating the LLM into the text editor, and requests of the LLM are more like requests of the text editor directly. The mental model I use is the editor has become an AI Agent itself. I've also done with with spreadsheets, web page editors, various tools in project management software. It's an incredible perspective that works.
And it sounds like you’ve had a lot of success with this approach in an impressive variety of application types. May I ask what tooling you usually use for this (eg custom python for each hack? MCP? some agent framework like LangGraph/ADK/etc, other?)
In the end, each app is both what it was before, as well as can be driven by prompts. I've also specialized each to have 4 agents that are as I describe, but they each have a different representation of the app's internal data; for example, a word processor has the "content, the document" in HTML/CSS as well as raw text. When one wants to manipulate the text, requests use the HTML/CSS representation, and selections go through a slightly separate logic than a request to be applied to the entire document. When one wants to critically analyze the text, it is ASCII text, no need for the HTML/CSS at all. When one wants to use the document as a knowledge base, outside the editor, that's yet another variant that uses the editor to output a RAG ready representation.
anko•11mo ago
I believe we juggle 7 (plus or minus 2) things in our short term memory. Maybe short term memory could be a tool!
We also don't have the knowledge of the entire internet in our heads, but meanwhile we can still be more effective at strategy/reasoning/planning. Maybe a much smaller model could be used if the only thing it had to do is use tools and have a basic grasp on a language.
dijit•11mo ago
It was good advice for me.