Do you see these trajectories being used to fine tune a model automatically in some way rather then just replay, that way similar workflows might be improved too?

This seems like a much more powerful version of what I wanted MCP "prompts" to be - and I'm curious to know if I have that right.

For me, I want to reduce the friction on repeatable tasks. For example, I often need to create a new GraphQL query, which also requires updating the example query collection, creating a basic new integration test, etc. If I had a MCP-accessible prompt, I hoped the agent would realize I have a set of instructions on how to handle this request when I make it.

would just caching llm responses work here?

I love the idea!

I think the problem will be defining wether there is a cache-hit or not, since "agents" are loosely defined and the tasks include basically anything.

This is a neat idea and is similar to something I've been turning over in my head. LLMs are very powerful for taking a bunch of disparate tools/information/etc and generating good results but the speed is a big issue as well as reproducibility.

I keep imagining an Agent that writes a bunch of custom tools when it needs it and "saves" them for later use. Creating pipelines in code/config that it can reuse instead of solving from 0 each time.

Essentially, I want to use LLM for what they are good for (edge cases, fuzzy instructions/data) and have it turn around to write reusable tools so that the next time it doesn't have to run the full LLM, it can use a tiny LLM router up front to determine if there exists a tool to do this already. I'm not talking about MCP (though that is cool), this would use MCP tools but it could make new ones from the existing.

Here is an example.

Imagine I have an Agent with MCP tools to read/write to my email, calendar, ticketing system, and slack. I can ask the LLM to slack me every morning with an overview of my events for the day and anything outstanding I need to address. Maybe the first pass uses a frontier model to determine which tools to use and it accomplishes this task. Once I'm happy with the output then the Agent feeds the conversation/tool calls into another LLM to distill it to a Python/Node/Bash/whatever script. That script would call the MCP tools to do the same thing and use small LLMs to glue the results together and then it creates a cron (or similar) entry to have that run every morning.

I feel like this would remove a ton of the uncertainty when it comes to which tools an LLM uses without requiring humans to write custom flows with limited tools available for each task.

So the first pass would be:

    User: Please check my email, calendar, and slack for what I need to focus on today.
    
    LLM: Tool Call: Read Unread Email
    
    LLM: Tool Call: Read last 7 days of emails the user replied to
    
    LLM: Tool Call: Read this week's events from calendar
    
    LLM: Tool Call: Read unread slack messages

    LLM: Tool Call: Read tickets in this sprint

    LLM: Tool Call: Read unread comments on tickets assigned to me
    
    LLM: Tool Call: Read slack messages conversations from yesterday
    
    LLM: Please use the following data to determine what the user needs to focus on today: <Inject context from tool calls>
    
    LLM: It looks like have 3 meetings today at.....

Then a fresh LLM reviews that and writes a script to do all the tool calls and jump to the last "Please use the following data" prompt which can be reused (cron'd or just called when it makes sense).

I might be way off-base and I don't work in the space (I just play around the edges) but this feels like a way to let agents "learn" and grow. I've just found that in practice you don't get good results from throwing all your tools at 1 big LLM with your prompt, you're better off limiting the tools and even creating compound tools for certain jobs you do over and over. I've found that lots of little tool calls add up and take a long time so a way for the agent to dynamically create tools from combining other tools seems like a huge win.

Wrote something similar into my rules - obtained mixed results.

Curious how effective this is.

accessibility (a11y) trees are super helpful for LLMs; we use them extensively in stagehand! the context is nice for browsers, since you have existing frameworks like selenium/playwright/puppeteer for actually acting on nodes in the a11y tree.

what does that analog look like in more traditional computer use?

Feels kinda like JIT compiling your agent prompts into code. Awesome concept, hope it pans out.

I love the minimal approach and general-use focus.

If I understand correctly, the engine caches trajectories in the simplest way possible, so if you have a cached trajectory a-b-c, and you encounter c-b-d, there's no way to get a "partial" cache hit, right? As I'm wrapping my head around this, I'm thinking that the engine would have to be a great deal more complicated to be able to judge when it's a safe partial hit.

Basically, I'm trying to imagine how applicable this approach could be to a significantly noisier environment.

Very cool concept!

I wish v0, lovable, bolt et al did something like this with their suggested prompts

It’s such a poor user experience to pick a template, wait 3-5min to generate and then be dumped randomly either on an incredibly refined prototype, or a ton of code that just errors out. And in either case, having no clue what to do next

Not sure if this can work. We played around with something similar too for computer use, but comparing embeddings to cache validate the starting position is super gray, no clear threshold. For example, the datetime on the bottom right changes. Or if it's an app with a database etc, it can change the embeddings arbitrarily. Also, you must do this in every step, because as you said, things might break at any point. I just don't see how you can reliably validate. If anything, if models are cheap, you could use another cheaper llm call to compare screenshots, or adjust the playwright/api script on the fly. We ended up writing up a quite different approach that worked surprisingly well.

There are definitely a lot of potential solutions, I'm curious where this goes. IMO an embeddings approach won't be enough. I'm more than happy to discuss what we did internally to achieve a decent rate though, the space is super promising for sure.

> At Pig, we built computer-use agents for automating legacy Windows applications (healthcare, lending, manufacturing, etc).

How do you justify this vs fixing the software to enable scripting? That seems both cheaper and easier to achieve and with far higher yields. Assume market rate servicing of course.

Plus; how do you force an "agent" to correct its behavior?

now that's a beautiful api!