I made Windows-Use, an open-source tool that lets (all LangChain-supported) LLMs execute tasks directly on the Windows desktop using tool calling.
It allows you to build AI agents that can interact directly with GUI elements in Windows apps using natural language. Basically, it acts like a layer between the Windows OS and the AI, making desktop automation much simpler. It uses the coordinates of interactive elements to perform actions, so you don’t have to write a separate script for every task.
The goal was to take care of the hard parts, so others don’t have to:
- Accessing the accessibility tree and preprocessing it to make it LLM-friendly (interactive elements + screenshot).
- Providing solid tools to interact with the desktop (clicking, typing, etc.).
- Creating a reusable agent setup.
I built the first version last September. It only took 2 days to get something working, but improving the speed took much longer. I had to do a lot of experiments and dive deep into how the Windows accessibility tree works. Initially, grounding took around 20 seconds… then, 3 days later, Anthropic released their "computer-use"
Since then, I’ve been steadily improving it; now the grounding time is down to 1.7 seconds, and the toolset has improved a lot.
The vision: Just prompt the agent, and it does the task; no need to worry about how. I call it “vibe automation.”
Demos I’ve made:
- Generate Word docs on any topic: searches the web, writes the content, opens Word, and saves it.
- Book flights on Google Flights using a browser.
- Navigate files in Explorer and open a specific file (e.g., "Open this file in D:\ drive").
- Change the desktop theme from dark to light, like a user would do manually.
You can install it on Windows using:
pip install windows-use
Try it out and let me know how it works for you!
I'm Jeomon George