I started building Brobot in 2018 to automate gameplay - I wanted to understand why my automation kept breaking. The more I dug in, the more I realized this was a fundamental problem in GUI automation itself.
Two problems kept surfacing:
1. Script fragility - automation breaks constantly from minor GUI changes
2. Inability to test - no way to verify automation works before deploying
Research in GUI testing shows that the vast majority of test failures come from UI changes, not actual bugs. Yet you can't write integration tests for traditional GUI automation. You just run it and hope.
The root cause: traditional automation uses sequential scripts (do A, then B, then C). Making this robust requires exponential code growth - a 30-state automation has 6.36 trillion possible paths. You can't test all paths, can't guarantee it works.
Model-based GUI automation solves both problems by borrowing from robotics navigation. Instead of writing step-by-step scripts, you create a navigable map of the GUI. The framework handles pathfinding, state management, and error recovery automatically.
Key results:
• Reduces complexity from exponential to polynomial (mathematically proven)
• Makes GUI automation testable for the first time (integration tests, path verification)
• Enables reliable visual APIs for RL agents
• Supports robust dataset generation for model training
• Works for games, business apps, web interfaces - any GUI
Over 7 years, I developed and formalized this approach through both mathematical theory and real-world validation. Springer SoSyM published it in late October.
I'm also building a visual builder (qontinui-web, Feb 2026 launch) for no-code automation - point-and-click designer that creates JSON configs the runner executes locally. Available now in early access (breaking changes possible before launch, but migration tools provided for format changes).
The research provides the mathematical foundation, the Python stack lets you use it today (code-based or visual). Wanted to contribute something useful to the AI/RL community.
jspinak•2h ago
I started building Brobot in 2018 to automate gameplay - I wanted to understand why my automation kept breaking. The more I dug in, the more I realized this was a fundamental problem in GUI automation itself.
Two problems kept surfacing:
1. Script fragility - automation breaks constantly from minor GUI changes
2. Inability to test - no way to verify automation works before deploying
Research in GUI testing shows that the vast majority of test failures come from UI changes, not actual bugs. Yet you can't write integration tests for traditional GUI automation. You just run it and hope.
The root cause: traditional automation uses sequential scripts (do A, then B, then C). Making this robust requires exponential code growth - a 30-state automation has 6.36 trillion possible paths. You can't test all paths, can't guarantee it works.
Model-based GUI automation solves both problems by borrowing from robotics navigation. Instead of writing step-by-step scripts, you create a navigable map of the GUI. The framework handles pathfinding, state management, and error recovery automatically.
Key results:
• Reduces complexity from exponential to polynomial (mathematically proven)
• Makes GUI automation testable for the first time (integration tests, path verification)
• Enables reliable visual APIs for RL agents
• Supports robust dataset generation for model training
• Works for games, business apps, web interfaces - any GUI
Over 7 years, I developed and formalized this approach through both mathematical theory and real-world validation. Springer SoSyM published it in late October.
Open-source implementation: https://github.com/qontinui
• qontinui (Python) - Core automation library (pip install qontinui)
• multistate (Python) - State machine (pip install multistate)
• qontinui-runner (Rust/TypeScript) - Desktop execution engine
• qontinui-api (Python/FastAPI) - REST API bridge (pip install qontinui-api)
Interactive docs & playground: https://qontinui.github.io/multistate/
Original Java version (Brobot, 2018-2025): https://github.com/jspinak/brobot
I'm also building a visual builder (qontinui-web, Feb 2026 launch) for no-code automation - point-and-click designer that creates JSON configs the runner executes locally. Available now in early access (breaking changes possible before launch, but migration tools provided for format changes).
The research provides the mathematical foundation, the Python stack lets you use it today (code-based or visual). Wanted to contribute something useful to the AI/RL community.
Demos:
• Mobile game image collection/labeling: https://jspinak.github.io/brobot/docs/tutorials/tutorial-bas...
• More examples: https://jspinak.github.io/brobot/
Paper: https://link.springer.com/article/10.1007/s10270-025-01319-9
Story behind the name: https://jspinak.github.io/brobot/docs/theoretical-foundation...