Originally I was interested in using computer use agents to automate testing of desktop applications. The tools I tried gave agents the ability to click a mouse and fed them screenshots, but the agents struggled to use the mouse accurately. A couple projects like OmniParser use specialized screenshot reading models and element labeling instead of pixel coordinates for interactions. But even when they work, they're slow and expensive to run since they rely on models.
I then looked at accessibility APIs as a way to read the screen. Initial experiments worked and I found that accessibility APIs are a well-known tool for desktop automation. But when I wrote tests for a cross-platform desktop app, I didn't find a library that worked across Mac, Windows, and Linux. Additionally, my initial tests were flaky because elements take time to appear and get new IDs as the UI re-renders.
I built xa11y around two ideas to close these gaps: 1) create an accessibility abstraction that works across all platforms and 2) emulate Playwright's auto-waiting, selectors, and locator patterns which make web automation robust.
The hardest part was finding the right abstraction which faithfully represents the platform-specific APIs in a common interface. One example: on Windows, the accessibility tree for an entire app can be read in one API call, but on Linux each element attribute requires a separate call. The library originally fetched the entire accessibility tree for each query, then walked it to return results, but on Linux for large apps, reading all the data could take 10 seconds or more. As a result, the library evaluates the filter as it walks the tree (instead of after) and only returns the matching elements (not subtrees).
The library is written in Rust, has Python and JS bindings, and is MIT licensed. Blog post with more details: https://crowecawcaw.github.io/general/2026/05/30/accessibili...
Any feedback welcome!