Hi HN, I'm Smyan and I enjoy building agents. Modern multimodal LLMs are great at vision and perception but are quite poor at localization. This naturally creates a massive problem when we try to take our RPA frameworks and give them to agents to perform computer use tasks.

For browsers, we have been able to solve this by using the DOM tree to supply the LLM with structural hints and now more recently modern browser use frameworks use Set-Of-Marks prompting which take the structural information of the webpage and convert it into visual bounding boxes with labels, which allow the LLM to use it's strong vision and perception and accurately convert it to a form of localization. Functionally, this means the LLM now needs to simply say "click 4" instead of having to say "click 443 213".

This methodology however fails horribly when we try to apply it to native OS automation. The accessibility tree, which is often exists for native apps, is usually quite brittle, exposes non-deterministic selectors and often stripped by developers, which can make it hard to localize elements. Fuzzy matching can help with this, but it is still none the less very hard to get right.

This is exactly why I made SoMatic. SoMatic is a pure vision based framework that uses a finetuned YOLO model (highly inspired from OmniParser v2) to identify text and interactable elements in a UI. The YOLO model runs locally on the CPU with ONNX and is quite fast. SoMatic draws the bounding boxes and labels and then maps the id for each bounding box to the coordinates for the center of the given box. This therefore enables Set-Of-Marks prompting for in principal ANY user interface.

I ran an ablation benchmark using the framework with GPT-5.5 (high) and was able to acquire a ~ 20% higher accuracy than just the raw model. What was however surprising was that the model performed slightly better with knowing just the location of the bounding boxes (without actually seeing them). This could be due to the threshold tuning for the YOLO model either drawing too many or too few boxes (I'm not entirely sure).

Either way, if you have been wanting to give your AI agents full autonomy of your computer (Windows, Mac and Linux), you can download the CLI with

   npm install -g somatic-cli/cli

and the corresponding skill with

   npx skills add Smyan1909/SoMatic

The CLI also comes with a stdio MCP server if you want the model to directly parse the screenshots (b64 encoded) from the chosen API instead of it having to read the image after each screenshot.

I'd love to get your feedback on the vision-only approach. Are we at the point where we can finally abandon the mess that is the OS accessibility tree for automation?

Why Patagonia?

Dell Bulks Up Hardware as AI Infrastructure Shifts to On-Premises

Show HN: Daily word puzzle game based on polysemy

Opus 4.7 vs. Sonnet 4.6

SpaceX IPO filing filled with AI bets, Starship dreams, and Musk at the center

Sadiq Khan blocks £50M Met police deal with Palantir

Show HN: GoKubeDownscaler – Off-Hours Kubernetes Scaling Cuts Costs by 70%

Ask HN: What are you working on? (May 21)

4.4 Magnitude Earthquake in Naples

The Sovereign Retailer: Building a Spaceship in My Own Backyard

Show HN: I gave my AI video generator 86 MCP tools so Claude Code can drive it

Show HN: See how much gold, plutonium and cocaine SpaceX's Bitcoin buys

LoongForge-A high-performance training framework for LLMs, VLMs, diffusion, vlas

Amiga newspaper article from June 2000

Mechanism design for large language models (2025)

Show HN: (Better) Chrome Tab Manager

Show HN: SoMatic – Vision-based OS automation framework for AI agents

Back up your TOTP codes by printing their QR codes

CPPL: A Circuit Prompt Programming Language

Wikimedia Foundation Community Tech team has been disbanded, engineers laid off

Show HN: Physics AI – Visualizing physics problems with step-by-step derivations

Study: ChatGPT and other AI bots made errors before Scottish election

LLM System Design Benchmark

How NetEase Built an Authentic Wuxia Open World in Where Winds Meet

Planetary research: Innovative Mars rovers 'swim' through the sand

Opal Pathtracer

My Mental Model for "Is It Worth Automating?" Has Changed

U.S. to Award Nine Quantum-Computing Firms $2B and Take Equity Stakes

Show HN: A timeline of recent open source CVE intensity and volume

Show HN: PocketWebTools – Private AI tools that run in the browser

Show HN: Boatswain, A macOS menu bar app for Fathom Analytics