There's some early exploration of using accessibility APIs to empower LLM's. This feels like it's sort of also a super simple & direct intermediate format, that maybe could be a more direct app model that LLMs could use.
More broadly it feels like we have a small forming crisis of computing having too many forms. We had cli tools, unix. Thenw we made gui's, which are yet another way to involve tools (and more). Then webapps where the page is what expresses tools (and more). Then react virtualized the page, supplanted dom. Now we have json that expresses views & tool calling. Also tools need to now be expressed as MCP as well, for ai to use it. Serverless and http and endless trpc and cap'n proto and protobuf ways to call functions/invoke tools. We keep making news ways to execute! Do we have value that each one is distinct, that they all have their own specific channels of execution, all distinct?
any examples come to mind?
I guess if you’re totally insensitive to the cost you can use this.
I use bash-tool and Vercel sandbox to generate charts (Echarts) or tables (Tanstack table) from json data, and then json-render to render the charts, tables and markdown into a dashboard.
What I like about this is that ides of a catalog which is what most business systems have in the form of their records and objects. Giving an AI accessible structure to this gets AI into the realm of the various 4GLs back in late 90s which made user created forms so much easier. Anybody remember that Informix 4GL for building simple apps from the db schema?
how would it relate to ui?
For instance, strings would get a text box, enums would get a dropdown, etc., with validation and everything.
Check this out as an example: https://prismatic.io/docs/jsonforms/playground/
For instance, you can specify that "first_name","last_name" and "email" are strings, but not that first/last name should be next to each other and email in its own row.
There are supersets of OpenAPI that can control the look and feel more. JSON Forms, for instance.
This is cool, too, though.
This describes *User Interfaces*. The closest alternative would be to just return React JS code directly.
But this adds a layer of constraint and control, preventing LLMs to generate e.g. malicious React JS code.
barbazoo•2w ago