This is a little project we’ve been working on, which started as multimodal models physically controlling gamepads. It’s evolved a little bit since then and we now have a setup where an agent can control movement, cameras and taking selfies (where game allows) in an open world.

The project sprung actually from wanting to try out some observability platforms for LLMs, and sort of kept evolving. Hope you enjoy this little free roam around SF. It’s helped us learn a lot about prompting, observability and actuation of agents in the process.

We’ll be posting more hopefully, but very much a “hack” as it won’t take any time away from us actually playing these titles. Hope you enjoy this video and encourage you to play this awesome game.