I feel like people are already currently doing this. Essentially storyboarding first.

This guy a month ago for example: https://youtu.be/SGJC4Hnz3m0

I've been experimenting with that from a slightly different angle: teaching Claude how to play and referee a pencil-and-paper RPG that I developed over about 20 years starting in the mid 1970s. Claude can't quite do it yet for reasons related to permanence and learning over time, but it can do surprisingly well up until it runs into those problems, and it's possible to help it past some obstacles.

The game is called "Explorers' Guild", or "xg" for short. It's easier for Claude to act as a player than a director (xg's version of a dungeon master or game master), again mainly because of permance and learning issues, but to the extent that I can help it past those issues it's also fairly good at acting as a director. It does require some pretty specific stuff in the system prompt to, for example, avoid confabulating stuff that doesn't fit the world or the scenario.

But to really build a version of xg on Claude it needs better ways to remember and improve what it has learned about playing the game, and what it has learned about a specific group of players in a specific scenario as it develops over time.

...and then, the pneumatics in your living room.

Isn't this still essentially "vibe simulation" inferred from videos? Surface-level visual realism is one thing, but expecting it to figure out the exact physical mechanics of sailing just by observing YouTube videos of boats, and usefully abstract them into a "gamified" form, is another thing entirely.

The bottleneck for games of any size is always whether they are good. There are plenty of small indies which do not put out good games. I don't see world models improving game design or fun factors.

If I am wrong, then the huge supply of fun games will completely saturate demand and be no easier for indie game devs to stand out.

It's very impressive tech but subject to the same limitations as other generative AI: Inconsistency, inaccurate physics, limited time, lag, massively expensive computation.

You COULD create a sailing sim but after ten minutes you might be walking on water, or in the bath, and it would use more power than a small ferry.

There's no way this tech can run on a PS5 or anything close to it.

Honestly getting a Sunfish is probably cheaper than the a VR headset if you want to "train sailing"

I understand the ultimate end goal to be simulation of life. A near perfect replica of the real world we can use to simulate and test medicine, economy, and social impact.

I would think that building a environment which can be managed by a game engine is the first pass. In a few years when we are able to render more than 60 seconds it could very well replace the game engine entirely by just rendering everything in realtime based on user interactions. The final phase is just prompts which turn directly into interactive games, maybe even multiplayer. When I see the progress we've made on things like DOOM, where it can infer the proper rendering of actions like firing weapons and even updating scores on hits and such it doesn't feel like we're very far off, a few years at most. For a game studio that could mean cutting out almost everything between keyboard and display, but for now just replacing the asset pipeline is huge.

It seems to be generating images in real time, not 3d scenes. It might still be useful for prototyping.

It means you should go the other way. Open world winning against smaller, handcrafted environments and stories was generally a mistake, and so is this.

Most people don't like putting on VR headsets, no matter what the content is. It just never broke out of the tech enthusiast niche.

LeCun wasn't producing results. He was obstinate and insistent on his own theories and ideas which weren't, and possibly aren't, going anywhere. He refused to engage with LLMs and compete in the market that exists, and spent all his effort and energy on unproven ideas and research, which split the company's mission and competitiveness. They lost their place as one of the top 4 AI companies, and are now a full generation behind, in part due to the split efforts and lack of enthusiastic participation by all the Meta AI team. If you look at the chaos and churn at the highest levels across the industry, there's not a lot of room for mission creep by leadership, and LeCun thoroughly demonstrated he wasn't suited for the mission desired by Meta.

I think he's lucky he got out with his reputation relatively intact.

Isn't it more like this: JEPA looks at the video, "a dog walks out of the door, the mailman comes, dog is happy" and the next frame would need to look like "mailman must move to mailbox, dog will run happily towards him", which then an image/video generator would need to render.

Genie looks at the video, "when this group of pixels looks like this and the user presses 'jump', I will render the group different in this way in the next frame."

Genie is an artist drawing a flipbook. To tell you what happens next, it must draw the page. If it doesn't draw it, the story doesn't exist.

JEPA is a novelist writing a summary. To tell you what happens next, it just writes "The car crashes." It doesn't need to describe what the twisted metal looks like to know the crash happened.

It's super cool but I see it as a much more flexible open ended take on the idea of procedurally generated worlds where hard-coded deterministic math and rendering parameters are replaced by prompt-able models.

The deadness you're talking about is there in procedural worlds too, and it stems from the fact that there's not actually much "there." Think of it as a kind of illusion or a magic trick with math. It replicates some of the macro structure of the world but the true information content is low.

Search YouTube for procedural landscape examples. Some of them are actually a lot more visually impressive than this, but without the interactivity. It's a popular topic in the demo scene too where people have made tiny demos (e.g. under 1k in size) that generate impressive scenes.

I expect to see generative AI techniques like this show up in games, though it might take a bit due to their high computational cost compared to traditional procedural generation.

This sort of comment reminds me about the comments by programmers two years ago.

"Sure it can write a single function but the code is terrible when it tries to write a whole class..."

And what if I go somewhere then go back there a week later?