Yeah, but I find this fascinating regardless.

This is getting into the direction of a kind of simulation where stuff is not determined by code but a kind of "real" physics.

Yeah, as interesting as the concept is, the lack of frame to frame consistency is a real problem. It also seems like the computing requirements would be immense—the article mentions burning through $10 in seconds.

It seems it's because OP is generating the whole screen every frame / every move. Of course that will give inconsistent results.

I wonder if this approach would work better:

1. generate the whole screen once

2. on update, create a mask for all changed elements of the underlying data

3. do an inpainting pass with this mask, with regional prompting to specify which parts have changed how

4. when moving the camera, do outpainting

This might not be possible with cloud based solutions, but I can see it being possible locally.

> The "real-time" version looks awful, etc

Dang man it's just a guy showing off a neat thing he did for fun. This reaction seems excessive.

I like the idea behind https://oasis-ai.org/ where you can actually try to take advantage of the 'dream logic' inconsistency of each frame being procedurally generated based on the last one. For example, instead of building a house, build the corner of a house, look at that, then look back up and check if it hallucinated the rest of your ephemeral house for you. Of course that uses AI as the entire gameplay loop and not just a graphics filter. It's also... not great, but an interesting concept that I could see producing a fun dream logic game in the future.

Do you mean you want to use AI to generate new scenes in ANSI-art style, or do you mean you want to use AI to render pre-existing scenes as ANSI art?

And those glyphs are not ASCII

This is ASCII: https://commons.wikimedia.org/wiki/File:ASCII-Table-wide.svg

This was built in September 2022, and it's still pretty mind-blowing today:

https://madebyoll.in/posts/game_emulation_via_dnn/demo/

https://madebyoll.in/posts/game_emulation_via_dnn/

Hook world state up to a server and you have multiplayer.

2025 update:

https://madebyoll.in/posts/world_emulation_via_dnn/

https://madebyoll.in/posts/world_emulation_via_dnn/demo/

…but then you just have a graphics card, built to render graphics, that you could tap instead through traditional tooling that’s already widely known and which produces consistent output via local assets.

While the results of the experiment here are interesting from an academic standpoint, it’s the same issue as remote game streaming: the amount of time you have to process input from the player, render visuals and sound, and transmit it back to the player precludes remote rendering for all but the most latency-insensitive games and framerates. It’s publishers and IP owners trying to solve the problem of ownership (in that they don’t want anyone to own anything, ever) rather than tackling any actually important issues (such as inefficient rendering pipelines, improving asset compression and delivery methods, improving the sandboxing of game code, etc).

Trying to make AI render real-time visuals is the wrongest use of the technology.

LCM is what Krea used to gain massive momentum and raise their first $30M.

The tactile reaction to playing with this tech is that it feels utterly sci-fi. It's so freaking cool. Watching videos does not do it justice.

Not enough companies or teams are trying this stuff. This is really cool tech, and I doubt we've seen the peak of what real time rendering can do.

The rendering artifacts and quality make the utility for production use cases a little questionable, but it can certainly do "art therapy" and dreaming / ideation.

Is there any chance you'd open up the source for those projects so others can play with them?

Some people on HN will surely cheer for it, it's the peak of efficiency, you don't even have to move anymore!

I think that’ll be one of the few good parts of it imho

In some ways that’s a lot like how consciousness works isn’t it?

The tech will improve to far exceed the capabilities of a game engine. Real time improvisation and infinite choices, scope, etc.

It makes no sense when people say AI can't do this or that. It will do it next week.

hack, learn and have fun, that's it.

> I do not get the point of this at all

Dunno, this seems like an avenue definitely worth exploring.

Plenty of game applications today already have a render path of input -> pass through AI model -> final image. That's what the AI-based scaling and frame interpolation features like DLSS and FSR are.

In those cases, you have a very high-fidelity input doing most of the heavy lifting, and the AI pass filling in gaps.

Experiments like the OP's are about moving that boundary and "prompting" with a lower-fidelity input and having the model do more.

Depending on how well you can tune and steer the model, and where you place the boundary line, this might well be a compelling and efficient compute path for some applications, especially as HW acceleration for model workloads improves.

No doubt we will see games do variations of this theme, just like games have thoroughly explored other technology to generate assets from lower-fidelity seeds, e.g. classical proc gen. This is all super in the wheelhouse of game development.

Some kind of AI-first demoscene would be a pretty cool thing too. What's a trained model if not another fancy compressor?

It's an interesting tech demo—I think one interesting use case for AI rendering is changing the style on the fly. For example, a certain power-up could change the look to a hyper-saturated comic book style. Definitely achievable with traditional methods, but because AI is prompt-based, you could combine or extend styles dynamically.

Visually speaking, there's always visual issues in tying disparate assets together in a seamless fashion. I can see how AI could be easily used to "hide the seams" so to speak. I think a hybrid approach would be an improvement definitely.