I co-authored this paper. It's a new technique to generate 3D graphics as source code instead of a point cloud.
Under the hood, it generates 3D objects with separate, sophisticated internal assembly, producing an editable "kit of parts" (instead of monolithic blobs).
E.g. imagine you generated a 3D washing machine via this approach. It's not merely going to be just "geometry" that looks like a washing machine. We actually know that there is a `Door`, `Drum`, `Control_panel` etc. Which things belong to which assemblies. What moves and where its pivot is. And eventually what those components are supposed to do.
Most current 3D GenAI cannot do this since it generates "monolithic blobs" that look good, but are unusable in downstream workflows (e.g. game engines). I.e. if you generate a 3D bicycle using traditional approaches, it's basically a blob. When you need the wheels to turn, a human (or another AI) must spend time cutting the blob into parts, naming them, placing pivots and rigging joints. I.e. you need post-generation segmentation workflows of some sort.
The paper breaks down the whole technique, and there's a showcase (+ github repo) you can play around with: https://nova3d.xyz/
bckr•30m ago
How far are we from speaking a GameCube-era game into existence as a pastime?
baigy•27m ago
GameCube-era game (with full synchronous multiplayer play): 1-2 quarters. Early "self-generating" Metaverse: ~ 3-6 quarters. The Matrix: ~5-7 years
avaer•2m ago
I am doubtful about the timeline.
Not because AI can't do it; it totally can. LLMs have been able to run the full artist + code pipeline at least since the beginning of the year. I've built several physics-synced network simulation stacks without reading a single line of code. Agents playtest my games overnight and I wake up to a list of technical issues fixed, and FPS boosted. If you know how to ask the shaders will look great.
The problem is that making a game actually worth playing (something that Nintendo would allow to be released) isn't something that was ever possible to do as a pasttime, AI or not. You have to be in front of the computer all day guiding it. Worse, AI does not have any notion of experiencing or evaluating fun, so you can't automate this. That would be a killer research problem to tackle, though!
If we're talking about making something that passes a sniff test, you could make a metaverse right now. It just wouldn't beat the bottom of the Steam barrel in a real world playtest.
WilcoKruijer•7m ago
This looks super interesting. I'm trying out the hosted app using "bring your own key", I've added an OpenAI key but it doesn't seem to let me generate a 3d model. It's still saying I need credits. Is this expected?
baigy•1h ago
Under the hood, it generates 3D objects with separate, sophisticated internal assembly, producing an editable "kit of parts" (instead of monolithic blobs).
E.g. imagine you generated a 3D washing machine via this approach. It's not merely going to be just "geometry" that looks like a washing machine. We actually know that there is a `Door`, `Drum`, `Control_panel` etc. Which things belong to which assemblies. What moves and where its pivot is. And eventually what those components are supposed to do.
Most current 3D GenAI cannot do this since it generates "monolithic blobs" that look good, but are unusable in downstream workflows (e.g. game engines). I.e. if you generate a 3D bicycle using traditional approaches, it's basically a blob. When you need the wheels to turn, a human (or another AI) must spend time cutting the blob into parts, naming them, placing pivots and rigging joints. I.e. you need post-generation segmentation workflows of some sort.
The paper breaks down the whole technique, and there's a showcase (+ github repo) you can play around with: https://nova3d.xyz/
bckr•30m ago
baigy•27m ago
avaer•2m ago
Not because AI can't do it; it totally can. LLMs have been able to run the full artist + code pipeline at least since the beginning of the year. I've built several physics-synced network simulation stacks without reading a single line of code. Agents playtest my games overnight and I wake up to a list of technical issues fixed, and FPS boosted. If you know how to ask the shaders will look great.
The problem is that making a game actually worth playing (something that Nintendo would allow to be released) isn't something that was ever possible to do as a pasttime, AI or not. You have to be in front of the computer all day guiding it. Worse, AI does not have any notion of experiencing or evaluating fun, so you can't automate this. That would be a killer research problem to tackle, though!
If we're talking about making something that passes a sniff test, you could make a metaverse right now. It just wouldn't beat the bottom of the Steam barrel in a real world playtest.
WilcoKruijer•7m ago