Show HN: Controlling 3D models with voice and hand gestures

https://github.com/collidingScopes/3d-model-playground

98•getToTheChopin•8mo ago

I'm sharing my project to control 3D models with voice commands and hand gestures:

- use voice commands to change interaction mode (drag, rotate, scale, animate)

- use hand gestures to control the 3D model

- drag/drop to import other models (only GLTF format supported for now)

Created using threejs, mediapipe, web speech API, rosebud AI, and Quaternius 3D models

Githhub repo: https://github.com/collidingScopes/3d-model-playground

Demo: https://xcancel.com/measure_plan/status/1929900748235550912

I'd love to get your feedback! Thank you

Comments

getToTheChopin•8mo ago

Here's a quick video demo showing how it works: https://x.com/measure_plan/status/1929900748235550912

pvg•8mo ago

https://xcancel.com/measure_plan/status/1929900748235550912 for the tweetless

xnx•8mo ago

Sounds very cool, but I could not make sense of the on-screen instructions. Some images or animations would go a long way to explain the controls.

getToTheChopin•8mo ago

Sorry about that, the instructions need to be improved.

Does this video demo help?

https://x.com/measure_plan/status/1929900748235550912

If it makes it clearer, I'll upload it to the github repo directly

xnx•8mo ago

That video did help. I think I was thrown off by two things: 1) I was expecting 3D controls with more direct mapping (e.g. rotating my hand rotates the model). This is more like gesture mouse controls. 2) Some of the controls were too subtle. The scaling between my gesture size and effect on screen was smaller than I expected.

Great area to develop though. There's so much untapped potential in applying Mediapipe.

getToTheChopin•8mo ago

Thank you for the feedback. I'll continue to work on it!

ednite•8mo ago

Great job! Looks very useful for interactive content creations and product showcasing. Definitely will testing it more. Thanks for sharing.

getToTheChopin•8mo ago

yes I'd love to go further with this concept so that 3D / CAD designers could easily present their models during video calls.

thank you!

corysama•8mo ago

I understand you need your face in the videos for the demos. But, want to mention that you should make sure your system works with your hands in your lap. As shown, the user is going to experience "gorilla arm" fatigue very quickly.

getToTheChopin•8mo ago

Good points, maybe a second camera (phone?) pointed downwards at the tabletop would be good for that. Then the user can rest their hands in a "normal" position.

Thank you for the feedback!

doakes•8mo ago

Very cool! I like the different modes. I've always been fascinated with this space and products like Leap Motion: https://www.youtube.com/watch?v=zXghYjh6Gro

getToTheChopin•8mo ago

It's a shame that computer vision tech like Leap / Eyetoy / Kinect didn't have lasting power. So much fun to build with

danvoell•8mo ago

Awesome, nice work! This type of tech opens up a world of physical games.

getToTheChopin•8mo ago

I made a demo game where you need to dodge the evil bouncing DVD logo by moving your body: https://x.com/measure_plan/status/1924830500541157570

I'm working on a couple other body movement concepts and hope to share soon :)

hulium•8mo ago

See also Webcam Mania: https://webcam.sulat.net/

It uses a bit simpler approach, only detecting movement, but it works good enough for such games.

ugh123•8mo ago

Amazing! Maybe use specific finger positions/gestures to trigger a rotation and scale functions (index finger up and within a bounding box of the model perhaps for rotation, similar for pinch to fingers to scale).

getToTheChopin•8mo ago

I'll try it, thank you! I separated them into completely different interaction modes to avoid misfires, but there's definitely room for efficiencies

Ey7NFZ3P0nzAe•8mo ago

I would be surprised if sign language didn't have an efficient way to convey digits.

chimpansteve•8mo ago

Slightly on topic - anyone remember LeapMotion and is anyone aware of any current support for that? Found an original one in a drawer when I was having a clearout the other day

one_more_q•8mo ago

I have one as well! Sorry no idea what it can be used for nowadays

Atlas Airborne (Boston Dynamics and RAI Institute) [video]

Zen Tools

Is the Detachment in the Room? – Agents, Cruelty, and Empathy

The purpose of Continuous Integration is to fail

Apfelstrudel: Live coding music environment with AI agent chat

What Is Stoicism?

What happens when a neighborhood is built around a farm

Every major galaxy is speeding away from the Milky Way, except one

Extreme Inequality Presages the Revolt Against It

There's no such thing as "tech" (Ten years later)

What Really Killed Flash Player: A Six-Year Campaign of Deliberate Platform Work

Ask HN: Anyone orchestrating multiple AI coding agents in parallel?

Show HN: Knowledge-Bank

Show HN: The Codeverse Hub Linux

Take a trip to Japan's Dododo Land, the most irritating place on Earth

British drivers over 70 to face eye tests every three years

BookTalk: A Reading Companion That Captures Your Voice

Is AI "good" yet? – tracking HN's sentiment on AI coding

Show HN: Amdb – Tree-sitter based memory for AI agents (Rust)

OpenClaw Partners with VirusTotal for Skill Security

Show HN: Seedance 2.0 Release

Leisure Suit Larry's Al Lowe on model trains, funny deaths and Disney

Towards Self-Driving Codebases

VCF West: Whirlwind Software Restoration – Guy Fedorkow [video]

Show HN: COGext – A minimalist, open-source system monitor for Chrome (<550KB)

FOSDEM 26 – My Hallway Track Takeaways

Show HN: Env-shelf – Open-source desktop app to manage .env files

Show HN: Almostnode – Run Node.js, Next.js, and Express in the Browser

Dell support (and hardware) is so bad, I almost sued them

Project Pterodactyl: Incremental Architecture

Atlas Airborne (Boston Dynamics and RAI Institute) [video]

Zen Tools

Is the Detachment in the Room? – Agents, Cruelty, and Empathy

The purpose of Continuous Integration is to fail

Apfelstrudel: Live coding music environment with AI agent chat

What Is Stoicism?

What happens when a neighborhood is built around a farm

Every major galaxy is speeding away from the Milky Way, except one

Extreme Inequality Presages the Revolt Against It

There's no such thing as "tech" (Ten years later)

What Really Killed Flash Player: A Six-Year Campaign of Deliberate Platform Work

Ask HN: Anyone orchestrating multiple AI coding agents in parallel?

Show HN: Knowledge-Bank

Show HN: The Codeverse Hub Linux

Take a trip to Japan's Dododo Land, the most irritating place on Earth

British drivers over 70 to face eye tests every three years

BookTalk: A Reading Companion That Captures Your Voice

Is AI "good" yet? – tracking HN's sentiment on AI coding

Show HN: Amdb – Tree-sitter based memory for AI agents (Rust)

OpenClaw Partners with VirusTotal for Skill Security

Show HN: Seedance 2.0 Release

Leisure Suit Larry's Al Lowe on model trains, funny deaths and Disney

Towards Self-Driving Codebases

VCF West: Whirlwind Software Restoration – Guy Fedorkow [video]

Show HN: COGext – A minimalist, open-source system monitor for Chrome (<550KB)

FOSDEM 26 – My Hallway Track Takeaways

Show HN: Env-shelf – Open-source desktop app to manage .env files

Show HN: Almostnode – Run Node.js, Next.js, and Express in the Browser

Dell support (and hardware) is so bad, I almost sued them

Project Pterodactyl: Incremental Architecture

Show HN: Controlling 3D models with voice and hand gestures

Comments