Hi HN, Frank here. We placed second overall at Cal Hacks 13.0 (700+ projects), four of us from McGill.
Over 200,000 people are diagnosed with throat cancer each year, and for many, the surgery removes their voice box. More than half a million people are already living without one.
Their lips, tongue and jaw still move in the same way, but with nothing there to create any sound, they lose the ability to speak forever.
In 36 hours we built Silent Voice to give them their voice back.
With four different AI systems working together, Austin held a full phone call without making a sound, and the caller heard him in his own voice.
• Muscles: we built a sensor rig using four surface EMG pads ($20 each) to measure the electrical activity in his face 1,000 times a second, and used a neural network (SVR-1) we'd trained to infer the movement of his tongue and jaw (which a camera can't see)
• Lips: Auto-AVSR is a lip-reading technique which only works on complete video. We rebuilt it to work in real-time, re-reading Austin's sentence 10 times a second
• Face: computer vision tracks his expressions and carries his emotion through the voice
• Context: Claude uses both sides of the conversation to predict what Austin may say, and compensates for common confusions such as 'bills' and 'pills'
SVR-1 is ~45M params, 12 transformer layers on top of 3 conv blocks, pretrained on Berkeley's 19h silent speech set then fine tuned on ~70 min of Austin. With only our 4 pads it picks the right sentence out of 10 lip look-alikes 72.3% of the time, Berkeley's released model gets 27.7% (theirs was trained on 8 pads though).
I led the voice pipeline. Standard techniques rely on complete input, but we were only receiving individual words as Austin formed them. Waiting for a complete sentence meant callers were subjected to 4.9 seconds of dead air, whilst sending single words sounded like a robot reading a list.
So I re-engineered the entire streaming layer into Cartesia, delivering complete (though brief) phrases in step with Austin's natural pauses, and cut the silence added by the model after each phrase (around half a second) down to a more natural (i.e. human) breath. His expression can also shift the emotion mid-sentence.
As a result, callers were hearing Austin 0.6 seconds after his lips began to move, in real-time.
gottaguo•1h ago
Over 200,000 people are diagnosed with throat cancer each year, and for many, the surgery removes their voice box. More than half a million people are already living without one.
Their lips, tongue and jaw still move in the same way, but with nothing there to create any sound, they lose the ability to speak forever.
In 36 hours we built Silent Voice to give them their voice back.
With four different AI systems working together, Austin held a full phone call without making a sound, and the caller heard him in his own voice.
• Muscles: we built a sensor rig using four surface EMG pads ($20 each) to measure the electrical activity in his face 1,000 times a second, and used a neural network (SVR-1) we'd trained to infer the movement of his tongue and jaw (which a camera can't see)
• Lips: Auto-AVSR is a lip-reading technique which only works on complete video. We rebuilt it to work in real-time, re-reading Austin's sentence 10 times a second
• Face: computer vision tracks his expressions and carries his emotion through the voice
• Context: Claude uses both sides of the conversation to predict what Austin may say, and compensates for common confusions such as 'bills' and 'pills'
SVR-1 is ~45M params, 12 transformer layers on top of 3 conv blocks, pretrained on Berkeley's 19h silent speech set then fine tuned on ~70 min of Austin. With only our 4 pads it picks the right sentence out of 10 lip look-alikes 72.3% of the time, Berkeley's released model gets 27.7% (theirs was trained on 8 pads though).
I led the voice pipeline. Standard techniques rely on complete input, but we were only receiving individual words as Austin formed them. Waiting for a complete sentence meant callers were subjected to 4.9 seconds of dead air, whilst sending single words sounded like a robot reading a list.
So I re-engineered the entire streaming layer into Cartesia, delivering complete (though brief) phrases in step with Austin's natural pauses, and cut the silence added by the model after each phrase (around half a second) down to a more natural (i.e. human) breath. His expression can also shift the emotion mid-sentence.
As a result, callers were hearing Austin 0.6 seconds after his lips began to move, in real-time.
It's a prototype, Austin is healthy and no patients have tried it yet. Code is open source, the demo is on the Devpost: https://devpost.com/software/silent-voice-s1f74p
If you've had (or know someone who's had) a laryngectomy, or work with them as a speech therapist, I'd love to hear from you.