I have been working on a project called VoxaDub: an open source local-first AI dubbing tool which can translate videos into other languages while preserving the original voices of the speakers.
A major issue when translating is ensuring the translation matches with the length of speech in the target language. Even slight differences in timing can cause unnatural dubbing. It becomes more complex when there are multiple speakers in the original video.
I have created a pipeline that uses Whisper/WhisperX to transcribe and align audio, Community-1 to identify speakers, Hy-MT2 to translate contextually and VoxCPM2 to generate speaker aware audio.
The main feature of VoxaDub is the ability to improve the translation using the available time for the audio to be spoken for, limit the time changes to within certain bounds and analyse for any timing issues.
There is also some experimental Live Dubbing for YouTube videos and live streams through a Chrome extension. This is actively being developed and is not yet production ready.
The main project runs fully locally once models are set up, no external translation or TTS APIs are required. Optional GPU usage is available using Colab and Modal.
I am pleased to see that the project was recently added to the Ecosystem & Community section of OpenBMB's official VoxCPM GitHub repository (PR #389). While this is just a community project listing it was nice to see this progress.
True examples of the English to Spanish, Japanese and Turkish dubbing can be found in the README. Source videos, information on how it was implemented and any limitations are also present.
The project is MIT licensed and can be found on GitHub.
I would really appreciate any technical feedback on the timing of the translations, speaker consistency and simplifying the local setup. There is a lot of room for improvement so any thoughts would be great!