After my friend shared his plans of helping facilitate medical conference which was bilingual and some of the audience members might feel missed out due to English not being their first language, I thought it's a nice opportunity to create a tool for it. It was an opportunity to work with audio, ASR, accessibility and websockets. It's something new for me.
Another inspiration was Formula 1 captions which I noticed during their broadcasts, which are generated in real-time. After digging a bit into underlying technology I thought to create something similar for event organisers and AV teams. Not as a replacement for human interpreters, but as a more accessible alternative to quite costly hardware options.
That's how Stage Captions was built. Audio comes from browser, gets streamed via WSS to backend, goes through processing pipeline and then captions are streamed back to viewers. Those captions can be displayed on venue TVs, opened on attendees devices by scanning QR code or incorporated into a broadcast using OBS for example.
Obviously, ASR and MT models are third party and not created by me, but there were still some fun challenges like scaling websockets (it was trickier than just adding one more container on load), audio processing (finding optimal chunking and buffering approach) and obviously stabilization algorithms for partial transcription / translation results.
As a developer I was happy to experiment with tech stack which is different from my daily work. And as a human it felt right to build something which can make in-person events easier to follow for non-native speakers and deaf/hard of hearing attendees.