I'm a former YC founder/visiting partner and independent musician/producer in SF. About a year ago I started collaborating with the Magenta research team as an artist to explore the role of realtime generative music models in studio production and live performances
Through our early experiments we recognized the need to bridge the gap between server based generative music models and professional audio software, and to design interfaces for generative models that are more steerable and controllable than the current "push button to generate song" paradigm. We built and released a DAW plugin which has used by some of our favorite artists — including a wonderful showcase with Daito Manabe in Tokyo — and was featured as an exciting new music tool at NAMM
Today we're open-sourcing the plugin under Apache 2.0 for the community to fork and build new interfaces with
The plugin is architected using React/TypeScript for the UI layer and JUCE/C++ for DAW connection, audio processing, and websocket audio streaming from the Gemini/Lyria API. This allows rapid iteration on the frontend using hot-reload (Shadcn/Tailwind), while ensuring latency sensitive operations (audio streaming and playback) happen in a compiled and unmanaged language with a tight clock. State is synced between TypeScript and C++ using Zustand’s state management and nlohmann json
The JUCE + React structure was inspired by Spencer Salazar's talk at ADC in 2020 (https://www.youtube.com/watch?v=XvsCaQd2VFE) and hopefully can help make plugin development easier web developers and vibe-coders
We'd love to get your input and ideas for how to improve on it. In the near term, we hope to update the plugin to support on-device inference of the Magenta RealTime open-weights music model for offline use. In the long term, we hope to support future music models with improved controls, such as audio and MIDI input
Cheers!