Yet another silly audio visualizer. Why? I built this project for three reasons. First, I wanted to explore and keep up with the latest developments in software engineering, so I built this in collaboration with AntiGravity, Google's AI IDE. Secondly, I wanted to explore the current state of the "Web Platform" and understand the current capabilities of this evolving and ubiquitous platform. Lastly, I wanted to see what it would look like if I could animate an image to music in real-time within a web app. You can drag and drop your own .mp3s and .pngs onto the canvas to test it out.
I knew that performance would be a problem. Images have millions of individual pixels. So I applied Wheeler's Aphorism: “All problems in computer science can be solved by another level of indirection.” I store the image not as a matrix of RGB values, but rather as a matrix of indices into a fixed size palette of color values. I use the median cut algorithm to compute a new palette each time a new image is loaded. I can then animate the palette in real-time, and the only per-pixel operation is a single indirection through the palette.
I instructed AntiGravity on what to build and specified that I wanted the computation-heavy graphics parts (image quantization, palette animation, and image rendering) to be done in Rust compiled to WebAssembly (WASM). AntiGravity chose the architecture: React for the UI and loading of files, and the native Web Audio API to handle audio playback and the Fast Fourier Transforms to compute the power spectrum. To achieve 60fps without choking the browser's garbage collector we allocate static chunks of zero-copy memory buffers for the image, for the quantized image, and for the power spectrum values inside the WASM package instead of passing massive arrays of pixel data back and forth. JavaScript tells WebAudio to write the audio frequencies directly into Rust's memory, and Rust writes the modified image pixels directly into a shared buffer in the WASM package that the HTML5 <canvas> paints from. Having mutable static arrays is the antithesis of the Rust philosophy which results in all the functions living inside unsafe{} blocks, but I do believe this memory scheme minimizes data copying.
It’s been a long, strange trip getting here. I have been a programmer for 54 years, and have programmed home computers for 44 years. The entire software development environment (fig Forth) on my first home computer (Atari 800) lived on a single 90KB floppy disk, and ran in 32KB of RAM. Software distribution was either by copying a floppy disk or through a publisher and a distributor and a retailer. Today, I'm directing an AI to build an app that with one button push I can publish to the web with a potential audience of billions. I will say that the recent rise of AI assisted software development is by far the biggest jump in capability that I have ever seen. In fact, my reaction to the new paradigm of AI coding is best captured by the default image for this app: the Flammarion engraving.
I'd love to hear your thoughts, feedback, or any code critiques on the Rust/JS bridge!
If you'd like to chat or collaborate, my email is in my profile.
dmaynard•50m ago
Yet another silly audio visualizer. Why? I built this project for three reasons. First, I wanted to explore and keep up with the latest developments in software engineering, so I built this in collaboration with AntiGravity, Google's AI IDE. Secondly, I wanted to explore the current state of the "Web Platform" and understand the current capabilities of this evolving and ubiquitous platform. Lastly, I wanted to see what it would look like if I could animate an image to music in real-time within a web app. You can drag and drop your own .mp3s and .pngs onto the canvas to test it out.
I knew that performance would be a problem. Images have millions of individual pixels. So I applied Wheeler's Aphorism: “All problems in computer science can be solved by another level of indirection.” I store the image not as a matrix of RGB values, but rather as a matrix of indices into a fixed size palette of color values. I use the median cut algorithm to compute a new palette each time a new image is loaded. I can then animate the palette in real-time, and the only per-pixel operation is a single indirection through the palette.
I instructed AntiGravity on what to build and specified that I wanted the computation-heavy graphics parts (image quantization, palette animation, and image rendering) to be done in Rust compiled to WebAssembly (WASM). AntiGravity chose the architecture: React for the UI and loading of files, and the native Web Audio API to handle audio playback and the Fast Fourier Transforms to compute the power spectrum. To achieve 60fps without choking the browser's garbage collector we allocate static chunks of zero-copy memory buffers for the image, for the quantized image, and for the power spectrum values inside the WASM package instead of passing massive arrays of pixel data back and forth. JavaScript tells WebAudio to write the audio frequencies directly into Rust's memory, and Rust writes the modified image pixels directly into a shared buffer in the WASM package that the HTML5 <canvas> paints from. Having mutable static arrays is the antithesis of the Rust philosophy which results in all the functions living inside unsafe{} blocks, but I do believe this memory scheme minimizes data copying.
It’s been a long, strange trip getting here. I have been a programmer for 54 years, and have programmed home computers for 44 years. The entire software development environment (fig Forth) on my first home computer (Atari 800) lived on a single 90KB floppy disk, and ran in 32KB of RAM. Software distribution was either by copying a floppy disk or through a publisher and a distributor and a retailer. Today, I'm directing an AI to build an app that with one button push I can publish to the web with a potential audience of billions. I will say that the recent rise of AI assisted software development is by far the biggest jump in capability that I have ever seen. In fact, my reaction to the new paradigm of AI coding is best captured by the default image for this app: the Flammarion engraving.
If you want to read a bit more about the technical architecture, I wrote a short deep-dive on my blog here: https://software-artist.com/blog/rust-audio-visualizer
I'd love to hear your thoughts, feedback, or any code critiques on the Rust/JS bridge! If you'd like to chat or collaborate, my email is in my profile.