I’m Ryan, lead author. I’ve been contributing open source generative audio stuff for a while now, audio reactive Comfy nodes, extended ACEstep support in Comfy, etc.. I just opened-sourced a new audio project that I've been working on for a few months and I want to tell y'all about it.

WHAT IS IS DEMON: Diffusion Engine for Musical Orchestrated Noise

This is StreamDiffusion but with audio instead of images, and ACEStep 1.5 instead of Stable Diffusion. It’s responsive enough that you can play it like an instrument, and remix in near real-time.

I also distilled the ACEStep VAE: it’s faster at the expense of some quality.

I also trained something like 200 lora/dora for ACEStep 1.5 and 1.5XL: I will release these in batches of 5 or 10 or something

WHY IT IS Two reasons: 1) Making music is an inherently real-time activity 2) Why not bro

SOME RUNTIME CAPABILITIES -Real-time remixing of songs -Denoise, structure, timbre strength adjustment -Reference track swapping -Prompt blending, parameter scheduling with curves -LoRA hotswapping, runtime strength adjustment -Latent channel (research preview) -Feedback -Vocal stem cutting/pasting with melformer (s/o u/BuffMcBigHuge) -XL support (its less stable, working out VRAM pressure issues and whatnot) -Lyrics/vocals SOON -Spectral quality SOON -Other stuff

SOME LIMITATIONS -ACEStep (correctly) ‘begins’ and ‘ends’ the song. This system is optimized for remixing either an entire song, or continuously remixing a loop. The loop works fine, but this is not pure, continuous music. Autogression wins here. -Many others, for a more exhaustive list, please see the full writeup via the project page -Please let us know if you find any, I would love to try and address them if possible

LINKS My YouTube (DEMON tutorial): https://youtu.be/FBv1b5gmjcE Github: https://github.com/daydreamlive/DEMON Project page: https://daydreamlive.github.io/DEMON LoRA: https://civitai.com/models/2416425/acestep-loras DreamVAE: https://huggingface.co/daydreamlive/DreamVAE Try it w/o installing: https://music.daydream.live