Some of these songs are _really_ good. And pretty far ahead of where I thought audio AI was. Does anyone know what models and tools are used? I imagine once you have stems you can maybe style / genre transfer and use a reference audio file perhaps? Like image and video models.
When I composed this piece of classical music over nearly a decade ago, it was originally intended for strings. A few months ago I tried using GenAI to do a cover in a chamber quartet style by providing a WAV file of the original piano performance and it stayed almost 1:1 faithful to the original in terms of motifs and structure.
Related blog post with comparisons of the original piano performance and the cover for reference.
incomingpain•3mo ago
Where you can input your lyrics, styles, etc. You can create all original songs by having AI generate the lyrics even.
>Does anyone know what models and tools are used?
One of the best in my opinion: https://huggingface.co/tencent/SongGeneration
One of the most popular: https://huggingface.co/facebook/musicgen-medium
The "transformers" is just a python library and you just add all the parameters in a constructor:
processor( text=["80s pop track with bassy drums and synth", "90s rock song with loud guitars and heavy drums"], padding=True, return_tensors="pt", )
This one can run on like 16GB of vram and modern cards produce them trivially.
>I imagine once you have stems you can maybe style / genre transfer and use a reference audio file perhaps?
You can do audio to audio; but it's not as easy as just generating the whole thing from text.
afro88•3mo ago
I had a look at what you provided. The "ai song generator" websites and these models aren't able to genre transfer with the exceptional quality of the linked youtube channel. These have incredibly good alignment to the original songs, but the genre totally transformed. For example, the Nirvana one on that channel. Song structure is entirely preserved, as well as melodic and harmonic content. But the genre is transformed to 80s synth pop. It's the same for any recent upload to that channel.