Music Generation AI Models

I wonder if this article is AI generated.

> Vocal Synthesis: This allows one to generate new audio that sounds like someone singing. One can write lyrics, as well as melody, and have the AI generate an audio that can match it. You could even specify how you want the voice to sound like. Google has also presented models capable of vocal synthesis, such as googlesingsong.

Google's singsong paper does the exact opposite. Given human vocals, it produces an musical accompaniment.

I got into AI music back in 2017, kind of sparked by AlphaGo. Started by looking at machine listening stuff, like Nick Collins' work. Always been really curious about AI doing music live coding.

In 2019, I built this thing called RaveForce [github.com/chaosprint/RaveForce]. It was a fun project.

Back then, GANsynth was a big deal, looked amazing. But the sound quality… felt a bit lossy, you know? And MIDI generation, well, didn't really feel like "music generation" to me.

Now, I'm thinking about these things differently. Maybe the sound quality thing is like MP3 at first, then it becomes "good enough" – like a "retina moment" for audio? Diffusion models seem to be pushing this idea too. And MIDI, if used the right way, could be a really powerful tool.

Vocals synthesis and conversion are super cool. Feels like plugins, but next level. Really useful.

But what I really want to see is AI understanding music from the ground up. Like, a robot learning how synth parameters work. Then we can do 8bit music like the DRL breakthrough. Not just training on tons of copyrighted music, making variations, and selling it, which is very cheap.

Are there models that generare MIDI instead of audio?

IMO this would be much more useful.

One obvious area of improvement will be allowing you to tweak specific sections of an AI generated song. I was recently playing around with Suno, and while the results with their latest models are really impressive, sometimes you just want a little bit more control over specific sections of a track. To give a concrete example: I used deepseek-r1 to generate lyrics for a song about assabiyyah, and then used to Suno to generate the track [0]. The result was mostly fine, but it pronounced assabiyyah as ah-sa-BI-yah instead of ah-sah-BEE-yah. A relatively minor nitpick.

[0] https://suno.com/song/0caf26e0-073e-4480-91c4-71ae79ec0497

From the article:

> Stem Splitting: This allows one to take an existing song, and split the audio into distinct tracks, such as vocals, guitar, drums and bass. Demucs by Meta is an AI model for stem splitting.

+1 for Demucs (free and open source).

Our band went back and used Demucs-GUI on a bunch of our really old pre-DAW stuff - all we had was the final WAVs and it did a really good job splitting out drums, piano, bass, vocals, etc. with the htdemucs_6s model. There was some slight bleed between some of the stems but other than that it was seamless.

https://github.com/CarlGao4/Demucs-Gui

In the future we may have music gen models that dynamically generate a soundtrack to our life, based off of ongoing events, emotions, etc. as well as our preferences.

If this happens, main character syndrome may get a bit worse :)

> code is now being written with the help of LLMs, and almost all graphic design uses photoshop.

AI models are tools, and engineers and artists should use them to do more per unit time.

Text prompted final results are lame and boring, but complex workflows orchestrated by domain practitioners are incredible.

We're entering an era where small teams will have big reach. Small studio movies will rival Pixar, electronic musicians will be able to conquer any genre, and indie game studios will take on AAA game releases.

The problem will be discovery. There will be a long tail of content that caters to diverse audiences, but not everyone will make it.

AI tools can also emulate analog signal processors like guitar amps (e.g. NeuralDSP). I made an emulation of a popular studio EQ that sounds great.

Are there any music generation models that work with sheet music or produce sheet music outputs that are actually good?