One thing I wonder about .. he mentions CD-audio (Redbook?) as being one capability of the system. But the CD-Audio games like X-Wing vs Tie Fighter were much more limited in that sense. You'd literally just hear the music switch to the new track. And the Force Unleashed, the last game that used iMuse, wasn't particularly remarkable if memory serves. I wonder if that was a limitation they just couldn't quite make as seamless?
I figure today you could do it, with a "virtual MIDI" system using MP3 audio of individual instrument sounds ..
Edited to add: that last sentence is essential what a DAW provides.
And that's only the most obvious examples - games like Deus Ex and others have featured dynamic music transitions decades ago.
Modern games have similar reactive music systems but I've never heard one I felt was better than X-Wing's. They got it right on the first try.
This gave the series a leg up in that the music could actually communicate information effectively -- a tense moment, the shifting tide of the battle, the calm after a victory -- whereas other games simply had to put up waveforms that sounded pleasing.
To be fair many games experimented with sound design in this era, but few had such legendary IP to build with. An unfair advantage to say the least. The folks wielding iMUSE clearly knew what they had.
MIDI isn't really that much better, though - it's a compatibility-centric protocol, so it doesn't get at the heart of the issue with dynamic audio of "how do I coordinate this". All it is responsible for is an abstract "channel, patch number, event" system, leaving the details involved in coordinating multiple MIDI sequences and triggering appropriate sounds to be worked out in implementation. An implementation that does everything a DAW does with MIDI sequences has to also implement all the DSP effects and configuration surfaces, which is out of scope for most projects, although FMOD does enable something close to that.
I think the best approach for exploring dynamic and interactive right now is really to make use of systems that allow for live coding - Pure Data, Supercollider, etc. These untangle the principal assumptions of "either audio tracks or event sequences" and allow choice, making it more straightforward to coordinate everything centrally, do some synthesis or processing, some sequencing, adopt novel methods of notation. The downside is that these are big runtimes with a lot of deployment footprint, so they aren't something that people just drop into game engines.
Reinventing tracker music, in other words? =D
Games definitely do this.
You will be pleased to hear that plenty of games since then have continued to use that same technique, and there are in fact entire realms of game dev systems dedicated to enable that experience!
1. A video demo: https://www.youtube.com/watch?v=7N41TEcjcvM
2. Some details: https://mixnmojo.com/features/sitefeatures/LucasArts-Secret-...
(Whether or not the game actually does anything interesting with them is its own question.)
It's a fundamental limitation of CD audio. There isn't enough buffering to keep playing sound while the laser seeks to the next track, so there must be a gap. The gap isn't even predictable, the seek time will vary from drive to drive and even vary on the same drive.
With CD audio, your CD-ROM drive actually switches mode to become a regular CD player. The digital samples don't get sent to your sound card, the drive actually has all the electronics required to decode the digital audio and convert it to analog. All your sound card does is mix the analog output from your CD ROM drive with everything else.
The game can only really send "skip to track" style commands to the drive, more or less the same set of commands you could send with a proper CD player's remote.
From a musical theater composition perspective, it's almost like building around vamp sections: https://romanbenedict.com/vamp-safety-repeat/ - you build a neutral, repeatable motif that you can easily lay under unpredictably timed segments (e.g. spoken dialogue), that's primed to "explode" into a memorable melody whenever the on-stage timing calls for it!
The Force Unleashed: this is one of those "succeeds if it's invisible" things. The music is procedural based on mixing rhythmic and arrhythmic stems. That allowed continuous cross fades without needing to precisely match beats. That was a limitation again of not being able to precisely line up stems. The other fun thing that was introduced was physics driven synthesis. The DMM system fed information about strain, impacts, and other events into a granular synthesizer. The bussing and ducking architecture was derived from this paper by Walter Murch: https://transom.org/wp-content/uploads/2005/04/200504.review... Fun anecdote: I was at a party with some audio nerds, and raving about the paper to a new acquaintance, who interrupted me and said, "Oh, I wrote that!"
iMUSE was used for some really beautiful music in its time, so LucasArts had this figured out. But I'd be curious to learn how they did it.
Today we're starting to see the growth of the Musician-Programmer. Modern DAWs and middleware like FMOD and Wwise allow a lot of the dynamicism and layering for the musicians willing to learn it, dabble in just enough technical knowledge to be dangerous (as a good thing).
There's probably always going to be an interesting Venn diagram intersection of Musician and Programmer. They are related mathematically and historically. Most of the earliest versions of punch cards were for looms (weaving), then for music, with computers being the third application. The tools and processes change over time, but the relationships remain.
You must play MM1 and then MM2 to truly appreciate the difference.
Direct comparison: https://www.youtube.com/watch?v=JkMoHEFtnLQ
Hopefully the patent served more in creating a safe space for musical proceduralism in games rather than being chilling. As mentioned in this thread there has been brilliant proceduralism in so many games since Monkey 2.
I figured it was high time to highlight the innovations in iMuse because I realized in discussions that the core principles weren't well known, and difficult to extract from public sources like the patent unless you are really fluent with that language.
riotnrrd•9mo ago