This is me having a go at real-time audio- (and human) reactive music visualization, made with untiy.
- Have Unity-scenes, midi-controllers and some real-time beat detection and frequency analysis.
- Use the midi-controllers to steer prepared or procedurally generated elements in the unity scenes. The elements might or might not be acting on their own based on the information gathered through frequency analysis and beat detection.
- Be able to improvise these "visuals" as my friends dj.
- Using Keijro's RtMidi wrapper, and a yet to be determined MQTT Unity PLugin I'm able to get user input from midi controllers or from an MQTT publisher into Unity.
- Using a very nice third-party programm called Wavesum, I get a beatclock signal (over midi) into Unity. This signal is detected in real-time listening to the microphone input of my computer.
- Using Unity's own FFT-Implentation I'm able to get a real-time frequency spectrum analysis.
- I like the idea of molding the elements of music, as you listen to it, in visual shapes and behaviours. To me, adding a second dimension to the music visually would mean to support it in telling its story.
- When I was looking for solutions to do this, the only really viable one I came across was Magic Music Visuals. It has a great community and tremendous devs, but due to some limitations (by design), I started looking further until I acknowledged, that a game engine like Unity is probably the perfect tool for me. The great advantage of Magic Music Visuals is that you don't need any programming knowledge, which was what scared me off game-engines earlier as well.
Nothing really, this repo is for us to have a nice Git and Github workflow, and I don't see a reason to keep the code private. Most likely the project is not going to compile or run for you though, as there are assets in use, that I'm either not willing or able to share. Have a look at the licence, use at your own risk :), as usual.
If you still want to try, continue reading.
- have a look at ProjectSettings/ProjectVersion.txt to find out which unity version to use
- clone repo
- open in Unity
- resolve problems resuling from the last paragraph above. Specifically:
- remove the line starting with "ch.brogli.richfx-as-unitypackage" from Packages/manifest.json. Rich FX is a great asset from the Unity Asset Store. For my convenience I've packaged it into a seperate repo, so the Unity Package Manager can automatically pull it and it doesn't clutter the Assets folder. Since this is a payed asset, I can't share this here. So you either have to buy the asset and replace any references to it, or remove any references to it allotgether.
- any videos, images, image sequence or other files that I keep external of this git repo and the runtime itself. They're referenced by the app, so you'd have to remove any reference or live with the exceptions.
Use Ctrl+N or File > New Scene, then select the BSII_templateScene template. Save your new scene to Assets/_bsII/_Scenes.
The template scene contains "TempDevObjects", these are needed while developing the scene, but will remove themselves once in play mode. You can ignore them.
Please clone the volume profile (on the main cam), otherwise your changes there will edit the profile used in the templateScene. You can disable most overrides, except you must not remove "camera copy". This is needed so the output of the main cam is picked up and displayed on the UI cam.
Once you have finished implementing your scene, you must add it to the build settings. Then in Assets/_bsII/Resources/SceneScreenshots add a screenshot of your scene with its filename matching its scene index. It will then get picked up by the UI and scene selection logic.