You want a rhythm follow ? need only find out when the beats happen and take action?
I can not do it for you, but I can give you a lol north ...
These algorithms use a filtros combo to decompose the audio signal into different frequency bands, the most noticeable beats will always be in the lower frequency bands, if you apply filters in the desired frequency bands you will have to return the envelope of the amplitude contained in the audio, the envelope of the amplitudes will signal where the beats are happening ...
I've written a brief explanation of how this works for a long time now here , my code calculates how many beats per minute a song has, to identify this you need to find the beats contained in the audio, you can use this in your code and take action ...
My code to calculate bpm (beats per minute) will solve only half of the problem, you will also need to identify the frequency of the beat (serious or acute), for this you will need to develop a Pitch track , I developed one for tarsos , my code can be found #
There is a simpler way to identify the beats in real time, you can calculate the difference of decibels of the current frame with the previous frame, if the decibels exceed a certain limit proposed by you a beat happened, I wrote something very simple of as this can be done here , of course this method is not anything robust.
If the beat is not important, do you just want to find out where the bass and treble are? you'll just need a pitch track, important detail my code for pitch track will not work well on polyphonic audios ...
EDIT
I took a look at youtube in melody's escape gameplays, it's clear what the algorithm does, it processes all the audio before, it analyzes piece by piece, it discovers how many beats per minute the audio contains, it repairs when the beats by minute is high the game gets faster, it also processes the pitch of the audio, this is done offline (before starting the game), after finishing the analysis it will already know which are the audio locations where the beats are fast or slow, where it has severe and acute tonalities, after that he only needs to mount the level / obstacles of the game based on the analysis, of course I noticed that he also does other analysis to leave the rhythm sensation of the player more real, for example calculates if it has an abrupt difference of beats, when this happens without the presence of druns e kicks the algorithm focuses on the extraction of the pitch (frequency), it focuses on if some frequency has support and creates a sliding bar for the note in question ....
