Realtime Procedural Audio and Synthesized Piano in Unity 3D

An introduction to mathematically generate the sound waveforms and create any sound you wish with endless variations

  1. How does the speaker generate sounds
  2. Programming the speaker cone
  3. Implementation details in Unity
  4. Creating musical notes
  5. Final thoughts and further improvements
  6. Further Readings

How Does The Speaker Generate Sounds

Generating sounds procedurally requires you to have a top level understanding of how a speaker generates sounds! Whatever I say here is an extreme over simplification. If you are interested to know more, I suggest reading more on the topic.

Programming The Speaker Cone

The speaker cone has one parameter which is changing over time, its displacement along its forward axis. In a normal audio recording, there would be an array of float values representing where the speaker cone should be as time goes on. An application would open an audio file, read this float array and provide it to the speaker driver. Given that, surely we can generate the values in this array procedurally and generate whatever sound we want.

Speaker Cone Displacement = sin(time* 2π *frequency)
Or
float waveFunction(time, frequency)
{
return sin(time* 2π *frequency);
}

Implementation Details in Unity

How do you implement the above in Unity? Can you just supply the speaker’s driver with a sinus function and call it a day? For many different reasons — which I won’t get into — the information is provided as a series of samples instead of series of wave functions.

for(int i = 0; i<data[].length; i = i +2) 
{
data[i] = waveFunction(currentThreadTime + i/sampleRate, freque)
data[i+1] = data[i];
}
for(int i = 0; i<data[].length; i = i + channels) 
{
data[i] = waveFunction(currentThreadTime + i/sampleRate, freque)
for(int j = 1; j<channels; j++)
{
data[i+j] = data[i];
}
}

Floating Point Accuracy Issues

For those of you who also write shaders, you might already know that, if you pass on the game Time to your procedural noise functions to generate procedural visuals, after a long time some really weird stuff can happen. This is due to accuracy issues that pop up when a float gets too large and it is used in a context where small variation are still of great importance. Let’s see how that applies to our case!

Keeping Track of The Phase

One way to deal with this problem is to keep track of the phase of the wave as it completes its cycle. This phase parameter will keep track of how far we are at a given time in the periodic cycle of a wave.

float increment = (frequency * 2π) / sampleRate
for(int i = 0; i<data[].length; i = i + channels)
{
phase += increment;
if(phase>2π) phase = 0;
data[i] = sin(phase)
for(int j = 1; j<channels; j++)
{
data[i+j] = data[i];
}
}

Modulo as Alternative

The dspTime which Unity provides the audio thread with is a double as mentioned. If you convert the time properly to a float for calculations, you can get away with generating your desired tone without having to keep a phase float parameter for every single wave you need to add to your procedural sound.

phase =(float)(totalTime %(1.0/(double)frequency)) * frequency * 2π;
phase =(float)(totalTime %(1.0/(double)frequency));
phase = (phase + i/sampleRate) * frequency * 2π;
data[i] = Mathf.Sin(phase);

Creating Musical Notes

You almost have a real life instrument! You can play different frequencies which correlate to different musical notes. The only problem is that musical instruments never generate a single wave but many different waves overlapping each other!

Notes With Harmonics

Generating harmonics along your fundamental frequency is actually quite trivial. To simulate the overlapping or superimposing which happens between different frequencies when you play a note, you can add a bunch of sinus waves together with different frequencies.

data[i] = waveFunction(time, fundementalFrequency       ) +
waveFunction(time, fundementalFrequency * 2.0f);

Note Envelopes and Playing Several Notes Simultaneously

When you play a note on the piano the sound doesn’t go on indefinitely. After the string is hammered it pretty immediately reaches maximum loudness and slowly decades away. You can shorten the decaying phase by letting go of the key you played which brings in the pianos damping system that stops the string from vibrating. How long a note is supposed to be held is determined by the music sheet!

Notice how while AD and R are duration S is a measure of volume/ loudness

Final Thoughts

Playing a Piece

The repo has a bunch of classes which can play a musical piece. The Pianist class takes in a MusicSheet instance and a beat per minute (tempo) and plays that sheet by using the press and release key API of the piano class we wrote above!

Visualizer

The visualizer contains an actual piano and representation for the wave form. In the demo I am zooming in the waveform for better legibility. Both are done using shaders. On update whenever the sound form has changed or a new key has been pressed/ released, I update a structured buffer containing all the nessecerly information and then procedurally draw it in a shader.

Further Improvements

Here are some stuff worth improving if you want to continue with a instrument synthesizer.

Further Reading

  1. Designing Sound by Andy Farnell, a book I highly recommend if you wish to read more on procedural sounds and theory of sounds — the physics or psychoacoustics: https://mitpress.mit.edu/books/designing-sound
  2. Procedural Audio Repo by Konstantinos Sfikas, covering how to do frequency modulation : https://github.com/konsfik/Unity3D-Coding-Examples/blob/7650111bf68f49287317810275db3ffbdae7d65f/3-Procedural-Audio/ProceduralAudioUnityProject/Assets/Scripts/Classes/SinusWave.cs#L26
  3. Short animation show casing how the piano string is played along with stoppers: https://youtu.be/2kikWX2yOto
  4. Show casing simple procedural sound usage in games: https://www.gamedeveloper.com/audio/procedural-audio-in-unity
  5. Sound Envelopes on wikipedia: https://en.wikipedia.org/wiki/Envelope_(music)
  6. Wikipedia article on the Equal Loudness curvers: https://en.wikipedia.org/wiki/Equal-loudness_contour

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store