Shader Studies: Matrix Effect

Break down of the matrix shader effect written by Will Kirby. And implementation of a real time matrix Shader in Unity 3D with Triplanar mapping. This means the shader can be dropped on any mesh and it would work without the need of a specific asset preparation.

Original Shader: https://www.shadertoy.com/view/ldccW4

My Github Repository containing the code: https://github.com/IRCSS/MatrixVFX

In this post I will first look at the shader toy implementation and explain the lines, with the hope of demystifying some of the standard techniques, and then explain very briefly the Unity implementation.

Main

When you want to understand a shader, one place you can start is the very end. In this case, the main function is a good starting point.

Starting from the main function, there are two main components to this shader. The rain drop effect, and the text. One good way to understand how things work, is to isolate each function and visualize only the return value of each.

void mainImage( out vec4 fragColor, in vec2 fragCoord )
{
 fragColor = vec4(text(fragCoord)*rain(fragCoord),1.0);
}

The rain effect works as a sort of a mask, which hides the text. Mathematically this is done through the multiplication, when the rain function returns 0, that pixel will be black. If we take out the text part and only show the rain, we will see this:

void mainImage( out vec4 fragColor, in vec2 fragCoord )
{
 vec3 rain = rain(fragCoord);
 fragColor = vec4(rain,1.0);
}

I will look in the rain function later. But what this is, is basically columns moving down.

If you isolate the text component by taking out the rain component, you will see a bunch of characters in a grid. These characters are animated, and each cell switches to a new character in fixed intervals. More on that in the text function.

void mainImage( out vec4 fragColor, in vec2 fragCoord )
{
 float text =text(fragCoord);
 fragColor = vec4(0,text,0.0,1.0);}

Text Function

This function is responsible for creating the visualization of the characters. Here is the entire function body:

float text(vec2 fragCoord)
{
 vec2 uv    = mod(fragCoord.xy, 16.)*.0625;
 vec2 block = fragCoord*.0625 — uv;
      uv    = uv*.8+.1; // scale the letters up a bit
      uv   += floor(texture(iChannel1,              block/iChannelResolution[1].xy + iTime*.002).xy * 16.); // randomize letters
      uv   *= .0625; // bring back into 0–1 range
      uv.x  = -uv.x; // flip letters horizontally
 return texture(iChannel0, uv).r;
}

The main job of this function is to return a color per pixel which creates a different (desirably random) character in each grid cell.

The first two lines, create the grid structure with cells. Each cell holds a character. This is a very common technique in shaders, to sub divide the space in to a repeating grid used for creating repeating effects. More on that in Book of Shaders.

Typically a fract or mod function is used to obtain a coordinate system per cell. Which can be used as uv coordinates to sample a texture. A floor function is used to get a unique id for each cell. Another way of getting the unique id (which is what is done here) is explained below.

Together with a random number generation, the unique id can be used to add variation. Although you are repeating the space and doing the same calculation for each cell, due to the difference in the unique id, you end up with different visuals. This can be used in our case to sample a different letter for each block.

You can visualize this by outputting the first line of the function as your final color. You will get a bunch of grids going from 0–1 in both direction (like uv coordinates).

void mainImage( out vec4 fragColor, in vec2 fragCoord )
{
 vec2 uv = mod(fragCoord.xy, 16.)*.0625;
 fragColor = vec4(uv.xy,0.,1.0);
}

This is mathematically equivalent of doing a fract.

vec2 uv = fract(fragCoord.xy /16.);

The fragCoord are not normalized here, so they go between 0 to whatever the width/ height of the canvas is in pixels. That is why fract(uv/16) would make a repeating tiles every 16 pixels.

In the original line 0.0625 is 1/16. The reason why I prefer fract is purely because that is what I learned first and I am more used to. There might be other reason why you should use one or the other.

Line two of that function calculates the unique id per block.

vec2 block = fragCoord*.0625 — uv;

The idea is simple if I manage to explain it well, if you divide the number A by B you kind of have two components together in one number. One is how many Bs fit in A without any fractional rest, the other is a fractional rest. And together they make A/B. For example 7 apples divided by 3, is 2 apples plus 1 rest:

3*2 + 1 = 7 or

(2.0 + 0.33…)*3 = 2.33… *3 = 7

Here:

A = 7, B = 3, A/B_WholeNumber = 2, A/B_Fraction = 0.33… and A/B = 2.33

Very simple math, yet powerful for creating repeating coordinate systems. In the line two of the function the uv is the fractional component and block the whole number component. Notice how in the above notation you can always get one of the numbers from the other two. If you have A/B and A/B_Fraction you could get the wholeNumber component by doing one minus the other.

A/B_WholeNumber = A/B -A/B_Fraction

This is exactly what is happening on the second line where the block is calculated. Just to make sure you understand this, run an example for a pixel. What numbers would you get if you are running this code for pixel (24, 0)?

Hopefully the above makes sense. Using these two components we can now sample a random letter from the font texture. The font texture is a grid made up of 16x16 letters. To get a random letter, we need to choose a random cell from these 16 in 16 grids. Given a random number generator, we can pass on the block id we generated above and use this random number to get a random offset to a random grid.

This line is doing exactly that:

uv += floor(texture(iChannel1, block/iChannelResolution[1].xy + iTime*.002).xy * 16.); // randomize letters

The iChannel1 contains a noise texture, the author is dividing it by the texture resolution, so that each block in the grid corresponds to one pixel to the noise texture. This helps to avoid artifacts such as aliasing and makes the best use of the noise texture. The iTime addition is the part responsible for the letters changing. If you take this out for example, you will still have the rain effect, but on static letters background.

As you can see this random value is then added on top of the fractional part we calculated before. Since the random value goes between 0 and 1 we multiply it by 16, to get a random offset between 0 and 16 in both x and y components. The floor function is there to insure that our offsets are only whole number jumps and are the same for all pixels of the same grid.

After this operation, the uv holds a value between 0 and 16. Since the font texture has the uv range of 0–1, we divide the uv by 16 to renormalize our coordinates.

uv *= .0625; // bring back into 0–1 range

With this uv, we can finally sample our font texture and return its color. We are using only the red component since in this font texture the red is a mask for the letters.

return texture(iChannel0, uv).r;

I implemented this exact same function with my own syntax in Unity with comments. You can have a look at it for further break down.

Rain Function

The rain functions main responsibility is to create the columns that move down.

Here is the entire body of the function:

vec3 rain(vec2 fragCoord)
{
 fragCoord.x -= mod(fragCoord.x, 16.);

 float offset = sin(fragCoord.x*15.);
 float speed  = cos(fragCoord.x*3.)*.3+.7;
 
 float y  = fract(fragCoord.y/iResolution.y + iTime*speed + offset);
 return vec3(.1,1,.35) / (y*20.);
}

Let’s analyze this function from the end. If we output only the simplified version of the last line:

void mainImage( out vec4 fragColor, in vec2 fragCoord )
{
 float   y = fract(fragCoord.y/iResolution.y );
 fragColor.xyz = vec3(.1,1,.35) / (y*20.);
}

We will see a gradient from top to the bottom. The fragCoord.y divided by the resolution of the screen normalizes the float y in a way that it has the value of 1 at the top and 0 below. Multiplying it by a large number and dividing the final color by it will cause the top part to be black and bottom part to have a large value.

If you add time within the fract function, you will be moving this gradient. This is already much closer to our desired result.

void mainImage( out vec4 fragColor, in vec2 fragCoord )
{
 float y  = fract(fragCoord.y/iResolution.y + iTime*0.5 );
 fragColor.xyz = vec3(.1,1,.35) / (y*20.);
}

Now the question is, how do we make columns out of this? First we need to create these columns, then make sure in each of these columns the gradient moves down at different speed and with different offset.

This is what the final version of the function does. In order to give a random offset and speed to each column, it again uses the mod technique to divide the screen in columns, calculating a unique id for each column and using that plus a sinus and cosines function to give each column a different value for speed and offset. The sinus/ cosines could be replaced by a random number generator, although that wont be necessarily here.

Color Exercise

One good way to find out if you have really understood something is to modify it yourself. Here is a challenge, let’s say you want each of the rain drops to have a different color. How would you do it?

A tip is to start giving each column a different color, then finding a way to differentiate between different drops within the same column.

If you just want to see an implementation, here is mine for the colored matrix effect.

Applying to a Scene

How could we use this in actual production? This effect is running on the pixel shader of a 2d surface. If you want to apply this to a mesh, you have a couple of options.

I am using a mesh from Global Digital Heritage which is licensed under CC non commercial use.

To apply this effect on a 3D topology, you need a parameterisation of the mesh, which would describe its surface as 2D coordinates. The most common way of doing this is a uv unwrap.

If you apply this effect on the unwrap of a mesh, you might get exactly what you want, or you might notice artifacts on the edges of islands. If this was a film production, or a game where you don’t have to apply this effect to a lot of meshes, uv coordinates would have been the best option. Mainly because it gives the artist complete control in which direction the columns of text move.

But if you want to apply this on every mesh in the game, and dont want to create a complicated asset creation pipeline to parameterise meshes in specific ways, you can use triplanar mapping. I am not going to go in depth on what triplanar is and how to use it, since there are a LOT of good posts on this. Examples are from cat like coding, ben golous and Ronja.

Here is my implementation with triplanar and the matrix effect. The lines are very well documented, so you should be able to follow along fine.

There are some differences between my version and this. I am generating the noise texture on a compute shader, and also controlling there how often the cells refresh their character. Other than that, I have also the option of using the colored matrix effect, plus some minor difference in how I calculate the block ids and uvs.

As usual, thanks for reading. You can follow me on my twitter: IRCSS

Resources

1. Original shader on shadertoy: https://www.shadertoy.com/view/ldccW4
2. Catlike Codings triplanar post: https://catlikecoding.com/unity/tutorials/advanced-rendering/triplanar-mapping/
3. Ben Golous’ post on triplanar: https://medium.com/@bgolus/normal-mapping-for-a-triplanar-shader-10bf39dca05a
4. Ronja’s post on triplanar: https://www.ronja-tutorials.com/2018/05/11/triplanar-mapping.html