Painting with Code

Combining simple mathematical functions, you can paint beautiful images, practice your maths skills and wrap your head around how to program for SIMD style concurrent programming.

This painting was done as a demo for my talk in Codetalks 2023. You can find the code on my Shadertoy, and a recording of the talk will be uploaded later to Youtube:

https://www.shadertoy.com/view/dtSfzR

This painting done in this demo is based on the works Ryo Takemasa.

If you perfer a video version, I have covered this topic in Code.Talks Hamburg 2023:

What is a Moving Painting?

The image below is what I call a moving painting (code also available on shadertoy)

Everything in the image is generated mathematiclly. I am not using any images. No Photoshop or GIMP is involved. The whole thing is around 600 lines of code, which is rendering everything you see. This painting is so low on tech stack, that you can program it in any programming language within minutes. You can run it on an excel sheet, or a graphical calculator.

Because everything is generated on the fly, I can continue this painting forever, without the stuff ever repeating themselves. I can easily adjust how the painting looks, and generate variations of it.

My fascination with these paintings started with watching Harry Potter. In the film there are magical paintings and photographs. The magic is that the images are alive and continue on forever. I wondered whether I can combine my love of maths and arts to create something similar.

But the initial motivation aside, why do I continue making these? Why do I think maybe you should too?

Why Paint with Code?

I think creating these images make me a better programmer. Let’s start by how you would write a program like this.

Imagine you have a grid. The grid is made out of cells. You have properties associated with each cell, for example the cell coordinates.

Our grid is going between 0 and whatever the cell dimension is. In this case buttom left would be X,Y=(0,0), top right (16,16) and middle (8,8).

Now imagine we write a very simple program. It contains two nested for loops.

for( int x = 0; x < 16; x++) {
    for(int y = 0; y < 16; y++) {
   { 
        Cell_Color = PaintPixel(x, y); 
   }
}

All the program is doing, is iterating through our grid horizontaly and vertically. That means visiting each cell once.

On each cell, we call a function called PaintPixel. This function can be a black box for now. All that matters is that the function calculates a color (a RGB value) which we assign to the color of our cell.

Visually, the process looks something like this.

We are visiting each cell once per frame, and calling the following PaintPixel function:

vec3 PaintPixel(x, y) {
return vec3(x/16.0, y/16.0, 0.0);
}

This is one of the simplest functions you can write. All it is doing is visualizing the coordinate system. We take the X and Y coordinates and divide it by the dimension of our grid. This normalizes the coordinate system so that it goes between 0 and 1.

Then we assign the X coordinate to the Red channel and the Y coordinate to the Green channel. This would mean that as we go from left to right, our cells will get redder and redder. If we go from bottom to top, they will get greener and greener.

Top right, where it is equally green and red, we get a yellow/ orange which is the mixture of both.

There are two important points to notice about this setup.

Point number one, we are calling the exact same function to color all the cells. It is not like we shade the cell 0,0 with one function and 5,6 with another. But we want the cells to have different colors! How can we ensure that we get meanigful variation in output between the different cells, if we apply the exact same transformation on all of them? Tricky stuff!

Point number two, this function is incredibly stupid. The process of calculating the color of a cell is not dependent on the results of any other cell. In other words while calculating the cell (0,0), it doesn’t matter whether cell (5,10) is already calculated or what the result is. The lack of dependency between the cells or to global information is unique.

These two points mean that we can easily convert our algorithmn to multithreaded! Typically solving a problem in a way that is faster multithreaded is not that trivial. We would need to deal with problems such as how to divide the task between different threads and how to deal with conflicts when the parallel threads want to write to the same information.

With this program we have neither of these problems. We can generate a thread per cell, and each thread executes the function of the cell whenever it wants. Since there are no dependencies and the threads are in charge of calculating different cells, they can execute whenever they want, without any racing conditions popping up!

This sounds like a very nice paradigm for concurrent programming. As a matter of fact it is and it is called SIMD.

SIMD Programming

Single Instruction, Multiple Data is a way of thinking about how to solve a problem. You apply the exact same process on a lot of data points. It is usually useful if you have a lot of data which you need to process with the exact same transformation. However, you would be surprise how much of the real world problems can be transformed in clever ways to fit within SIMD paradigm.

At its core, SIMD is closer to something like functional programming. Once you express a problem in this way, it provides an easy to implement multithread algorithm for the problem. This algorithm runs per design very fast. However, SIMDs are quite challenging to program for!

Most programmers are used to single threaded Object Oriented programming. You have classes that have parameters and methods. You call those methods which call other methods of other classes. Everyone talks to everyone, a player shoots an enemy, the enemy talks to a manager, the manager triggers an audio system etc. Functions within these classes are encapsulated within an instance. You have access to so much memory to express internal states of your instances. The entities in your program constantly change the internal states of themselves and eachother as the program progresses. Based on these states the different instances can have vastly different behaviours.

You can do absolutely none of these in SIMDs.

At its core SIMDs functions are stateless. They take an incoming data, and they mass transform it to an outgoing data. The lack of access to these states is half the problem. As I mentioned before, you still need to figure out how to get the desired meaningful output variation, despite using the exact same transformation. In other words whatever variation your output has (different colors between the pixels), is solely based on the variation in your input data (in this case the coordinate of the cells).

This is very very challenging way of programming for those used to single threaded OO. It requires a different way of seeing problems and thinking about it.

Yet SIMDs are very important! The GPU architecture is based on SIMDs. So all the amazing stuff you can do with them requires you to master SIMDs. This includes creating visuals for games and films!

Various simulations and iterative solvers can also be expressed in SIMDs and get a boost in speed. This touches anything from weather forecasting to medicine and various fields of science. Data compression, computer vision and audio processing are also some of the fields within computer sciences which get a boost from SIMDs.

I hope I gave you a bit of a background on how important this way of thinking is, and how hard it is to actually do it. Let’s say you wanted to practice this style of programming. If you decide to use any of the domains I mentioned for practicing, you would have to deal with so much overhead! Take fluid simulation for example. You would need to understand the maths and physics, which can take you months! Your actual program will be made out off 80 percent non SIMD code which deals with loading resources, changing states, cleaning up the programmer etc. All this overhead to get some practice!

The painting program I propose is simply few lines of code. 1 minute and you are ready to practice!

Similarly, programming paintings is a fantastic way to get in shape in your mathematical skill. These train your maths muscles which you can regularly use in your actual job.

Last not but not least, this can give you an artistic outlet! Making art is wonderful! It is a rewarding process for you and those around you. Yet it is hard to make art. It is hard to sing, dance and draw. If you are decent in programming and maths, this can be a new medium where you can express your self and have fun. As a matter of fact some of the best people in this field can’t draw a stickman to save their life!

Building Blocks

I hope I gave you some background on why we do this. Now let’s talk about HOW we do these. I would like to present to you some of the building blocks that you would need, in order to create images like these!

Shaping Functions

This is Sinus. One of the most amazing functions I have come across. So much of our natural world can be expressed using this simple function. From the way water moves, to the way clouds form, or how the ground dries! It is in the shape of flower petals, or shells of animals.

One of the things I learned in highschool maths, was that we can visually do things to a function. If we add or multiply a function with a constant, we can move it around.

f(x + c)
f(x) + c
f(x * c)
f(x) * c
abs(f(x))
f(abs(x))

If you multiply a function within the bracket, you stretch it horizontally. In other words if you multiply the X before you apply the transformation.

If you multiply a function outside of the bracket, you stretch it horizontally! In other words, once you are finished calculating, multiplying the result by a number.

You can move a function left and right if you add a number within the brackets (before applying the transformation to the X).

Similarly you can move the function vertically.

These are called shaping functions because they enable you to mold, sculpt and shape functions so that they look a very specific way.

These are not just useful for procedural graphics, but also procedural audio and game play programming. Any time you have an entity which needs to behave a certain way over time (or any other dimension), you can construct a mathematical function that describes this behaviour using shaping functions!

For example let’s say we want an animated Sinus function that moves forever. We can take the application time, which is a timer that starts at 0 at the beginning of the application and counts up to infinity every frame, and add that to our sinus function!

Y = Sin(X + Application_Time);

Since adding to the function moves it horizontally, this means the wave will move from right to left forever.

Adding constants to a function is not all we can do. We can also add, subtract, multiply or divide functions with eachother. In physics the process of adding two waves is called superposition. If you add two functions to eachother, you will get a third function which is visually a combination of the two put together.

Notice how the third wave looks visually more detailed. It is as if we took the first wave, and added the second wave as a detail on top.

We can take the concept above, and apply it to our moving sinus function.

Let’s take the function above, and color it:

The way we color it is simple at heart. For each cell, we evaluate the value of our wave function for that given x. This tells us how high the function is at that specific slice of X. Then we check the Y coordinate of that cell, if it is smaller than the value of the function, the cell is below the wave, if it is bigger it is above.

if(Y < wave(x)))
PixelColor = darkBlue ;
else
PixelColor = lightBlue;

Let’s look at the image above again. Doesn’t it kind of look like a mountain? Or maybe water waves, or clouds?

For a mountain the wave is a bit too soft right now, so let’s see if we can use shaping functions to make it look abit harder. First thing first, we can switch from sinus wave to a triangular wave.

Here is the formula for how to generate a triangular wave. It is a good example of how to use shaping function to go from a linear line to the jigsaw shape.

f(x) = abs(fract(x) — 0.5)

We start by a straight line, then we use the frac function to create a repeating pattern (more on this later). We move the line down so that half of it is below zero, and use absolute to mirror it across the X axis. I won’t cover this in more detail, but this is the core idea.

If we switch the sinus wave for the harsher looking triangular wave and add several triangular waves together, we get something like this:

float mountain = 0.0
 for(int i = 0; i< 5; i++){
    mountain += TriangularWave( X * period) * amplitude 
    amplitude *= 0.85
    period *= 1.25
 }

In this case we are adding 5 different triangular waves together. In each iteration we are increasing the frequency and decreasing amplitude. Like this each iteration is responsible for adding tinnier and tinnier details.

We can pack the code for this in a function. Once we have it in a function, we can call it a bunch of times. Each time we can change the color of the mountain or where it is drawn on the screen.

Remember, this is a function. As demostrated before, functions can move, so we can get this:

We can move each layer at a different speed. Due to perceived parallax, our brain starts imagining a depth to our image.

Spaces and Coordinate Systems

So far everything we have done has been in the Cartesian coordinate system. This is the standard left/ right, up/ down space. However, this is one of the many spaces we can operate in. For example let’s consider the polar coordinate system.

In the polar coordinate system, everything is defined with respect to a circle. For every point in the space we draw a line from the origin to the point. We are interested in 2 properties.

First, how far away is the point from the origin, in other words the radius. Second how far along is the line along the circle in rotation. This is the theta angle.

Source Wikimedia https://commons.wikimedia.org/wiki/File:Point_in_Polar_coordinates_grid.svg

We can take a very simple funciton such as Y = 1. This function is a simple straight line that goes from left to right. If we draw this function in polar coordinates instead of the cartesian one, we get a circle. To understand why this happens, we need to imagine how the space itself is wrapping around a circle. If you wrap a line around a circle, you get the circumference of a circle.

The main point here is that the same function looks different in different spaces. We commonly construct very spatial spaces which make it easier for us to draw a certain thing or do certain calculations.

One such calculation could be coloring any pixel that is within a certain radius. Doing this a bunch of time, we can get these series of nested circles.

A line is not the only function you can wrap around a circle. You can wrap any function around a circle. For example here, I am wrapping a wave around a circle.

If we take a sinus and wrap it around a circle, we start getting something that looks very much like flower petals.

Similarly to how we constructed the triangular wave, we can start constructing a specific wave function that looks a lot closer to flower petals. Using shaping function we need to ensure that the top is very thin, the middle is the thickest and the base is half as thick.

I am skipping over how I am applying shaping function to get those petal shapes. This post is already long as is, so I will leave that for you to figure out from the code.

This is still too regular for my taste. We can further use shaping functions to deform the flower so that it looks more natural.

We can say that the further the petal gets from the center, the more we would move it left and right using some sort of wave function. Below is the deformation maths itself.

polar.x += Sin(polar.y);

It is worth mentioning that these are still mathematical functions. Functions take in parameters. So if you call the same function again and again with different parameters, you would get different looking flowers. This is where the endless variation comes from.

Domain Repition

I mentioned that we would talk more about the function Fraction.

f(x) = Frac(x);

This little function is one of the corner stones of computer graphics. If you give fraction a number, it keeps only the fractional part of it. In other words, if the input number is between 0 and 1, it won’t touch it. If it is 1.7, it will return 0.7. 5.3 becomes 0.3 and 3.5 becomes 0.5.

What is happening visually is that the function takes its shape between 0 and 1, and is repeating it again and again along the X axis. This practically means that it breaks the space down in a repeating pattern!

vec3 PaintPixel(x, y) {
return
vec3(frac(x), y, 0);
}

For example, we can take our simple Paint Pixel function which visualized the coordinate system, and call Frac along its X axis. As you can see, we get a repeating set of tiles. We can do the same thing along the Y:

vec3 PaintPixel(x, y) {
return
vec3(frac(x), frac(y), 0);
}

Now we are repeating in both dimensions, which gives us a grid within a grid. This repeating grid is how we do tiling textures in video games, which enables us to add detail to huge CG worlds!

What we now have, is a repeating coordinate system that goes between 0 and 1. Which is good news for us, because all the functions we have written so far, can be called within this space. For example, we can call the Sunflower function and get this:

We can start playing around with this idea of fracturing spaces. How about the space fracturing more and more the closer it gets to the horizon?

We are starting to get something that looks like a flower field! But it is still not there. The flower placement is too regular and the flowers never overlap! So we can pack what we have in a function, move it around and call it a bunch of time, and we start to get pretty decent results!

Here starts the refinement process. Adding detail and adjusting visuals using shaping functions. For example the closer we get to the horizon, the more we blend the color to yellow. At a certain distance we can’t identify individual flowers anymore and only see a sea of yellow!

With further refinment, we can add noise to the surfaces:

Constructing noise is a very straight forward business. We sample a very high frequency wave function, and interpolate between these samples. Then we add this visual noise on our images:

Remember, these are still mathematical functions, so we can move them around at different speed, and get depth!

I already added the clouds to the shader. Here is something for you to think about, how would you do the clouds? It probably starts with a sinus function!

Beyond 2D

There is no reason why you would stop at 2D.

The two above images were done in Blender Geometry node, which is also effectively a SIMD structure. The same concepts discussed here for 2D are applied on 3D surfaces which you can also move around. Because everything you see there is defined mathematically, I can easily change the scene from winter to fall!

Or my endless Chinese landscape painting generator. Beside the calligraphy, everything else is mathematically defined!

There are even more fun ways to generate scenes like these using methods such as ray marching. If you find the 2D version fun, your journey is just beginning!

Hope this gave you some inspiration to get started with your own project. If you have any questions or wanna get in touch, you can find all the info you need on: https://ircss.github.io/