Real-time Cubism Shader
I was recently reading the art book of Spider Man Into the Spider-Verse and got inspired by their art style choices for the portal scenes and the influence of cubism on it. So I decided to create a Cubism shader in Unity 3D.
As usual you can find the code on my Github: https://github.com/IRCSS/Cubism-Shader
You can also try it out in either normal PC or VR by downloading the build: https://github.com/IRCSS/Cubism-Shader/releases/tag/v1.0
The mesh used for show case the shade is from The Hallwyl Museum, scanned by ErikLernestal.
Cubism is a good match for the themes of Spider man: Into the Spider Verse. Dean Gordon an art director on the project states in the book: “We knew we wanted to represent parallel worlds from Multiverse intruding into our own. Cubism seemed a natural idea since it represents multiple views of a scene within a single picture frame. So in our version, space is fractured and within each fracture we see a different angle, scale and rendering of the same place. Each fracture shows a a parallel but slightly different version of our own world existing in another universe and now trying to resolve itself in ours. ”
I love the idea of a fractured space. Those that know my other experiments, are familiar with my ever lasting quest of trying to confuse my brain. I had some ideas on how to recreate that feel in real time rendering and decided to give it a try.
The general three ways that I could think of were:
- Render the scene multiple times from different angles with different rendering settings, and composite them per hand on top of each other.
- Render the scene the same way, but with automated compositing. Either by projecting all renders on to the same depth buffer or some screen space segmentation/ blending.
- Divide the space in different segments and alter the MVP matrix based on those segments in the shader. The MVP matrix is used for projecting the geometries on to the screen.
I suspect the number 1 is what they would do in a film. It gives the highest artistic control and you can always guaranty interesting composition that supports the themes of the scene. For obvious reasons, that one is out for real time.
Number 2 is super interesting. I can imagine you could get really funky stuff, like duplicated meshes coming out of each other, each shaded differently. It will come closer to some of the prob in the film and how they looked.
The number 2 is a very expensive effect. You would have to render the scene several times, which on the CPU side means a lot of time on the GPU driver for state changes, and on the GPU lots of vertices to process. I think I will experiment with it at some point, but I decided to go for something more real time.
The number 3 is a good pick for real time. segmenting the space in vertex shader is trivial, and each mesh is only rendered once. If you want each segment to be rendered with different rendering setting, you could switch to deferred, and output a material ID from the vertex shader based on those segments. In this post, I am going to only focus on the fractured space part.
Segmenting the Space
Segmenting the space in 3D is very much like segmenting it in 2D, but with the added dimension. I usually start with a grid, and work my way from there. In this shader you can see the code for how to achieve this effect.
The idea is that you divide the global coordinate system in endless numbers of repeating segments. For each segment you assign an ID. the segment next to the origin for example is (0,0,0), the one to its right is (1, 0, 0) and so on.
In the scene below, for visualization, I am using these IDs as a unique seed to generate a color for each segment. For better understanding, read the Book of Shader’s explanation on how to do it for 2D and look at my code for 3D.
Voronoi Segmentation
Grid segmentation is too regular. It doesn’t feel like a fracture. A good fit for that is voronoi. In the VoronoiSegmentation shader you can see how I am achieving this.
I offset the center of the grids segments by a random offset and per vertex calculate the id of the closest neighbor to that vertex. Using that ID, I am generating a random color for that vertex, which ends up with the following effect.
If you want to draw lines between each voronoi segment to enhance the cubist feel, have a look at this technique by Inigo Quilez.
Modifying the View per Segment
The idea is that for each segment the eye is placed in a slightly different place and rotated a bit to still look to the same object. For this we need to modify the ModelViewProjection Matrix. Specifically we need to modify the View matrix.
The View matrix is basically the World to Object matrix of the Camera game object. It is made up of:
There are two steps to what we want to achieve. One is to move the camera for each segment by a slight nudge. This we can achieve by adding a vector to the 4th column of the View Matrix. Second is to rotate the camera so that it looks still at the subject mater.
The rotation part is a bit trickier. We have to decide how much and in which direction the camera should be rotated. For my case, I decided to rotate the camera, so that it is always rotated after the displacement to face the projection of the center of my segment on the forward axis of the camera. You can see the code for this in the Cubism Shader, specifically in this function:
Modifying the Projection Matrix
The above moves and rotates the camera per segment. It already gives a fractured space/ cubist feel, but I wanted to try out the idea that each segment is also shot with a different lens. For that I was interested in changing the field of view.
To do this we need to change the Projection Matrix of the camera. Projection matrices typically look like this:
Where s is the aspect ration, f and n the far and near plane and g the projection plane calculate from field of view. The value of g is calculated by:
tan (verticalFieldOfView / 2) = 1 /g
The important thing here is that to adjust the field of view we actually only need to change the 00 and 11 entries of the projection matrix and leave the rest unchanged. For simplicity I will just multiply these two entries by a constant factor which differs from segment to segment. If this factor is smaller than 1 you are increasing field view, and if it is bigger than 1 you are decreasing it. The shader CubismWithProjection does this. Specifically this function:
Even More Cubism
I am going to stop here for now, but there is much more cubism to be explored here. As you might notice this entry was focused on changing the properties of the camera, not rendering. Each fragment of the cubist painting is also shaded in a different way with different colors. You can do this by marking the segments ID in your vertex shader and either adjust texture colors / shading in your fragment shader, or if you want to do whole screen effects, outputting these IDs as an extra render texture from your pass and use it for blending/ deferred in screen space.
I hope you enjoyed reading. You can follow me on my Twitter: IRCSS
Resources and Further Reading
- Entry in Book of Shaders on creating 2D grids: https://thebookofshaders.com/09/
- Article from Inigo Quilez onVoronoi noise generation: https://www.iquilezles.org/www/articles/voronoise/voronoise.htm
- Article from Inigo Quilez on calculating lines between voronoi segments: https://www.iquilezles.org/www/articles/voronoilines/voronoilines.htm