When we come to talk about baking, there’s a lot of misinformation about it. And therefore a lot of errors.
On this tutorial we are going to look at the basic techniques to have good bake and have good normal maps afterward. We will also see for the other maps.
Starting with the basics, what's a bake?
A bake is a projection of detail from one surface to another. It can be lot of different information, the normals, the height, the albedo, ... It’s just a transfer of data.
Image from Marmoset Toolbag Documentation
Why should I bake?
In a game engine, most of the time you can’t directly import high poly, wich is too heavy for the engine. So, you’re transferring the information (bake) from the high poly mesh into a low poly one that the engine can easly handle.
Also, during baking you can retrieve various information that can help you while texturing, shading, ... For example, the ambient occlusion, curvature, ...
If it’s so simple, how can I mess it up?
When transferring the information, some of it are based on some data of your meshes (UVs, vertex vormal, smoothing group, etc.), and if these aren’t good, the bake will surely go wrong.
Also, when baking information that depends +on some light calculation (ambient occlusion, lighting, ...) you need to correctly setup the parameter to have a proper map afterwards.
For this tutorial I will use Marmoset Toolbag for the baking software. It’s an opinion, but for me marmoset is the best baking tool currently on the market. (this is not sponsorised)
Before digging into advanced stuff, lets cover the basic rules: READ THE MANUAL
In fact, Marmoset redone their documentation, and it’s super clear and helpful.
Taking 15 minutes to read the whole baking documentation of Marmoset can really help you a lot understanding what you’re doing,so enhancing the quality of your bake.
There are few points in that documentation that I want to highlight for later:
Quick loader
Triangulation
Image from Marmoset Toolbag Documentation
The high Poly
First if you want to transfer information from the high poly, make sure that the high poly is clean.
For example, if you exported from Zbrush, some time with the dynamesh or zremesh, it can mess up a bit on thin area and create some small inner vertices island. After the bake, it can create some black spot artefact on the Ambiant occlusion, and/or artefact on the normal map.
To solve that, in zbrush you can make polygroup from unconnected mesh (auto group) then only show you main mesh (with ctrl + alt + lmb) and delete the hidden one.
Also, another error that you can encounter is some faceting on the normal. That might just be to a lack of polygon on the high poly, some subdivision will help you get rid of that.
Here I faked the inner vertices with those few planes that I maked visible. But it's the same workflow
The low poly
The low poly that will receive the information need also to have the right data to bake well.
Most important one are listed here:
UVs
Smoothing Group
Normals
UVs:
Good UVs are important for a lot of things, optimization of the texture space, better texel density for the same texture, get rid of aliasing artefacts, better mipping in engine, easier to paint on, ...
So, if you messed your UVs you can have some artefacts on the baking.
On of this most common one, Is the weird pixelization artefact that appear on the edges of your uvs island (seams). This might appear due to aliasing on non-straight island, and this effect will be even more accentuate in engine, with mipmapping
You can find just here a great UV tutorial that my friend Eli just made.
It will show you how to get rid of that will straightening your island.
Smoothing group and Vertex normals:
By smoothing group what I mean is Sharp edges (for Blender user).
There's a general rule about sharp/hard edges, they should only be on seams (UV cut). However, not all seams need to have sharp/hard edges.
Know that in fact when you add a seam, for the engine is like you duplicate the vertices on the same position.
And when you set a sharp edge, it’s dividing the vertex normal in 2, by the same process. Duplicating the Vertice at the same position. So if you need to split for normal purpose, do it also on the texture to avoid artefact, so with a seam.
So, if you need to handle a change of normal that is above 90°, most of the time if you don’t set up a sharp/hard edge you will have too much gradient on your Tangent Normal Map. (Personally, most of the time I question myself when it's above 45°, your limit can be less or more)
So, when you have gradient on your tangent normal map it’s cause it try to get the changing of orientation of your face baked on something that too much smooth. And that’s where the gradient appears, to get back that original angle.
As you can see, with or without the hard edges, we can't really see any differences in World Normal on the baking soft.
But we can definitely see it on the tangental normal, all the gradient here is just to compensate for the vertex normal.
What the problem with gradient on the normals ?
To be quick, the shading. A bit same as the UVs problem, you’re shading can work well with a high-res texture. But while put in engine it will be compressed and mipped.
And so, if you pixelate (while mipping) a gradient, you will get Aliasing / banding artefact. And that’s something we don’t want.
We want our object to look okay, even when it mips. Also, the gradient can be too strong, and even with a high-res map, the shading will look bad in the current or others 3d software.
Here's a side by side comparaison with forced mipmapping, the left one without the sharp edges.
How can we avoid them?
As we saw just above, splitting the vertex normal of the edges of strong angle can help a lot reduce them, but even with that some time you will face some weird issue.
Most of the time it came from your original vertex normal of your low poly.
Make sure you’re vertex normal are well oriented and there are based on your faces. You can apply an weighted normal while maintaning your sharp edges. That will tweak your vertex normal to fit a bit more the tangent of the faces. It can really help reduce those gradient.
Also there’s the miss triangulation artefact. Know that an engine only calculate with triangles. So if you have a quads for example, it will be calculated as 2 triangles.
The fact is that every 3d software have their own algoritm of triangulation, so you can have a difference of triangulation between, your modeling software, your baking soft, and your final engine.
The point is that triangulation will affect your vertex normal, and if you have a different triangulation from the one you’ve bake on, you will have some gradient and artefact on the shading.
To avoid that, you should triangulate before baking, and using this low poly triangulate throught all the 3d software afterward. This will help you maintaning the right triangulation.
Image from Marmoset Toolbag Documentation
Autopsy of bad baking
As you can see here, the first normal map is really messed up.
This is based on few differents problems. My vertex normal aren't well oriented based on the lowpoly face. The smoothing group might not be correctly setup. They might be not enough polygon to handle some rotation of vertex normal.
One this one I make sure to really accentuare those artefact but it gave you an idea of what you don't want. A great example is those green gradient triangular shape in the normal of the blade that show miss setup of the smoothing group (hard/sharp edges).
Here to faked the messed low poly I decimate my current one, that's why there are some weird triangles and uv deformation. But it showcase correctly the artefact of gradient on the normal map.
Most of the time if your lowpoly look weird in term of normal in your 3d software, it will end up weird also the normal map too, du to the fact the normal map (tangeant normal map) is based on the vertex normal.
Marmoset Setup
While baking, you have the option to create bake group. That will avoid each group to bake on others. So it’s really helpfull to avoid mesh that bleed on each other on baking.
But it can be long to setup. That’s where the quick loader from marmoset is awesome, it allow you with the right naming convention to automatically import and setup the group for you. Trust me while itering over assets, this is a real time saver.
A great tip is also to keep your lowpoly and highpoly files seperate, while iterating on your lowpoly it will allow marmoset to only reload your lowpoly files, and so much less time of import.
Image from Marmoset Toolbag Documentation
Compute maps
While baking you want to retrieve other information that just the high poly normals.
Most of the time you will want Ambient Occlusion, curvature, Thickness, etc.
Here's few tips to enhance the quality of those maps.
Ambient Occlusion
For the ambient occlusion, note that from default the ray count is quite low, so don’t hesitate to increase it. Personally I use, for most of my project, 1024 ray count at least, It will decrease a lot the dithering that you can find on your Ao maps. Also while baking ambient occlusion you can have some cliping artefacts, that’s cause due to the difference of shape between your low poly and high poly.
To get rid of this artefact you can bake your low poly with the information of normal map to it self, and so you will retrieve the ambiant occlusion with the detail of the high poly one the mesh silhouette of your low poly. That’s magic !
Also good habits is too bake you 2 different map of ambiant occlusion, one classic, and one with floor occlusion. Depending of the assets it can be really helpfull to have both of them while texturing.
Height map
On tileable, some time you want to bake an height map to reuse it in trexturing or in shaders. But you need to tweek the settings of your height bake.
To ensure to have the best quality on your maps, you should adjust the value of the lowest and highest point. The Target is too really use the whole range of value of your maps. So the highest pour should be near to white (1) and deepest one neer to black (0).
Thickness map
Asjusting your thickness parametter can be a must for somme assets. So same as whith the heigt map try to adjust the slider to really get a nice range of value out of your bake
EXPORTED FORMAT
When you’ve done all of those previous work, you don’t want it to be destroyed by a huge compression on your files afterward. That’s why it’s important to export it well.
At first but not last, don't use png. Even it’s better than jpg, you still have compression on it, and you can lose some subtle information, especially on data maps.
The targa are great as an output, because they are 100% loss less. Also due to the fact that at the end they all have the same size on the disk based on their format. It’s easier for you to check if you have all the maps, you need just by looking at the size of your folder. But know that the targa can only store up to 8bit texture when using marmoset as a baker.
Small point about pixel depth
To resume, image are store with a certain amount of memory allowed per image. For an 8bit image the repartition of this 8bit are done like that : 8 bits RGB = R 3 bits; G 3 bits; B 2 bits. Here's a small table of some common image depth.
Image Depth | Distribution per Channels | Colors Range |
8 bits (RGB) | R3 G3 B2 | 256 |
16 bits (RGBA) | R3 G3 B2 A8 | 256 + Alpha |
16 bits (RGB) | R5 G6 B5 | 65 536 |
24 bits (RGBA) | R5 G6 B5 A8 | 65 536 + Alpha |
24 bits (RGB) | R8 G8 B8 | 16 777 216 |
32 bits (RGB) | R8 G8 B8 A8 | 16 777 216 + Alpha |
More information here
When you want an high value range on a texture, for example the height map (most of the time an 8bit texture for an height map is not enough if you want to rework it, and will result with banding artefact). You should store it as a PSD to make sure that you can output the best value range out of Marmoset with a an higher pixel depth than 8 bits/channels.
Check List
Here a little check list if you have some error on your bake:
High poly
enough polygon ?
inner faces ?
flipped normal ?
Low Poly
UVs
Straight ?
Well pack ?
Enough margin ?
No Overlapp ?
Smoothing groups
Hard/ sharp edges on +45° angle?
Placed on seam ?
Vertex Normals
Shading in 3d modeling soft okay ?
weighted normals ?
Smoothed ?
Triangulate ?
Compute maps
enough res on the ambiant occlusion ?
thickness and height adjusted if needed ?
Export format
enought definition for my texture size target ?
Right file type ?
Lossless ?
enough byte per channel for my data ?
The dagger used in the example was made based on a great concept of Baldi Konjin.
You can find it here.
Disclaimer
This tutorial is based on my knowledge and experience. Some tips or methods may not be the best option depending on what you are doing. This was made in the prism of an Environment Artist, so for "static" object.
Thanks
Thanks to all the peoples who help me throught my 3D journey, and help me learned those knowledges. Especially the Artside team, with Camille Delmeule, Sacha veyrier and Matthieu Laude.