We can differentiate two fundamentally different techniques:
- Per-Vertex-AO: In the Content-Tool every vertex of the mesh has a value associated that describes how brightly lit it is by the ambient light.
- Texture-AO: The level of brightness is "drawn" as shades of grey in a texture. This can be done either by hand or using a suitable graphics program. The texture is then used in the Content-Tool.
Both methods have certain advantages, disadvantages and peculiarities:
The first method is particularly well suited for filigree or small objects with pronounced shapes, like switches, buttons, handles, etc. and / or self-contained surfaces, such as cylinders and balls, like handrails. These usually have a high vertex density whereas a clean mapping of a texture is either difficult or impossible. The texture resolutions that would be required to achieve proper AO would have very high storage demands and requirements. The Per-Vertex-AO uses the secondary texture coordinates to store the brightness information. Because of this it is not possible to combine mesh-animations and Per-Vertex-AO in one mesh.
The second method is more suited for rather flat and rough or angular objects, which consist mainly of relatively flat surfaces meeting other surfaces at an angle. Examples of this are the interior and exterior walls, floor and roof. Here mapping a texture is mostly without problems. However, the vertex density is so low that the transitions between the vertices strongly distort the impression and, some details may even be completely ignored. Balancing this only works with a texture that has the necessary resolution.
For Texture-AO there are again two approaches:
- Using the "normal" texture coordinates
- Using the alternative/secondary texture coordinates
Here again the choice depends on what we want to achieve:
If the different areas on a texture are only used once in a mesh (like with large and individual photo textures for old buildings or instruments) or if the parts of the mesh containing this area are generally lit in a very similar way (like support beams or walls in a wagon hall) then you can use the "normal" texture coordinates. In that case the technique is simpler to use: The AO-texture is a grayscale texture (but in 24 bits file format) that shows the ambient brightness at the same location of the normal texture. It is also the only possibility to have AO-information on an object with a mesh animation.
If a texture is used repeatedly or in many different parts of the mesh that are also lit differently (like a floor being mapped with a tiled texture that is repeated) the second variant is better. In this method the secondary texture coordinates are used. Because of this you can use a completely different texture mapping. With this new mapping you can make sure that each area of the texture is only mapped to one part of the mesh. A texture with a low resolution can be used where this is possible, like for the previously mentioned floor. An automated and optimized texture resolution for AO is for example available in Blender. Because, like with Per-Vertex-AO, the secondary texture coordinates are used and so this method cannot be combined with mesh animations
In both cases the AO-texture can be created manually or using a suitable 3D program, like Blender. Next, like any other texture, it is copied into the working directory and can be selected in the Material properties.
2 How to do it?
For this select the mesh in the Content-Tool and in the section Selected Mesh click on "Generate Per-Vertex-AO". Cancelling can be done using the "x" button next to the progress bar.
Calculating AO with a high quality is comparatively time consuming because of the quality needed. Using a lower quality would result in an unrealistic effect and is really only suitable for tests. Before release you can then take the time to do the high quality calculation.
If the selected material has the type "Standard", it will be changed to "Complex". Important: Because the Content-Tool (automatically) updates the material for the Per-Vertex-AO, all other meshes that use this material can have serious drawing errors. Always apply Per-Vertex-AO on either all or none meshes using a material.
2.2 Texture-AO using normal coordinates
Setting up a material to use a AO-texture using normal coordinates is very simple: After the texture is created or generated it is stored in the working directory. The material type is then set to at least "Complex". Then "Intensity of AO texture" is set to 1 and both "Activate AO-texture" and "AO uses original mapping" are turned on. Lastly the texture is selected for "Ambient Occlusion (AO) Texture".
2.3 Texture-AO using secondary texture coordinates
To create the secondary texture coordinates a copy of the object is made in the 3D program. This is named as "@2m_" followed by the name of the original object. So if the object is name "House" the copy is named "@2m_House".
The copy has to be included in EVERY export of the original object and if the original object is changed a new copy must be created as before. In principle very small deviations in the vertex data (position, normals etc) is ok, but it is very highly recommended to create a new copy.
When we import the original object and the @2m_ copy in the Content-Tool the texture coordinates of the copy are added to the original object as secondary texture coordinates. After importing only the original object should show up, the @2m_ object should not appear in the list on the right!
The remainder of the setup is like with the "Texture-AO using normal coordinates", however "AO uses original mapping" is not set.
The value for "Intensity of AO" can be used in all three cases to "reduce" the AO effect. "Reduce" in this cases brightens the object overall.