Custom Celshader Shading Model

Here, I'll be talking about how I managed to get a custom shading model working in Unreal 5.3, by going into the source code and making some changes to .cpp and .ush/.usf files related to the deferred rendering pipeline and Unreal's GBuffers. It was a lot of trial and error, and even after having successfully done it, I still don't feel like I fully knew what I was doing. Nonetheless, the results speak for themselves, and give me a fantastic stylized look that can't be achieved using Post Processing methods.

Why Make A Custom Shading Model?

Under normal circumstances, Unreal gives great tools to create great shaders with a variety of purposes and properties. Their already provided shading models like Unlit, Default Lit, etc. are great for normal PBR purposes and most often than not, a game designer will hardly need much more functionality that is not already provided.

However, there is one small exception: NPR, or Non-Photorealistic Rendered, content.

I noticed this weakness pretty quickly the moment I tried to make a stylized game and realized just how poor support was for achieving a look that is extremely stylized. The main option was using Post Process solutions, but these did not satisfy my needs. 

Desperation

I previously tried to achieve my stylized look through Post Process materials; however, I was running into some major issues. Namely, while objects looked good in a vacuum, extending entire environments and complex shapes to the Post Process solution would reveal problems. 

1. Because of the way lighting information is roughly grabbed by grey scaling the diffuse color channel and dividing it, you lose out on access to information like lighting color and the true data of the lighting calculations.

This leads to things like colored lights not being properly represented and blending well, and noisy edges.

2. Moreover, you have no control over how things look within shadow. If you want a specific color for shadows cast on your object, or you want to be able to change the intensity of a shadow on a per-object basis? Tough cookies, Post Process solutions will not be able to get you there fully. At least not without compromising your sharpness.

This leaves us with only one option, one that I was reluctant to try due to my weakness in a text editor. We would have to go and add a custom Shading Model, which would be able to be customized and take the lighting information and change it to suit our stylized needs.

This would require tons of patience, as I would need to build and compile the engine from source code. And also, my minimal C++ and HLSL knowledge would be put through a trial by fire.

Making A Shading Model is Daunting

A custom shading model isn't a very straightforward process in Unreal, as the engine developers don't seem keen on giving the average user access to the custom GBuffers, which materials use to add additional functionality from the material graph. This is often used for purposes where additional data is needed for a shading model, such as the subsurface or cloth models. I had to follow a couple of tutorials which were outdated, and scour the darkest corners of the Unreal Engine's forums to find the proper information I needed to enable this for my own Shading Model.

Here were the resources which were most helpful :

https://medium.com/@solaslin/learning-unreal-engine-4-implement-cel-shading-w-outline-using-custom-shading-model-in-ue4-22-2-308bcbd7950c

https://dev.epicgames.com/community/learning/tutorials/2R5x/unreal-engine-new-shading-models-and-changing-the-gbuffer

On top of this, needing to build the engine from source is extremely space heavy (The Source Build of the Engine is 180GB), and introduces the issue of having a small error break the engine entirely and become a complete headache to find out where you went wrong.

The majority of my time was spent figuring out all of Unreal's various functions and macros which help add your shading model to the editor, and allow it to feed data to the custom GBuffer through a material graph input (which isn't necessary, but kind of the whole point of making a Shading Model in the first place). 

By the end of it all, I was left with a Cel-Shading Model which supported the ability to change the amount of cel-values ( or bands), and which supported colored lights unlike the Post Process solutions. Moreover, the shading model also support Cel-Shaded shadows, and eventually I plan on adding the ability to choose the color of what things look like in shadow.

Maybe someday I'll release a video tutorial on this subject, since there is a lack thereof. I'm sure plenty of other people have a hard time compromising between their game's style and dealing with base Unreal's toolkit.