Postprocessing: For real

[wazabear]

Since there's been a Lua roadmap I've decided to outline my plans for postprocessing as well. Even though it's far less of an architectural feat, it is important to outline my vision for a programmable post-process pipeline as there are some significant features and I don't want any surprises. I think a reasonable goal for this is 0.49.

Some terminology, I'll refer to a "pass" as a single filter on an image (like blur), and a "technique" as combination of passes (like blur + AO). Every technique has some rendering state like render targets, bound texture samplers, custom uniforms, etc.

The blockers for postprocessing are only these two issues:

• https://gitlab.com/OpenMW/openmw/-/issues/6078 : associated MR !889, needs final review and merge
• https://gitlab.com/OpenMW/openmw/-/issues/6199 : finished, will open MR once 6078 is resolved

Here are a few important points I'd like to make clear.

• Auto-hotreload and overlay. When writing shaders hotreload is an absolute necessity, so is being able to automatically reload the postprocess chain when a file is changed. There are some lightweight cross-platform file watching libraries out there, but initially I think a polling of last modification time will be more then sufficient. The numbers of shader files to watch is directly proportional to the number of techniques in use, and even a high number for this will likely never exceed 10. Adding such functionality to the VFS should not be too intrusive. The overlay I'd like to implement will include per-technique settings and debug options. In my testing branch I put together a 5-second working POC which looks something like this . Of course the final product will look professional and polished, certainly more flexible, this is just to give everyone a concrete image of what I'm talking about.
• Compute shader path. The traditional method for postprocessing includes binding vertex and fragment shaders and drawing to a fullscreen geometry. On more modern cards, we can actually utilize compute shaders which can be made extremely efficient for image filtering. Compute shaders barely resemble pixel shaders, and creating an abstraction between them would be a silly endeavor. I plan on allowing for both use of traditional pixel shaders and compute shaders, the latter having a much higher bar for entry.
• Technique files. All shaders utilize the VFS, technique files will be suffixed with "*.fx" and must be placed in a directory named "shaders".
  Technique files themselves will be slightly modified GLSL to allow multiple passes and definition of metadata. I've not decided on best way to approach this part yet, so I'll leave it for a later post. My initial attempts I've thrown out as they were pretty clunky and utilized the @defines used in the standard forward passes. One thing I know for sure, XML is not going to happen...ever. To define a shader chain it simply will be written in the settings.cfg as a list of technique files. Adding an in-game configuration will come naturally as I'm designing things with this kind of adaptability in mind.

  Code: Select all

  [Postprocessing]
  enabled = true
  chain = hdao, dof, godrays
  

I shared these a while ago on discord, but just to clarify this is a real thing and not a 2090 thing here are some samples of some over-the-top AO using my testing branch.

https://imgsli.com/NTMxNDY
https://imgsli.com/NTMxNDc
https://imgsli.com/NTMxNDg

[AnyOldName3]

Hey, it's The Witcher 2!

Everything here looks fairly sensible except giving the technique files the .fx file extension. DirectX effect files use that extension, and it's going to make syntax highlighting more of a pain if editors think they're dealing with HLSL with extras instead of GLSL with extras.

[psi29a]

So you're taking a page from Scrawl's playbook with shiny?

[urm]

but initially I think a polling of last modification time will be more then sufficient.

For Lua scripts we just have a "reloadlua" console command, would something similar not suffice for post processing?

[Br0ken]

chain = hdao, dof, godrays

It would be nice to see GTAO too. It's most accurate and modern AO technique, used in Blender's Eevee and bunch of AAA games.

[AnyOldName3]

GTAO is a stupid name as it's an approximation of Monte-Carlo ground truth, not Monte-Carlo ground truth. Plain SSAO that everybody rags on for looking bad was also an approximation of Monte-Carlo ground truth, and was groundbreaking and state-of-the-art when it was new. In a n years when either a better approximation is available or raytraced implementations are available, people are going to complain that ground-truth ambient occlusion looks unrealistic when really they mean that this particular algorithm looks unrealistic. Ground truth is real life, or the best theoretical model we've got for real life, so by definition can't be unrealistic.

By the way, I remember GTAO needs stuff like thickness information. How ludicrous is it going to be to implement things like that with this new format in a theoretical future when the renderer's actually capable of generating the information in the first place?

[wazabear]

For Lua scripts we just have a "reloadlua" console command, would something similar not suffice for post processing?

It would suffice, auto hotreload is just a huge QOL feature. Reshade has a live editor which is a godsend, I see no reason not to include the feature considering how much it improves the overall experience. This would just be a checkbox in the overlay, it's not something that is going to be always enabled.

By the way, I remember GTAO needs stuff like thickness information. How ludicrous is it going to be to implement things like that with this new format in a theoretical future when the renderer's actually capable of generating the information in the first place?

Well the plan was to add a special path for AO that plugged into the objects/terrain shaders, essentially having an AO sampler available which you can then apply to lighting/ambient directly. This requires some pre-pass to happen of the opaque bin. Normally you wouldn't upload information like materials when drawing this pre-pass, but there would be nothing stopping us from sending any information we want. If we're talking about AO as a postprocess this likely will never be possible with the system I have in mind.

[vtastek]

I know nothing about compute shaders, they would be nice to play with. Very excited.

I think LSAO can only be done via a compute shader, seen in Remedy's games. It is very fast.

It would be nice to know the cost of a shader(or passes) in ms.

[Br0ken]

https://developer.nvidia.com/sites/defa ... mework.pdf
https://github.com/tlorach/nvFX
Can it be useful?

[wazabear]

The nvFX syntax for technique is pretty much exactly what I implemented in my first iteration, ie blocks of code. I'm sticking with this approach for same reason pointed out in the powerpoint, there is no need to parse the GLSL (or whatever language the block contains). Just throw the block to the compiler, this greatly simplifies things. The only difference between nvidias approach and mine was naming conventions for blocks.

This isn't really a surprise, there are only so many ways you can implement single file-multipass shaders without a shader language parser.

Of course the scope of my postprocess pipeline is not so encompassing as nvFX, but that's fine. This is supposed to be another tool for modders, so I'm finding the best balance here between simplicity and scope. Controlling shaders through a Lua interface, for example, is incredibly powerful. Per-weather effects or a skooma trip. Those are the obvious effects, but we've seen really awesome ones with MGE-XE like this https://www.nexusmods.com/morrowind/mods/47565. Point is, I'm keeping this primarily a postprocessing pipeline (with some exceptions), not a general purpose FX pipeline.