osgFX::Cartoon

General discussion regarding the OpenMW project.
For technical support, please use the Support subforum.
SatoriLotus
Posts: 27
Joined: 22 Jun 2017, 16:03

osgFX::Cartoon

Post by SatoriLotus »

Hello,

Looking ahead, some devs using OpenMW may want to go with a cel-shaded look for their games, as it's becoming more popular these days. I have practically no knowledge of C++, which is why I'm here to ask; how difficult would it be to implement OSG's Cartoon/Outline effects in OMW?

Thanks in advance.
HeckinDogelet
Posts: 1
Joined: 26 May 2019, 05:11

Re: osgFX::Cartoon

Post by HeckinDogelet »

Definitely gonna second this. Using the Cartoon.fx in Reshade can *somewhat* replicate this, and the old Outline.fx (if you don't mind using an older Reshade or messing with code) works pretty well, but Cartoon.fx destroys text readability and Outline.fx isn't that good overall, but it works.

I'd love a proper OpenMW Shader that does this. I'm not good with coding so I don't know the feasibility of this, but it's an exciting thought. Cel shading and stylized graphics could be a performance friendly way to make Morrowind look great while maintaining performance.
Xenuria
Posts: 45
Joined: 26 Feb 2017, 22:35

Re: osgFX::Cartoon

Post by Xenuria »

You don't mean like Team Fortress 2 and fortnite do you? That would be disastrous to the art style of this game.
User avatar
lysol
Posts: 1513
Joined: 26 Mar 2013, 01:48
Location: Sweden

Re: osgFX::Cartoon

Post by lysol »

Having in-engine support for it would be really cool in my opinion, even though a cel-shaded Morrowind sounds, uh, strange. But who knows, maybe it would look great?
SatoriLotus
Posts: 27
Joined: 22 Jun 2017, 16:03

Re: osgFX::Cartoon

Post by SatoriLotus »

Xenuria wrote: 26 May 2019, 19:18 You don't mean like Team Fortress 2 and fortnite do you? That would be disastrous to the art style of this game.
I'm not really a fan, but technically yes. I actually had games like Breath of the Wild in mind.
lysol wrote: 26 May 2019, 20:43 a cel-shaded Morrowind sounds, uh, strange
I wasn't really thinking about it working with Morrowind assets, even if someone re-textured everything with cel shading it mind it might still look weird. To make MW itself look good in cel, you'd need to do a complete graphical overhaul, which would be quite the undertaking.

I was looking to the future, when (hopefully) people start making their own games with original assets. An artistic cel-shaded game with Morrowind's crpg-esque mechanics sounds pretty cool to me.
User avatar
Stomy
Posts: 47
Joined: 11 Dec 2018, 02:55
Location: Fhloston Paradise!
Contact:

Re: osgFX::Cartoon

Post by Stomy »

SatoriLotus wrote: 27 May 2019, 03:03 I was looking to the future, when (hopefully) people start making their own games with original assets. An artistic cel-shaded game with Morrowind's crpg-esque mechanics sounds pretty cool to me.
I'm something of an anti-future-proofing elitist so I'd hold on this until someone has actually started such a project (actually started, with stuff to show, not "planning stages")
thephantomfiddler
Posts: 32
Joined: 06 Mar 2019, 01:03

Re: osgFX::Cartoon

Post by thephantomfiddler »

There are barely any examples of OSG cel-shading, and plus it's outdated by >10 years judging from git commits.
SatoriLotus
Posts: 27
Joined: 22 Jun 2017, 16:03

Re: osgFX::Cartoon

Post by SatoriLotus »

Stomy wrote: 27 May 2019, 07:32 I'd hold on this until someone has actually started such a project
That's fair honestly, without a real reason to do it beyond hypothetical usage, it's just taking a coder's time away from more important aspects of OpenMW.

Really though I was just asking for an estimation of how complicated implementing it might be, rather than asking someone to actually do it.

thephantomfiddler wrote: 27 May 2019, 08:23 it's outdated by >10 years
I wasn't aware, that's disappointing.
User avatar
AnyOldName3
Posts: 2667
Joined: 26 Nov 2015, 03:25

Re: osgFX::Cartoon

Post by AnyOldName3 »

The modern way of doing the outlines, for example, would be to have a post-process shader that just set a pixel to black if it was near a depth discontinuity. I've not looked at how the lighting gets meddled with by this, but I'd be somewhat surprised if it couldn't be done just by using a custom lighting shader and sensible textures, and if you're making a whole new game in OpenMW, you're probably capable of finding someone who can play with the shaders.
Clement
Posts: 9
Joined: 03 Jul 2018, 10:41

Re: osgFX::Cartoon

Post by Clement »

Lighting test just for fun:

https://framapic.org/bXgmzWCgS6ll/nCnHXDZJvv8w.png
https://framapic.org/PSHWA3uoIhw7/YpXaBMRCzSpb.png

Animated light are a bit annoying with this kind of lighting. And faces have too many angles creating shadows, it looks betters on smoother models.

Code: Select all

--- /home/clement/src/openmw/files/shaders/lighting.glsl	2019-05-28 11:34:15.068488979 +0200
+++ /home/clement/opt/openmw-git/share/games/openmw/resources/shaders/lighting.glsl	2019-06-01 11:40:47.960994690 +0200
@@ -2,7 +2,21 @@
 
 uniform int colorMode;
 
-void perLight(out vec3 ambientOut, out vec3 diffuseOut, int lightIndex, vec3 viewPos, vec3 viewNormal, vec4 diffuse, vec3 ambient)
+float celShade(float l)
+{
+    return step(0.1, l) * 0.3 + step(0.5, l) * 0.4 + step(0.9, l) * 0.3;
+}
+
+vec3 celShade3(vec3 color)
+{
+    float l = 0.2126*color.r + 0.7152*color.g + 0.0722*color.b;
+    if (l == 0.0)
+        return color;
+    else
+        return color * (celShade(l) / l);    
+}
+
+void perLight(out vec3 ambientOut, out vec3 diffuseOut, int lightIndex, vec3 viewPos, vec3 viewNormal)
 {
     vec3 lightDir;
     float lightDistance;
@@ -12,8 +26,8 @@
     lightDir = normalize(lightDir);
     float illumination = clamp(1.0 / (gl_LightSource[lightIndex].constantAttenuation + gl_LightSource[lightIndex].linearAttenuation * lightDistance + gl_LightSource[lightIndex].quadraticAttenuation * lightDistance * lightDistance), 0.0, 1.0);
 
-    ambientOut = ambient * gl_LightSource[lightIndex].ambient.xyz * illumination;
-    diffuseOut = diffuse.xyz * gl_LightSource[lightIndex].diffuse.xyz * max(dot(viewNormal.xyz, lightDir), 0.0) * illumination;
+    ambientOut = gl_LightSource[lightIndex].ambient.xyz * illumination;
+    diffuseOut = gl_LightSource[lightIndex].diffuse.xyz * max(dot(viewNormal.xyz, lightDir), 0.0) * illumination;
 }
 
 #if PER_PIXEL_LIGHTING
@@ -42,19 +56,22 @@
     vec4 lightResult = vec4(0.0, 0.0, 0.0, diffuse.a);
 
     vec3 diffuseLight, ambientLight;
-    perLight(ambientLight, diffuseLight, 0, viewPos, viewNormal, diffuse, ambient);
+    vec3 diffuseSum = vec3(0.0), ambientSum = vec3(0.0);
+    perLight(ambientLight, diffuseLight, 0, viewPos, viewNormal);
 #if PER_PIXEL_LIGHTING
-    lightResult.xyz += diffuseLight * shadowing - diffuseLight; // This light gets added a second time in the loop to fix Mesa users' slowdown, so we need to negate its contribution here.
+    diffuseSum += diffuseLight * shadowing - diffuseLight; // This light gets added a second time in the loop to fix Mesa users' slowdown, so we need to negate its contribution here.
 #else
-    shadowDiffuse = diffuseLight;
-    lightResult.xyz -= shadowDiffuse; // This light gets added a second time in the loop to fix Mesa users' slowdown, so we need to negate its contribution here.
+    shadowDiffuse = diffuse.xyz * diffuseLight;
+    diffuseSum -= diffuseLight; // This light gets added a second time in the loop to fix Mesa users' slowdown, so we need to negate its contribution here.
 #endif
     for (int i=0; i<MAX_LIGHTS; ++i)
     {
-        perLight(ambientLight, diffuseLight, i, viewPos, viewNormal, diffuse, ambient);
-        lightResult.xyz += ambientLight + diffuseLight;
+        perLight(ambientLight, diffuseLight, i, viewPos, viewNormal);
+        ambientSum += ambientLight;
+        diffuseSum += diffuseLight;
     }
 
+    lightResult.xyz += ambient * ambientSum + diffuse.xyz * celShade3(diffuseSum);
     lightResult.xyz += gl_LightModel.ambient.xyz * ambient;
 
     if (colorMode == 1)
Outlines would require post-processing shaders, as AnyOldName3 said, so they are not available now. I did test that with MGE some time ago, my simple post-processing had the inconvenient of drawing lines on top of particle effects though. Maybe good graphic programmers know how to solve that.
Post Reply