Texture upscaling mod question

Questions specific to OpenMW-CS can be asked, and content development related topics can be discussed here
User avatar
Ravenwing
Posts: 335
Joined: 02 Jan 2016, 02:51

Texture upscaling mod question

Post by Ravenwing »

Tl;dr:
Ignoring how blurry it will be, is there anything I have to do to get an upscaled texture to display correctly? E.g. if I take a 64x64 texture and increase the image size to 256x256, will it still stretch over the mesh properly and get rid of the jagged pixels? I have no experience whatsoever with texture modding apart from downloading what other people have made, and all the guides and explanations I'm finding are about making textures from scratch and I have no patience for dealing with any 3d programs.

Backstory:
I love and adore lysol's textures, but I loathe having mismatched textures, so until he has a whole set, I'm up a creek in terms of what I use. Pure vanilla is not an option, but I do insist the textures be vanilla friendly. All these feelings eventually led me to use the watercolored texture mod. In general, I really like the aesthetic since it is literally vanilla filtered to get rid of the obvious pixelation and blurring from upscaling. One very large drawback though, is that many of the textures still show zigzags along where single pixels used to be. This gave me a new idea, that I think I will be able to pull off so let me know what you all think and any advice for different steps.

Objective:
Create a series of batch processes that will convert vanilla textures into larger textures that don't necessarily have to keep all the details, but should be "artistically" close.

Process:
First a simple batch upscale of all textures by 2 or 4 times depending on what works better in game. The problem is this often leaves things blurry and there will be weird areas where it's obvious where the pixels were. My idea is to transfer this into Illustrator and use it's vector trace to make all edges super clear and smooths everything out while still maintaining the essence of the image. From my preliminary tests, it seems like this actually would be an ok place to stop and convert back as it already somewhat resembles the watercolored mod. However, I really like the hand drawn aesthetic, so I would transfer this back to Photoshop and apply one of their stylistic filters. I might try to mimic the watercolor look, but there's an oil painting filter that has these really cool brush strokes so I'll probably see how that looks as well. Since this whole thing will be batch processed it will be easy to try a few different things like that. Then take that and resave into a DDS.

I think the most difficult part will be getting the image trace and the filter settings right so that it works on all different image sizes. I don't think I'll have the patience to come up with different settings for each different size. I had thought about upscaling everything to a single size, running all the processing and then rescaling the results down to their appropriate sizes, but I don't think batch processing is that smart. I'm an expert in Photoshop, but I've never used batch processing much, so that will be a bit of a challenge to get just right as well.

In terms of my actual questions:
  1. Do I need to treat the images in any once I've upscaled them? I don't want weird gaps or overlaps or other artifacts, but based on simply using texture replacers, this doesn't seem to be a problem.
  2. Many of the textures are mipmapped. I have a vague idea of what it's for, but is this necessary to keep? How does OpenMW utilize it?
  3. Obviously making the files larger will slow things down, but this is such an old game there should be some wiggle room. Regardless, how big is too big?
  4. What steps do I need to take to try and optimize things a little? I learned that DDS is just a container format so do any of you know any resources for deciding how to package the final files? Is there anything specific about OpenMW that I should know?
Hope you guys can help me out and are interested in the results! I'll be sure to post some results once I start testing things out properly.

PS, lysol, if you need any help with your stuff I'd be glad to help in any way I can.
User avatar
lysol
Posts: 1513
Joined: 26 Mar 2013, 01:48
Location: Sweden

Re: Texture upscaling mod question

Post by lysol »

A quick reply, sorry if I didn't read your post carefully enough:

As long as you upscale a texture while keeping the proportions (preferably by the power of two), it will stretch over the mesh properly. It won't necessarily get rid of jagged pixels though. Just simply upscaling textures doesn't really do much good, so you kind have two options here: Make a new texture, imitating the old one but in high resolution (which is what I did) or use an upscaling algorithm. The latter will never give you a perfect result for all textures, but it might work well on certain textures.

Packs using the latter technique are the watercolored pack, as you already mentioned, and also the Waifu pack. I've never used these myself, but I've read about them and seen screenshots. The waifu upscaler seems to be some kind of manga/anime upscaling algorithm, so it probably works well when upscaling faces, but probably less well on stuff like walls and ground. (Btw, this pack has mass produced normal maps. IMO, it's the same with this, just shoving a texture into a normal map generator might give you good results in some certain situations, but often it won't look that good).

So basically, my opinion on upscaling: Upscaling is a quick solution and all, but you will never get a great result, only a poor-to-sort-of-good result.
User avatar
AnyOldName3
Posts: 2667
Joined: 26 Nov 2015, 03:25

Re: Texture upscaling mod question

Post by AnyOldName3 »

Waifu 2x is a scaling filter based on a neural network. The network was trained by feeding it images, with the input being a half-sized version and the target being the original image. Effectively, the network remembers the extra detail from the target images it was trained with and can then transplant that detail into new images, which can work tremendously well when the input images are the same type as the training images, but not necessarily as well when it isn't. When it works, a scaling filter that works by transplanting detail is almost certainly always going to be better than one that doesn't.

This gave me an idea when Lysol mentioned the Waifu pack - we could generate a specialised scaling filter for games like Morrowind. I reckon that if we were to use Skyrim's official High Resolution Texture Pack and the regular textures as the training data, we should be able to get the right aesthetic. I don't know if the untrained Waifu network is easy to get hold of, but even if it wasn't, this wouldn't be the hardest thing in the world, and I reckon that someone should try it.
User avatar
wareya
Posts: 338
Joined: 09 May 2015, 13:07

Re: Texture upscaling mod question

Post by wareya »

If waifu2x causes weirdness, use nnedi3 instead. It has a simpler network (so it's less over-fit) and it was trained on mostly photographic-like data. Worst case scenario you can make your own.
User avatar
Ravenwing
Posts: 335
Joined: 02 Jan 2016, 02:51

Re: Texture upscaling mod question

Post by Ravenwing »

lysol wrote: 04 Jan 2018, 22:01 As long as you upscale a texture while keeping the proportions (preferably by the power of two), it will stretch over the mesh properly.
Great! That was my main concern. As for the rest, I completely agree. It will never be as good as a good texture made from scratch at a higher resolution, but with an artistic filter hopefully I can get rid of some of the sloppiness. One more related question. Most texture packs say 1k or 2k textures etc, so does that mean they've changed the sizes of all textures to 2k!? That seems very inefficient as an apple certainly doesn't need to be that high of resolution.
AnyOldName3 wrote: 04 Jan 2018, 22:13 This gave me an idea when Lysol mentioned the Waifu pack - we could generate a specialised scaling filter for games like Morrowind. I reckon that if we were to use Skyrim's official High Resolution Texture Pack and the regular textures as the training data, we should be able to get the right aesthetic. I don't know if the untrained Waifu network is easy to get hold of, but even if it wasn't, this wouldn't be the hardest thing in the world, and I reckon that someone should try it.
I think this is a great idea! Unfortunately it's several levels more advanced than I am capable of. Anything more complex than entering in a command with a folder to read from as an argument is more than I think I want to figure out. I did find a GUI over at waifu2x-caffe that's about my skill level lol. Based on the link lysol gave to someone who already did something similar, it doesn't look too promising, but perhaps it's a better starting point than PS's selection of interpolating algorithms. I would still want to run it through a vector trace and a rerastered filter. I'm currently in the middle of the vector trace step, but looks like it'll have to run overnight.
wareya wrote: 05 Jan 2018, 02:41 If waifu2x causes weirdness, use nnedi3 instead. It has a simpler network (so it's less over-fit) and it was trained on mostly photographic-like data. Worst case scenario you can make your own.
If I can find a GUI version of it, I'll certainly give it a try! If anyone is interested in training our own, let's start a new thread elsewhere as I think this could be a very interesting project, and maybe someone else will be better placed to accomplish it than I! I did find this so maybe waifu would be good enough after all.

Thanks all!
User avatar
lysol
Posts: 1513
Joined: 26 Mar 2013, 01:48
Location: Sweden

Re: Texture upscaling mod question

Post by lysol »

Ravenwing wrote: 05 Jan 2018, 07:21 Great! That was my main concern. As for the rest, I completely agree. It will never be as good as a good texture made from scratch at a higher resolution, but with an artistic filter hopefully I can get rid of some of the sloppiness. One more related question. Most texture packs say 1k or 2k textures etc, so does that mean they've changed the sizes of all textures to 2k!? That seems very inefficient as an apple certainly doesn't need to be that high of resolution.
I bolded that scentence because so many don't get this in the modding community. You see 4k mods of the smallest details in Skyrim for example, and this is just pure dumb.

Smart people think something like this: Most major textures in Morrowind is 256x256. These guys go up to 2048x2048 if you go for 2k, which is 8 times bigger than 256. Then you have some 64x64 textures. Go for 8x bigger again, which is 512. Etc etc. Most textures (they are also the largest ones) will then be 2048x2048, so you basically have a 2k pack now.

With this said, you can go for whatever size you want with a certain texture, but the vanilla textures give you kind of a hind of how big the texture needs to be.
User avatar
AnyOldName3
Posts: 2667
Joined: 26 Nov 2015, 03:25

Re: Texture upscaling mod question

Post by AnyOldName3 »

but the vanilla textures give you kind of a hind of how big the texture needs to be.
That's putting an awful lot of faith in the competency of Bethesda... although it's reasonable as a starting point. I'm now going to go into more detail than you're probably interested in.

The main things you should bear in mind is how much of the screen an object will fill, how effective it's UV map is at making sure the texture is used effectively, and also the Nyquist Sampling Theorem.

If an object is never going to fill a large number of pixels on your screen, it doesn't need a big texture. However, if you're rarely going to see the object take up many pixels, but sometimes you do (for example if the apple gets thrown really hard at your face in first person view in slow motion), you're going to notice the lack of resolution. Even then, it's possible that it needs a bigger texture than it fills pixels. For example, as the texture covers the whole object, if it takes up around 128x128 screen pixels, a 128x128 texture won't be adequate as some of those texels are used for the back of the object so you don't end up with a whole texel per pixel.

If the UV map of the object is really effective, though, you might find that the front and back of the object share the same region of the texture and you don't need a bigger one after all. If it's really bad, though, you might find that the front and back are both squashed into a tiny corner of the texture and the stalk of the apple takes up its vast majority. In this case, you'll need a much bigger texture than you expect. This happens a lot in Morrowind.

Nyquist's sampling theorem basically states that if you know the maximum frequency of a signal you only need to take two samples per cycle of the signal's maximum frequency to be able to reconstruct it. For example, if you had a stripy object like an especially regular zebra, you only need to have one row of pixels for each black stripe and one row for each white stripe. This particular example basically falls apart because zebra stripes are a square wave, which has infinite frequency if you represent it as a continuous signal, but if the stripes were blurred together to form a gradient from black to white to black again, it would work. The thing you should be taking away from this is that if there's not much going on in a texture, you can get away with it being lower resolution than the number of pixels it covers on screen. After all, if your apple had been expertly spray-painted a completely uniform red, it wouldn't matter how many red pixels your texture had if they were all identical.
User avatar
raevol
Posts: 3093
Joined: 07 Aug 2011, 01:12
Location: Caldera

Re: Texture upscaling mod question

Post by raevol »

AnyOldName3 wrote: 06 Jan 2018, 00:13 Nyquist's sampling theorem basically states that if you know the maximum frequency of a signal you only need to take two samples per cycle of the signal's maximum frequency to be able to reconstruct it. For example, if you had a stripy object like an especially regular zebra, you only need to have one row of pixels for each black stripe and one row for each white stripe. This particular example basically falls apart because zebra stripes are a square wave, which has infinite frequency if you represent it as a continuous signal, but if the stripes were blurred together to form a gradient from black to white to black again, it would work. The thing you should be taking away from this is that if there's not much going on in a texture, you can get away with it being lower resolution than the number of pixels it covers on screen. After all, if your apple had been expertly spray-painted a completely uniform red, it wouldn't matter how many red pixels your texture had if they were all identical.
You got any more of them... graphical rendering theorems? This is relevant to my interest.
User avatar
AnyOldName3
Posts: 2667
Joined: 26 Nov 2015, 03:25

Re: Texture upscaling mod question

Post by AnyOldName3 »

My information theory course was full of so many things that gave me wonderful insights into rendering and graphics. There was also a long article I stumbled upon about it that seemed really interesting when I read half of the first part, but I closed it and went back to doing the thing I was supposed to be doing as it had a really obvious title that I could google at a later date. I was going to link it here, but I can't find it or reliably remember the title.
User avatar
AnyOldName3
Posts: 2667
Joined: 26 Nov 2015, 03:25

Re: Texture upscaling mod question

Post by AnyOldName3 »

I just remembered that I listed it as further reading in a talk on anti-aliasing that I had to give. Here's the link: https://mynameismjp.wordpress.com/2012/ ... -graphics/

There are one or two follow up articles, too.
Post Reply