Cell preloading

General discussion regarding the OpenMW project.
For technical support, please use the Support subforum.
User avatar
scrawl
Posts: 2152
Joined: 18 Feb 2012, 11:51

Re: Cell preloading

Post by scrawl » 10 Feb 2016, 23:44

OK... I narrowed it down. The problem exists between these two commits:
That is because since that commit we're using the dds_flip osg option that contains the bug described in my last post.

User avatar
scrawl
Posts: 2152
Joined: 18 Feb 2012, 11:51

Re: Cell preloading

Post by scrawl » 11 Feb 2016, 00:12

As for why we're now using the dds_flip option, that is a long story.

OpenGL and DirectX use different conventions for where the "origin" (0,0) coordinate of a texture is. In OpenGL, it's in the lower left, in DirectX, top left. Since Morrowind used DirectX and we use OpenGL, we have to account for this difference. Previously (in OpenMW 0.38) we did that by flipping textures after loading them. The problem with that is, only Morrowind assets need the flipping, native OSG format assets don't. Which makes sharing and managing these textures difficult. This is why in the resource refactoring branch I changed the way this conversion is done: leave the textures alone and instead flip the texture coordinate of NIF files on load. Now you may be wondering, if we're no longer flipping textures, then why is the dds_flip option used now? Well, the OSG DDS reader (unlike TGA and others) apparently loads textures flipped to begin with, the dds_flip option just unflips them. That's all there is to that.

Chris
Posts: 1579
Joined: 04 Sep 2011, 08:33

Re: Cell preloading

Post by Chris » 11 Feb 2016, 02:25

scrawl wrote:As for why we're now using the dds_flip option, that is a long story.

OpenGL and DirectX use different conventions for where the "origin" (0,0) coordinate of a texture is. In OpenGL, it's in the lower left, in DirectX, top left. Since Morrowind used DirectX and we use OpenGL, we have to account for this difference.
Actually, you don't really. D3D considers 0,0 to be the top left and GL the bottom left, true, but the start of the image data given to glTexImage* is considered to be the bottom left and loads "upward", while D3D considers the start of the image data to be the top left and loads "downward". So while GL treats it flipped, it also effectively loads it flipped too (thus ultimately cancels itself out). This coming from years of experience with D3D->GL wrappers, and not once having to manually flip incoming texture data or mess around with vertex data to get UV coords right. There might be some gotchas when it comes to render-to-texture, but that's a different beast.

The problem really comes down to authoring. When you make images in a texture editor, if you consider the bottom left of the image to be 0,0 like OpenGL does, then it will come out wrong because the top left is the start of the image data that GL takes as 0,0. But if you consider 0,0 to be the top left in the image editor, then OpenGL will automagically handle it so 0,0 remains the same. This is why you can load a block-based format like DXT unaltered, and 0,0->1,1 covers the texture the same in both D3D and GL.

From what I can tell, OSG implicitly tries to flips textures when loading, so the bottom left you see in an image editor maps to 0,0 in GL. That's why we need to flip again, to get it back how it was. Ideally we'd just tell OSG to stop messing around with the texture like that, but when we can't, we need to flip it back around to be right side up again.

User avatar
psi29a
Posts: 4857
Joined: 29 Sep 2011, 10:13
Location: Belgium
Gitlab profile: https://gitlab.com/psi29a/
Contact:

Re: Cell preloading

Post by psi29a » 11 Feb 2016, 09:33

The DDS files are uncompressed. Might this be the reason the bug is triggered?

I don't think that rotating the images are going to solve this particular issue. ;)

Is there any way to work around this?

User avatar
scrawl
Posts: 2152
Joined: 18 Feb 2012, 11:51

Re: Cell preloading

Post by scrawl » 11 Feb 2016, 16:16

Is there any way to work around this?
Upgrade to OSG 3.4.

I think it may be about time to drop support for 3.2 anyway, less code to maintain, easier troubleshooting, this DDS crash fix is just the last straw.

User avatar
psi29a
Posts: 4857
Joined: 29 Sep 2011, 10:13
Location: Belgium
Gitlab profile: https://gitlab.com/psi29a/
Contact:

Re: Cell preloading

Post by psi29a » 11 Feb 2016, 16:30

Shit... more packages that I need to maintain for Ubuntu releases? :'(

Even 16.04 (Xenial, not yet released, due April).. the next big LTS is sticking with 3.2.1

User avatar
scrawl
Posts: 2152
Joined: 18 Feb 2012, 11:51

Re: Cell preloading

Post by scrawl » 11 Feb 2016, 16:44

OSG packaging needs some more love.

The newest in the 3.2 series would be 3.2.2, released on 08/04/2015.

Speaking for Ubuntu 14.04, they apparently ship 3.2.0-rc1, which is 3 years old and not even a stable release (release candidate!?).

I don't understand why they wouldn't at least ship the stable release 3.2.0, which was released 24. July 2013, way before Ubuntu 14.04 had its feature freeze. Right?

Plus, since the z in x.y.z are compatible bugfix releases, why wouldn't Ubuntu supply these as updates later on?

Robert puts a lot of effort in cherry-picking fixes to the older release branches, it's a shame that distributions aren't taking proper advantage of that.

User avatar
psi29a
Posts: 4857
Joined: 29 Sep 2011, 10:13
Location: Belgium
Gitlab profile: https://gitlab.com/psi29a/
Contact:

Re: Cell preloading

Post by psi29a » 11 Feb 2016, 17:55

Complain to debian, since ubuntu follows them mostly.

https://packages.debian.org/search?keyw ... egraph-dev

User avatar
scrawl
Posts: 2152
Joined: 18 Feb 2012, 11:51

Re: Cell preloading

Post by scrawl » 11 Feb 2016, 21:22

Finished bisect for the commit that fixed the dds flip crash:

https://github.com/openscenegraph/osg/c ... 2f58379b84

From Bradley Baker Searles, "We had someone replace a DDS texture with a GL_RGB/GL_UNSIGNED_SHORT_5_6_6 image, which would trigger a crash in Image.cpp while flipping the mipmap chain. The code was trying to flip the mipmaps using the rowStep for the full-size image (we have "dds_flip" set in the osgDB::ReaderWriter::Options in the osgDB::Registry)."

This commit is only included in OSG 3.4, an update to 3.2.3 wouldn't help.

The bug affects uncompressed DDS textures with built-in mipmaps. I'll have to check the Morrowind files if there's any in there. Edit: none of vanilla Morrowind textures trigger the affected code path.

Ascent
Posts: 39
Joined: 28 Jun 2014, 04:32

Re: Cell preloading

Post by Ascent » 12 Feb 2016, 00:55

Great effort, scrawl! :)
psi29a wrote:Complain to debian, since ubuntu follows them mostly.

https://packages.debian.org/search?keyw ... egraph-dev
Or better yet: https://tracker.debian.org/pkg/openscenegraph

High priority: new upstream available.
Depends on packages which need a new maintainer. Depends: libjasper1 Build-Depends: libjasper-dev
Jasper tracker page: 4 security issues in sid. Two of them crashes with crafted JPEG2000 images. Not good. Why do so many packages I have installed and use depend on this library? :shock:

Post Reply