Generic nif engine importer for openmw

Everything about development and the OpenMW source code.
crassell
Posts: 49
Joined: 26 Aug 2017, 21:10

Generic nif engine importer for openmw

Post by crassell »

In my attempt to understand format import options for animation, I wanted to first understand our entire asset import component set (nif, nifosg, nifbullet). After reviewing the nif streaming and parsing code, I noticed that will be a manual maintenance pain to support all of the various newer nif versions for asset import (oblivion, skyrim, fallout3/4).

I also noticed that another group has already gone through the pain of creating a parseable format for all of the nif versions:
https://github.com/niftools/nifxml/blob/develop/nif.xml

So in an effort to consolidate how we can accept new assets from various sources, I have begun writing a parser generator from the nif.xml. It will make a performant parser tailored to each game and it's corresponding nif version / versions. It will also share the same glue code between common nif objects to the engine. I'm writing the generator in perl so it's easy to change the scripting for it.

Ideally the parser generator can generate object serialization to json or other text formats as well.

The other awesome part is the nif.xml has some parts tailored for specifically nif but a majority of the format specification is asset format agnostic. You could conceivably define new OSG formats, Unreal Engine formats, etc.

Thoughts?
User avatar
psi29a
Posts: 5356
Joined: 29 Sep 2011, 10:13
Location: Belgium
Gitlab profile: https://gitlab.com/psi29a/
Contact:

Re: Generic nif engine importer for openmw

Post by psi29a »

I'm a member of the niftool effort. OpenMW chose to focus purely on Morrowind at the time (years ago) which simplified things for the project.

Correct me if I'm wrong Zini...

It might be worth considering going forward to look more into what was done. Be warned though, Niftool development is 'slow'. I joined to helped speed things up on the Python/Blender side of things. ;)

Ugh.. nif.xml (I hated this and it being broken for things while in development).
crassell
Posts: 49
Joined: 26 Aug 2017, 21:10

Re: Generic nif engine importer for openmw

Post by crassell »

psi29a wrote: 19 Sep 2017, 06:24 I'm a member of the niftool effort.
Awesome. Someone who knows some history behind niftools development!

psi29a wrote: 19 Sep 2017, 06:24 Ugh.. nif.xml (I hated this and it being broken for things while in development).
This doesn't give me much confidence in what i'm currently basing my work on :(. Is the nif.xml unreliable in general? It seems the niflib and other tools base their parsing code on it.
User avatar
psi29a
Posts: 5356
Joined: 29 Sep 2011, 10:13
Location: Belgium
Gitlab profile: https://gitlab.com/psi29a/
Contact:

Re: Generic nif engine importer for openmw

Post by psi29a »

The code, at least from pyffi and the python blender plugin is really convoluted. That being said, that is just the python side of things. The c library on the other hand is pretty nice and the end-result (NifSkope) is great. I ported it to Linux and make use of it quit a bit. :)

https://github.com/niftools/nifskope/pull/101
^-- for example :)

So we can use nifskope as an example on how to use the niflib and nifxml.

https://github.com/niftools/niflib

https://github.com/niftools/nifxml

To be clear, nifxml should be in niflib and not an external dep, but nifxml is also used by pyffi (and QSkope).

So development is convoluted, with submodules and the rest. The idea that the nif format is defined in xml is not that bad (read once), once loaded into memory is nifxml not longer needed.
User avatar
DestinedToDie
Posts: 1181
Joined: 29 Jun 2015, 09:08

Re: Generic nif engine importer for openmw

Post by DestinedToDie »

Is this parser like Blender .nif exporter or something else?
User avatar
psi29a
Posts: 5356
Joined: 29 Sep 2011, 10:13
Location: Belgium
Gitlab profile: https://gitlab.com/psi29a/
Contact:

Re: Generic nif engine importer for openmw

Post by psi29a »

DestinedToDie wrote: 19 Sep 2017, 09:27 Is this parser like Blender .nif exporter or something else?
It's a seperate beast, pyffi (that also uses nif.xml) is used by the blender import_export plugin.

The niflib is not used by the blender plugin.

There was some talk about using python bindings on niflib, which would be interesting in terms of speed when loading large NIFs (40+MiB files).
User avatar
Zini
Posts: 5538
Joined: 06 Aug 2011, 15:16

Re: Generic nif engine importer for openmw

Post by Zini »

I'm a member of the niftool effort. OpenMW chose to focus purely on Morrowind at the time (years ago) which simplified things for the project.

Correct me if I'm wrong Zini...
Partially. We are focussing on Morrowind file formats because its Open Morrowind. Other resources formats were considered of little significance and having readers for multiple formats would have just introduced unnecessary complications.
I am still convinced that a game engine is generally better off with only supporting a single file format for things like meshes (ideally one that is closest to the internal data structures and requires at little effort as possible for reading. Providing resources in the right format (converting if necessary) is best done with external tools.

But that is only my personal opinion and that was a long time ago. From what I remember since then a large amount of support has build up for choosing a different route.
User avatar
psi29a
Posts: 5356
Joined: 29 Sep 2011, 10:13
Location: Belgium
Gitlab profile: https://gitlab.com/psi29a/
Contact:

Re: Generic nif engine importer for openmw

Post by psi29a »

How do you (Zini) feel about adding support for new NIF additions? Wouldn't this help modders in the long run who are used to working with tools for Oblivion, Skyrim/SE, Fallout3/NV/4?

We're moving beyond Morrowind and allowing end users to create their own game, so I wondering how you felt about it now? Would you want to expand our own internal support or would it be a better idea to look at using niflib to do the heavy lifting and translate that into something OpenMW can use?
User avatar
scrawl
Posts: 2152
Joined: 18 Feb 2012, 11:51

Re: Generic nif engine importer for openmw

Post by scrawl »

Not ruling this out, but we should think about the implications before we jump into coding.

My first concern is how fast niflib would be. When I profiled our code a while ago, a lot of time spent loading was in parsing NIF files. So it's a bottleneck that we don't want to make any worse. As I understand niflib was mainly written in the context of NifSkope which uses QT's model/view for the Nif scene graph, and the niflib parsing is built around this model (everything is indexed by strings, etc). Which brings me to the concern that it might not be as fast as our custom written parser.

Second, parsing the file is only half the puzzle. Non-Morrowind versions include non-Morrowind features that need to be handled by the game (e.g. rigid body physics), or converted into OSG equivalents, and there's no point in reading something that can't be used. Some games also use the extra data fields to control game-specific features, like the shader tags in skyrim's nifs. 'simple' files could potentially work just fine if we ignore the new fields, while others will require support for the new features before the files can be used.

My main concern is what the goal should be. Support for any feature in any version of any NIF file is pretty much an impossible goal, with many of them not even decoded. (Even for Morrowind's version we only support a fraction of the NIF nodes that are theoretically possible, we just haven't run into them because Bethesda's assets don't use them and modders are usually too clueless to figure out what they are or how they might be useful). So if we go for additional support we'll want to have a reasonable and useful goal, say supporting Oblivion or Skyrim files. Unfortunately, Bethesda has stated they don't want assets ported between games which makes any specific support of Oblivion or Skyrim on our part risky (whereas supporting 'any' NIF file would be less risky, but too much work with no endgoal).
I am still convinced that a game engine is generally better off with only supporting a single file format for things like meshes (ideally one that is closest to the internal data structures and requires at little effort as possible for reading. Providing resources in the right format (converting if necessary) is best done with external tools.
I agree, partly because I don't want to deal with reports like "horses from empire earth 2 don't load into openmw". Don't want, don't care. We can reject them as unsupported, but that somewhat defeats the point of adding the code in the first place.
User avatar
psi29a
Posts: 5356
Joined: 29 Sep 2011, 10:13
Location: Belgium
Gitlab profile: https://gitlab.com/psi29a/
Contact:

Re: Generic nif engine importer for openmw

Post by psi29a »

scrawl wrote: 19 Sep 2017, 11:21Unfortunately, Bethesda has stated they don't want assets ported between games which makes any specific support of Oblivion or Skyrim on our part risky (whereas supporting 'any' NIF file would be less risky, but too much work with no endgoal).
I've discussed this at length in another thread, but this does not impact OpenMW. Bethesda made it clear that they didn't want assets to be ported around on their engines. They have no legal basis enforcing this with OpenMW. They can pursue modders on nexus web, because the IP of those assets are still the property of Bethesda but that is up to the end-user. They cannot enforce their EULA when people do port assets privately between their game-engines, that is just practically impossible. They can only ask for the content that is available online to be removed.

I see no legal risk (IANAL) in supporting NIFs from other games in OpenMW. It would allow modders of Oblivion and Skyrim (and others) to be able to use the content they have created with other tools to also be used in OpenMW which is fine and should be encouraged, especially when the assets are purely their own and not derivatives of Bethesda's IP.
Post Reply