A lot of discussion happened while I slept it seems.
crassell wrote: ↑19 Sep 2017, 06:42
So development is convoluted, with submodules and the rest. The idea that the nif format is defined in xml is not that bad (read once), once loaded into memory is nifxml not longer needed.
Good to hear that nifxml is a solid choice at least from a format spec. I'm going 1 step further in this generator by not loading the nifxml at all for the engine parsing code i'm creating but instead generating the c++ code that matches the format directly.
psi29a wrote: ↑19 Sep 2017, 10:09
There was some talk about using python bindings on niflib, which would be interesting in terms of speed when loading large NIFs (40+MiB files).
This was my fear with both the blender importers and even niflib after reviewing the version check code that's required. The speed is a huge concern which is why I'm going down generating parser code to target an exact subset of nif versions optimized for our uses.
scrawl wrote: ↑19 Sep 2017, 11:21
My first concern is how fast niflib would be. When I profiled our code a while ago, a lot of time spent loading was in parsing NIF files. So it's a bottleneck that we don't want to make any worse. As I understand niflib was mainly written in the context of NifSkope which uses QT's model/view for the Nif scene graph, and the niflib parsing is built around this model (everything is indexed by strings, etc). Which brings me to the concern that it might not be as fast as our custom written parser.
I'm definitely concerned about this as well which is why i'm not using niflib but instead going down a hopefully more performance tuned path. I already see some slight optimizations to be made on the existing nif format reads in openmw assuming we are not limited by file I/O already. I'll be doing benchmarking throughout this work so in the worst case we're no worse off and in the best case we get a perf improvement.
scrawl wrote: ↑19 Sep 2017, 11:21
Second, parsing the file is only half the puzzle. Non-Morrowind versions include non-Morrowind features that need to be handled by the game (e.g. rigid body physics), or converted into OSG equivalents, and there's no point in reading something that can't be used. Some games also use the extra data fields to control game-specific features, like the shader tags in skyrim's nifs. 'simple' files could potentially work just fine if we ignore the new fields, while others will require support for the new features before the files can be used.
This was already going to be a mixed bag as we have to unravel how the engine would even support some of the later nif versions and all of the asset modifications they introduce. Hopefully this will lay the groundwork for supporting other elder scrolls games even if there is much work to be done to support everything that is being read. I'm also going to use this parser to write converters for other formats which may lead into the OSG native support we want.
scrawl wrote: ↑19 Sep 2017, 11:21
So if we go for additional support we'll want to have a reasonable and useful goal, say supporting Oblivion or Skyrim files
This is more of the scope of work I would like to keep with the other recent Bethesda RPG's as an option (Fallout 3 / Fallout New Vegas / Fallout 4).
scrawl wrote: ↑19 Sep 2017, 11:21
I agree, partly because I don't want to deal with reports like "horses from empire earth 2 don't load into openmw". Don't want, don't care. We can reject them as unsupported, but that somewhat defeats the point of adding the code in the first place.
This is not my goal either and I should have better stated that in my initial post. The parser generation is merely a means to an end. I could for instance use a tweaked version of the nif.xml dtd to define a custom OSG format for animation. It could then share the import conversion logic between both the nif format path and the osg format path. I realize i'm speaking in ideals but it has potential.
Chris wrote: ↑19 Sep 2017, 14:04
I would actually prefer to not really focus on other NIF formats because of spotty support (and lack of proper documentation), but instead turn the attention to more readily usable formats like OpenGEX and glTF 2.0.
My thought here was to extend nif support for future endeavors with other bethesda titles as well getting new functionality with what the later asset versions offer but I do agree that reverse engineering is not for the faint of heart if nif still has a lot of mystery object data.
I'm starting to understand there's a lot of differing views on the right path forward going by what we discussed previously in this post:
viewtopic.php?f=22&t=4591&start=10
Chris wrote: ↑19 Sep 2017, 14:04
Ideally we should leverage a general model loading API to support multiple formats. Like how we use FFmpeg to support not only the wav, mp3, and bik media files, but also gain support for ogg, flac, and webm with no extra work on our part. OSG has a general model loading API, and while I understand there are issues with handling specific features of Morrowind's NIFs through it, if we could work that out it would be a big boon as we'd not have to worry much about model formats further. We can assume OSG has or will have import plugins for formats modders would like to use, and anyone that wants to add support for certain other formats would work with OSG rather than us (though we could still be available for advice), so it eases the overall amount of work we have to do for future format support.
Except for the problems with animations, couldn't this be done today if it's entirely in the realm of OSG supported plugins? I guess someone has to look through all the plugin options and separate the useful from the redundant.
scrawl wrote: ↑19 Sep 2017, 16:21
It should be noted that even with no support in OpenMW, one can still use osgconv to convert insert-obscure-format-here into .osg and load that into OpenMW.
I could conceivably make the generic nif parser i'm creating spit out OSG format. That would alleviate the maintenance issue on openmw directly (you wouldn't officially support oblivion/skyrim/fallout 4/etc. formats), but you could point to this tool as a means to convert bethesda archives of static assets to OSG. From there the problem is all the custom things for physics data and such that have nothing to do with the graphics engine. Not sure how that fits in.
scrawl wrote: ↑19 Sep 2017, 16:21
Animations are another issue. The closest thing to an animation 'standard' in OSG is the osgAnimation node kit, and this is also what the model importers turn animations into. But OpenMW uses a custom animation component, mostly for optimization reasons and because osgAnimation doesn't support a peculiarity in Morrowind's way of skinning. So the osg importers wouldn't work for animations anyway, unless we did some sort of conversion that loads osgAnimations into our custom system, or change the animation playback in the engine to look for both Morrowind-style controllers and 'osgAnimations' (sounds like a maintenance nightmare).
This was discussed in my previous developer application post. It seems like regardless of format, we will have to tailor some conversion to fit into openmw's animation format. This means OSG / OpenGEX / glTF 2.0 are all on the table. I still haven't given up on this.
raven wrote: ↑19 Sep 2017, 16:25
For performance you'll probably want something binary that does not require much, if any, parsing.
For any consideration of a text based object format without a binary option, I wouldn't even consider it without rolling a binary format to complement the text version.
scrawl wrote: ↑19 Sep 2017, 19:29
* Write a osgAnimation -> OpenMW converter and run this at load-time. This way we can, in theory, make use of osg's importer plugins for all sorts of formats and use the osg blender exporter as-is.
This is most likely the best path of the 2 options. Rolling our own exporters for blender would create redundant work.