Multiple ESMs and ESP support

[mark76]

Phew, finally feeling better, and I actually found enough spare time to return to this project.

Okay, here's what I have been thinking up how to continue with the task at hand. Earlier in this thread, I had mentioned several points that will need to be tested/improved/modified in a broader context (i.e. with many plugins, NIF files etc. in use). My main goal when posting this was to summarize those points where supporting multiple plugins introduces additional complicatons (compared to a single master configuration), partially based on bits of code I had seen during my early hacking attempts. Just wanted to get these points out in a somewhat centralized way so they can be addressed before the project enters the release candidate stage. (There's a nice example how it should not happen. Before the Linux Kernel went stable, there were 15! development versions of V0.99, mainly because new issues kept emerging no one thought of earlier.)

- Scripts and nif files are indeed a separate subsystem and can be updated independently of my branch.

- Any GMST fixes that might still be necessary can be fixed when the need arises, i.e. when game rules are actually being implemented. Similarly, Dialogue fixes that might be required can be added as a part of the dialogue subsystem; before some code that actually needs it is available, any kind of fixes are purely theoretical.

- The terrain issue concerning unclean landscape seams can be fixed independently of the multi-esX code as well; my current hotfixes prevent crashes and make everything look reasonably well. Maybe it is better to keep building on my existing branch, than making extra branches from my branch.

- Merging references and tracking changes (i.e. some preliminary support for game saves) actually seems to be one of the more important things. As already mentioned, it seems to touch issue #28 (i.e. when to load cells and references), and I expect that it requires some refactoring of code at different points in the cell streaming code.

So much for my thoughts, and here's what I propose I could to next.

- I will generate (and post) a few test plugins that specifically trigger conflicts related to the points mentioned above. This should make further testing/fixing testing much easier.

- I still need to test a few more things related to moving references between cells and make sure it does not crash on load (haven't checked yet if some more exotic tags are already supported or not).

In a nutshell, after fixing what might still need to be fixed, I'll polish the code, merge with the latest master branch, and send a pull request, so that you can work with it without having to branch from my branch - which is possible but tends to make things more complicated. (Plus, from what I have seen during the previous months, code that is unfinished but somehow works a bit can be pulled to the master branch if it is sufficiently stable.)

When this is done, I am willing to have a deeper look into the "moving references" issue, trying to support this uglyness in the file formats (and maybe reopening #28?). I'll also look into other topics and see if I can provide help.

[Zini]

When this is done, I am willing to have a deeper look into the "moving references" issue, trying to support this uglyness in the file formats (and maybe reopening #28?). I'll also look into other topics and see if I can provide help.

I am still note sure about the specifics about. I posted a question about it a while back. Maybe you missed it?

If there is really moving involved as you described, than we need to get back to the drawing board, because there is no way anything good can come from this. We will definitely not store references in this way. Maybe these cell changes can be tracked in a separate global table. Or we can come up with something else. But we will definitely not use a system that requires looking up a reference in cell A when it is in cell B. I am pretty certain that this kind of nonsense is responsible for a lot of MW's bugginess.

btw. I wouldn't worry about saving now. We can take care of that in a separate task (much later).

btw. (2) I noticed that you added a 255 plugin check. Remove that please. We need to get rid of it anyway, so we can as well have proper data structures right from the start.

Edit: Since this looks like a long running branch (we can't really merge in this feature until it is fully functional), I suggest you merge in my master branch once in a while, so that these two branches don't diverge too much.

[mark76]

Finally back from India, and learned a few new things. 1) If you can avoid it, don't go to India. 2) If you have recently had weird lung issues, don't transit in Dubai. 3) Don't go to India. 4) If you like to see Murphy's Law in action, go to India; otherwise, don't. 5) If you ever intend to drive a leased car, don't go to India. 6) If you want to get some sleep while in the taxi to the airport, don't go do India.

Okay, enough life coaching. I have finally found the time to look in to some more points. The existing dirty GMST check code looks sane to me (unfortunately, the netbook I am writing this from is too slow to run OpenMW in a debugger). For further testing, I have attached an archive with a few "dirty"/testing plugins that can be used to test whether some of the complications I have pointed out earlier will work as expected. One plugin with many dirty GMST entries, and two related to moving/deleting existing references. This should be helpful just in case somebody is/might be/will be working on some of these aspects.

@Zini: I did indeed miss your earlier question, but I'll have an answer up in a few days. I'll remove the 255 plugins check; unless somebody actually comes up with a plugin that depends on more than 255 parent master/plugin files, it should be save to convert everything on-the-fly to a more efficient (internal/binary) format. Or whatever is required to support moved references.

Unfortunately, you did understand it perfectly well. Moving a reference defined in a master file will cause the Construction Set to write the following two things in the target plugin (you can verify it with a hex editor and the test plugin attached to this post):

1) In the cell where the reference was initially defined, the following snippets will be added:
- a "MVRF" tag (= Moved Reference, currently unaccounted for in "cell.cpp")
- some additional tags that define the target of the cell. If the object is moved to a different cell, the target will be defined here.

2) In the cell where the reference is moved to (i.e. the "target" cell), there will be a "NAM0" tag with an integer data value of "1", which tells the engine that "1 modified reference is here". Here, "modified" seems to cover "new reference", "modified reference" and even "moved reference" - however, as you understood it quite right, there is *no* reference to what was actually moved here! This might actually be the reason why a heavily modded Morrowind is so (censored) slow - the overhead is growing quite badly. (Well, on the other hand, Bethsoft never expected the modding community to grow so huge, so it's almost understandable...)

Now, you mentioned that, in this case, it would be "back to the drawing board" - and in fact, this is one of the main reasons why I didn't make much progress with either OpenMW or Crystal Scrolls over the past time. Just in case it's helpful, here is what I considered to do when I last thought over it, maybe it helps with brainstorming to overcome the underlying problem:

Obviously, the data format used by Morrowind is horribly inefficient in some points. Then, I remembered something from Dungeon Siege 1 - when this game is started for the first time, you will get some messages saying "condensing (something... forgot the expression)... this should only happens once". So, here's my idea, which might allow to keep some compatibility with the native Morrowind file format: Generate a second, cached/precompiled version of the plugin files when this cache file does not exist yet/is out-of-date. This way, we could easily extract/reshuffle the data required by various subsystems that needs to be optimized, and bring it into an internal format that is only used by OpenMW, but not by Morrowind. On the other hand, we could also keep compatibility with the original Construction Set. But this stuff clearly must be worked out before somebody can start coding it.

Okay, this is essentially what I was thinking about, given the latest developments. Now, as you already mentioned to keep merging with your master branch - I recently tried to do this, but (still beeing comparably new to git) something seems to not quite work as expected. According to Google, the correct procedure to keep up with developments in "master" works like this:

Code: Select all

git checkout master // probably not necessary, but better safe than sorry
git fetch origin master // download current changes to "master"
git checkout multiple_esm_esp // make sure I am on the correct branch
git merge origin/master // merge downloaded changes into current branch

According to most things found on the Web, this should do some output related to merging the individual commits. However, all I am getting is "Already up-to-date". Google gives a lot of similar questions, but I can't seem to find the correct answer, as there are too many different variations of "developing with git" out there. So, before I try to start messing something up, I hope that somebody with more git experience can help me in figuring out why my git seems to be so unresponsive.

[Zini]

Obviously, the data format used by Morrowind is horribly inefficient in some points. Then, I remembered something from Dungeon Siege 1 - when this game is started for the first time, you will get some messages saying "condensing (something... forgot the expression)... this should only happens once". So, here's my idea, which might allow to keep some compatibility with the native Morrowind file format: Generate a second, cached/precompiled version of the plugin files when this cache file does not exist yet/is out-of-date. This way, we could easily extract/reshuffle the data required by various subsystems that needs to be optimized, and bring it into an internal format that is only used by OpenMW, but not by Morrowind. On the other hand, we could also keep compatibility with the original Construction Set. But this stuff clearly must be worked out before somebody can start coding it.

Sounds reasonable. But please keep in mind, that we will have to modify the ESX format anyway once OpenMW 1.0 is done (to support additional features). Compatibility with Morrowind or the original CS is not needed. So what we should do here is to only translate legacy ESX files (currently all). Newer ESX files will already be in the correct format.

So it seems we need to define some post-1.0 format extensions now. The basic structure should be kept unchanged (record based). I propose we that we store in the cells only the actual objects. No reference to other cells, no "moved" flags.
The moving of objects between cells should be recorded in a new record type, that will be used by our editor and also by OpenMW to track changes when a saved game is loaded with a different set of ESX files.

Maybe it would be a good idea to implement this feature (conversion from old to new format) as a separate command line tool? Might be useful in a few other cases later.

Does that makes sense?

According to most things found on the Web, this should do some output related to merging the individual commits. However, all I am getting is "Already up-to-date". Google gives a lot of similar questions, but I can't seem to find the correct answer, as there are too many different variations of "developing with git" out there. So, before I try to start messing something up, I hope that somebody with more git experience can help me in figuring out why my git seems to be so unresponsive.

No. That is mostly wrong.

Maybe a bit more theory will help: You are working with 3 different repositories. Your local repo, your repo on gitub and my repo on github.

When you do a fetch, you fetch or pull from another repo to your local repo (push is the opposite direction). Access to non-local repositories is managed via git remote.

origin is your repo on github. So when you did a git fetch origin, you fetched from your own repo which had not changed. Instead you should fetch from my repo, which you need to add via git remote.

github actually has very good help on this topic (far better than my explanations). I suggest you have a look.

[Chris]

2) In the cell where the reference is moved to (i.e. the "target" cell), there will be a "NAM0" tag with an integer data value of "1", which tells the engine that "1 modified reference is here". Here, "modified" seems to cover "new reference", "modified reference" and even "moved reference" - however, as you understood it quite right, there is *no* reference to what was actually moved here! This might actually be the reason why a heavily modded Morrowind is so (censored) slow - the overhead is growing quite badly. (Well, on the other hand, Bethsoft never expected the modding community to grow so huge, so it's almost understandable...)

That could be to account for something more happening to the reference. If the reference is deleted, then the target cell will be pointing to an invalid reference. Or if another plugin also moves the reference to a different cell, then two cells will both be using the same reference unbeknownst to each other.

By just putting a "reference goes here" marker in the target, and rely on the engine to actually move the original after all the plugins are accumulated, it avoids those problems. Once all the references are patched from the plugins (and save file), it does a one-time move to put it where it needs to go, and the extraneous NAM0 tags are simply ignored.

This should only affect initialization. As far as I understand it, all the esms and esps are loaded once when the game loads, and the ess is loaded when a save game is loaded. After the ess is loaded and the game data initialized, the engine shouldn't be bothered by how reference moves are specified in esx files.

[Zini]

That would require loading all cells on startup and then going through all of them to clean up the object movement mess. Say hello to extra long load times. And significantly increased memory requirements.

[Chris]

That would require loading all cells on startup and then going through all of them to clean up the object movement mess. Say hello to extra long load times. And significantly increased memory requirements.

It'd require loading all the esx data, which AFAIK is done anyway. Once it's all loaded, then it moves whatever references need to be moved. I don't see what would take particularly long about it.

If you want, however, you can probably look to see what Oblivion or Skyrim does. They manage to load up faster than Morrowind when many mods are in use. Though given that they use a 32-bit (8.24) int for IDs instead of strings, that could account for it somewhat.

[Zini]

It'd require loading all the esx data, which AFAIK is done anyway

Nope. We are skipping over references in cells and only load them when needed.

[Chris]

Nope. We are skipping over references in cells and only load them when needed.

That doesn't sound optimal. Can't scripts can change references that aren't in loaded cells? In the future, won't we allow AI processing for actors not in loaded cells (e.g. allowing guards to continue to track you after you leave the cell)?

I know when vanilla Morrowind starts up, it loads each esm/esp in turn then does an extra pass to "initialize" (i.e. fix up) the data. When you load a game, the ess gets loaded on top of this and another pass to initialize data is done, before it ever starts loading area data. There isn't that much data, either.. Morrowind.esm, Tribunal.esm, and Bloodmoon.esm are less than 100MB total, and I'd be surprised if my heavily-modded game (which includes Morrowind Patch Compilation Full 1.2 and Tamriel Rebuilt) tops 250MB in esm and esp files.

[Tarius]

Nope. We are skipping over references in cells and only load them when needed.

That doesn't sound optimal. Can't scripts can change references that aren't in loaded cells? In the future, won't we allow AI processing for actors not in loaded cells (e.g. allowing guards to continue to track you after you leave the cell)?

I know when vanilla Morrowind starts up, it loads each esm/esp in turn then does an extra pass to "initialize" (i.e. fix up) the data. When you load a game, the ess gets loaded on top of this and another pass to initialize data is done, before it ever starts loading area data. There isn't that much data, either.. Morrowind.esm, Tribunal.esm, and Bloodmoon.esm are less than 100MB total, and I'd be surprised if my heavily-modded game (which includes Morrowind Patch Compilation Full 1.2 and Tamriel Rebuilt) tops 250MB in esm and esp files.

Actually, I have over 425mb of esp and esm files. Granted I dont use all of them at once, but I think a couple hundred MBs isnt out of the question for heavily modded stuff.

That said, I agree that not loading things is not good. The main reason is because it elminates the possiblity of having NPCs move around. Although, I dont think there is any reason to load everything in existance and it would be fine to only do those nearby cells.
One thing to note, there is a mod called Enhanced Persuit that allows NPCs to chase you through cells.