Multiple ESMs and ESP support

[psi29a]

Shiny is a new submodule that was added in.

http://wiki.openmw.org/index.php?title= ... the_source

git clone git://github.com/zinnschlag/openmw.git
cd openmw
git submodule update --init

the 'git submodule update --init' will get you your shiny.

This has bitten me too.

[Zini]

I am beginning to think that we are approaching this issue in the wrong way. If we continue to treat it as a single self-contained task, it will probably take months before we get it done. That means we will have to drag the branch along for several releases with all the merging trouble that will result from it.

Maybe it is time for a bit divide and conquer. I don't want to include a half functioning support for multiple ESX support in a release, but we could subdivide it into meaningful section and release each of these sections when they are ready.

How about this:

1. initial ESX support, all records except LTEX, CELL, LAND, PGRD, DIAL and INFO are merged. This will allow us to add new IDs via a plugin that can be put into the world via the script console. Probably useful for some rendering testing too.

2. Add DIAL and INFO recrods (first special case)

3. Add CELL, LAND, PGRD, but leave out any new references (second special case)

4. Add reference handling (third special case)

[sirherrbatka]

@mark76

I'm happy each time you push code on github!

[mark76]

Many thanks to everybody who offered help in getting the shiny folder set up. Building and merging now works like a charm (except for the occasional, non-trivial merge conflict). In fact, I have finally found the time to do some actual work on my branch. The following is a list of things that do work now (well, did work before last weekend's merge that did break it again).

- References are now initialized from multiple plugins.

- Moved references are treated *almost* correctly. There are no duplicates, but spawning the reference still depends on the cell where the plugin where it was initially defined (instead of the new target cell).

- Deleting objects defined in a parent file works.

- Refnumers are now treated in a way similar to Morrowind, modified references seem to work as expected.

There are still a few points that need to be done, but - barring a new merge confilct - they should be easy.

- I need to have a look how Morrowind treats deleted references, and try to reproduce it. Objects in a container/NPC might have no Refnumber. Picking up an object resting in the world might delete the object and erase the Refnumber. Dropping an object from the inventory might produce a new "local" reference belonging to the save. I'll have to do some more tests on this, before I can verify this.

- I need to make sure that references moved into new cells are loaded with this new cell instead of the original one. This is more difficult than it sounds - if we were to move full ownership to the new cell, the result might be a duplicate of the corresponding object. There is supposed to be support for moving references between cells in the engine now, but I haven't found the corresponding lines yet.

Unfortunately, recent changes in the master branch did, again, break my work. Just recently, the file reclists.hpp was removed, being replaced/merged by store.hpp. After what I thought to be a regular merge, about 30% of my hacks (all in reclists.hpp) were gone, and I had to do what I had just finished a second time. All hacks outside of reclists.hpp/store.hpp have been restored. However, something has been changed in the internal workings of the global object store, which I haven't fully understood yet, so - you know the game - it will take longer than expected. (I actually hoped I could have everything ready until Christmas before this!)

A major complicating issue is that store.hpp is included by something like 30-50 files. Even though my hacks only touch code used by 2-3 source files, the build system decides to rebuild a *lot* more. Therefore, understanding how the new store.hpp works takes too much time, as most relevant snippets are found in the header file itself, so each time I try something out, it takes 8 minutes to rebuild everything (using "make -j4"). Some time ago, it only took 5 minutes, but maybe there was less to build back then. Anyway, this strongly interferes with actual progress. (Hacking this file is required, as it is the specific point where data objects loaded from esm files are inserted into the global, shared/merged store.)

There's another point I would like to bring up. Where should we actually draw the line between this task (multiple plugins) and secondary points (e.g. dialogue merging)? As this topic (multiple plugins) touches many existing subsystems, drawing this line isn't easy. I suggest that we could draw a line when all references are interpreted correctly, and when all (well, most) plugins are loaded without errors. This should allow to pull my branch to master without too much additional delay (read: before some other commit breaks my hacks again).

Oh, and before I forget - right now, my branch does not build at all. I just had to pull a stop after spending 12 hours trying to get everything right last weekend, and I had to commit thise changes that are correct. If you want to see a working version of the branch, check out the last commit before the most recent merge commit. I'll try to get everything to build again next weekend, before I depart on another business trip to the States. Exactly how much will be fixed until then is open - next Saturday, I have to help out at a birthday, and very early on Sunday, I have to go to the airport and will be away for a week. However, I *will* keep a more active eye at the forums from now on (*crosses fingers and prays that no more Real Life complications emerge*)

[Zini]

- I need to make sure that references moved into new cells are loaded with this new cell instead of the original one. This is more difficult than it sounds - if we were to move full ownership to the new cell, the result might be a duplicate of the corresponding object. There is supposed to be support for moving references between cells in the engine now, but I haven't found the corresponding lines yet.

That is only after loading the ESX files. During the load process just copy and delete manually. I suspect we need to keep track of cell changes during the load process. Maybe a separate global table (listing source cell, destination cell and reference)?

Unfortunately, recent changes in the master branch did, again, break my work. Just recently, the file reclists.hpp was removed, being replaced/merged by store.hpp. After what I thought to be a regular merge, about 30% of my hacks (all in reclists.hpp) were gone, and I had to do what I had just finished a second time. All hacks outside of reclists.hpp/store.hpp have been restored. However, something has been changed in the internal workings of the global object store, which I haven't fully understood yet, so - you know the game - it will take longer than expected. (I actually hoped I could have everything ready until Christmas before this!)

Yeah, that is unfortunate. I had hoped that the ESX loading would be finished sooner. And because of other complications we had to do the store rewrite much earlier than I had planned, which finally lead to this situation. That's why I proposed splitting your task up and getting parts into the master branch early (would have prevented most of the problems you have now).

There's another point I would like to bring up. Where should we actually draw the line between this task (multiple plugins) and secondary points (e.g. dialogue merging)? As this topic (multiple plugins) touches many existing subsystems, drawing this line isn't easy. I suggest that we could draw a line when all references are interpreted correctly, and when all (well, most) plugins are loaded without errors. This should allow to pull my branch to master without too much additional delay (read: before some other commit breaks my hacks again).

I assume you are referring to my task reorganization proposal above. Its perfectly okay to merge the code for loading only certain record types (and ignoring the rest). Basically do all the non-special cases and ignore the special case records for now.

[psi29a]

Hey mark76, thank you again for spending time on this task (now many tasks).

Tracking a moving target is a pain in the ass, especially when you only get bursts of time to work on it. I understand your soreness, just do not think it is a wasted effort. I hope you can understand that the project itself still needs to move on, and the lack of communication/coordination is what created this mess. Pull early, pull often / Push early, push often...

You are still doing good work!

I look forward to testing your code.

Cheers

[ElderTroll]

Yeah, thank you for your continued work, Mark. It will be great once this is implemented.

[Greendogo]

Thanks from me too Mark!

[WeirdSexy]

Indeed, this is a huge and important task. Thank you for your work!

[mark76]

Wow, is it already this late in 2013? I seriously underestimated how much two missing weeks in December (i.e. the pre-holiday season) can mess with a plan - I even missed the Mayan apocalypse . Ah well, a plan is just a list of things that don't happen.

Some things did happen, though. I have finished fixing my branch, and I have also added support for deleting references by plugins (which was a fast thing - yes, not everything is infinitely complicated). Last weekend, I mostly worked on my plan to implement plugins moving references between cells, and I expect I will find some time next weekend to do it.

What we need to support are the following cases:

1) A plugin moves a reference from a parent cell to another cell. Should be straightforward.

2) A second plugin moves the *same* reference to yet another cell. This is not quite as straightforward, as there must only be one instance of this reference. (And it is the reason why there were "compatibility plugins" required for some combinations.)

3) A plugin makes a duplicate of an existing reference and tries to move the duplicate to a new position, but somehow ends up moving the original, leaving the copy in place of the old reference. This is the source of a nasty "vanishing references" bug in Morrowind that is almost impossible to track down - but fortunately, I have found the right plugin combination to reproduce it reliably. If I can make my branch reproduce this behaviour, it is a good sign that I got everything right. If I can find a way to fix it, it's even better - otherwise, I'll make it a task/bug on the tracker.

And that's all I have to say for now; more to come next weekend, when I am back at my development system.

Oh yeah, I almost forgot: A happy new year to everybody!

@zini: let me just attempt to lay out my plan for that nasty thing with moving references:

- Morrowind implements moving references by introducing an "MREF" tag before the actual reference data (plus some extra data fields for the target cell). I *believe* (i.e. haven't tested it yet) that moved references are always found at the top of the reference list, so we might actually parse it at boot time.

- I have mostly understood how the store rewrite works, including the static/dynamic split and the cell store. Now, here's my idea how to track moved references. We introduce two extra lists (maps?) per cell, one for tracking references moved from "this" cell to a different cell. If another plugin decides to mess with the same reference without knowing about the history of this specific reference, the code knows where the reference is currently residing, and can move the actual reference again. (e.g. plugin 1 moved reference from "this" to "other", plugin 2 moves reference from "this" to "yet another". However, the reference in "this" was already deleted by plugin 1, so we actually need to know that the reference to be deleted currently resides in "other", and has to be deleted over there.)

- The second list is used to store references moved into "this" cell. I am not sure if these references need to be made "active" (as it is called in the code) prematurely, i.e. before the cell itself goes active, but if this is the case, we definitely need another list for these references (i.e. what you suggested as a global list, just locally, i.e. on a per-cell base). Fortunately, moving references seems to be comparably rare (unless you consider "high-traffic" cells such as Balmora or Seyda Neen), so the additional work required to track this should be manageable.

Now that you have mentioned a global list of "special" references, we may actually need this to track persistent references or other kinds of "special" references. I have recently produced a few save files (.ess) just to get an idea on which cells are preloaded by Morrowind (in OpenMW, loading Bloodmoon.esm preloads several cells on Solstheim, so the basic ability seems to be there). But this can be definitely split from the main topic.

Since the task was just pushed forward yet another target version , I think I'll try and do as many of these sub-tasks as possible next weekend (I don't have planned anything more important... yet). And then, I'll try a new merge and hope that no more major revisions are required, so we can pull what is there to the master branch.