Multiple ESMs and ESP support

Everything about development and the OpenMW source code.
User avatar
Zini
Posts: 5536
Joined: 06 Aug 2011, 15:16

Multiple ESMs and ESP support

Post by Zini » 02 Mar 2012, 12:50

This task is scheduled for 0.14.0 and I think lgro wanted to take it.

I would like to do a brain dump before work on it starts.

Basically the task consists of reading repeatedly ESM/ESP files in the predefined order and on each pass:

- adjust modified records in the ESMStore
- add new records to the ESMStore
- delete records from the ESMStore which are flagged as deleted

But there are two special cases that needs to be considered:

1. Cells

Cells are stored in two locations:

1a. the immutable part is stored in the ESMStore

1b. the mutable part (the table of references in the cell) are stored in ESMS::CellStore instances, which are managed by MWWorld::Cells. These are only loaded on demand.
The CellStore uses an optimisation mechanism that allows to quickly load a list of possibly contained IDs (this is required for various scripting functions). This is indicted by the state enum State_Preload. When a cell becomes active, the cell is fully loaded (State_Loaded).

So what needs to be done here is to implement mechanisms to preload and load cells from multiple ESX files.
Preloading should only add IDs, not delete them and not check for duplicates (we are trying to be as fast as possible here).

There might be a problem though. The cell record struct stores an instance of ESM_Context, which indicates a file position for fast access to the cell's record in the ESM file. I haven't looked into the implementation, but if this involves keeping the file open all the time, we have a problem. The number of ESX files can become very large and we probably shouldn't keep that much files open at the same time. So this needs to be looked into.

Also, I think when an ESX file modifies a cell the old reference list is complete discarded. But I could be wrong about it. That is a point that also needs to be checked.

2. Terrain textures

I don't have the details at hand. But one of the record types needs special treatment, because it references another record type, but not from the merged ESX stack, but from the current ESX file. There are probably more details somewhere in the terrain thread.


I suggest the following procedure:

- Implement the general case and special case #1 in a new branch (esx).
- Merge this branch into the terrain branch and implement speical case #2 there.

By doing it this way we make sure we can release the ESX feature, even if the terrain feature is not ready at this point (which mostly depends on how fast the OGRE guys are with getting 1.8 out of RC state).

Yacoby
Posts: 119
Joined: 30 Sep 2011, 09:42

Re: Multiple ESMs and ESP support

Post by Yacoby » 02 Mar 2012, 13:09

Zini wrote: 2. Terrain textures

I don't have the details at hand. But one of the record types needs special treatment, because it references another record type, but not from the merged ESX stack, but from the current ESX file. There are probably more details somewhere in the terrain thread.
To save someone having to do some digging. The LAND.VTEX contains an array of indexes to LTEX records in the same ESX file. So you can't merge the LTEX records.

User avatar
Zini
Posts: 5536
Joined: 06 Aug 2011, 15:16

Re: Multiple ESMs and ESP support

Post by Zini » 02 Mar 2012, 13:41

Thanks.

User avatar
Greendogo
Posts: 1376
Joined: 26 Aug 2011, 02:04

Re: Multiple ESMs and ESP support

Post by Greendogo » 24 May 2012, 07:09

Is the Crystal Scrolls developer still planning on doing this feature? There hasn't been any work done on it yet (so says the tracker) and it's been pushed back several releases. Might someone else finish it sooner?

User avatar
scrawl
Posts: 2152
Joined: 18 Feb 2012, 11:51

Re: Multiple ESMs and ESP support

Post by scrawl » 24 May 2012, 07:25

You should read his thread in the "New developers" section. He almost finished the task already (although it seems he doesn't like to share his progress very often)

User avatar
Greendogo
Posts: 1376
Joined: 26 Aug 2011, 02:04

Re: Multiple ESMs and ESP support

Post by Greendogo » 25 May 2012, 05:27

Ah, I see. I didn't think to look there at all, since it's not the place to look for it. :D

mark76
Posts: 17
Joined: 20 Feb 2012, 10:33

Re: Multiple ESMs and ESP support

Post by mark76 » 10 Jun 2012, 11:07

Yes, it is almost done, and I have just pushed the code to the new branch "multiple_esm_esp". Now that most technical issues are moved out of the way, let us continue the discussion here.

If you have already seen my recent updates in the application forum and the issue tracker, you are already aware of some stuff I was thinking about. To summarize, here is the current state of what works and what may need to be done, either in issue #29, or somewhere else:

What works:
- Master and plugin files can be selected from the launcher and are correctly passed to the main engine.
- Engine recognizes multiple esm/esp files and properly loads them.
- Sometimes a plugin deletes an entry defined in a master file. This is supported.
- Perhaps the most urgent point that needs to be addressed in multiple esX file support is supporting per-plugin terrain palettes (i.e. texture lists), which can easily result in completely wrong textures and crashes if not done correctly. I have added a quick hack to support multiple terrain palettes, but some extra work is required before it fully works.

What does not work/what needs to be made aware of multiple plugins (maybe in a separate issue topic, and in collaboration wih the people responsible for the individual code touched by this):
- As already mentioned, some (hopefully minor) points with the terrain subsystem.
- Testing whether the GMST fix already works or if more work is required.
- Support for new script commands. Zini already mentioned that scripting is a separate topic, but it would be nice to have the errors from the script compiler about unknown commands gone. Implementing them is a completely different issue, I have to agree with that.
- Some plugins are using some NIF features currently unsupported. I'll gladly help working stuff out once it has been identified. For example, Cait's squirrels did crash my own NIF loader because some stuff that I was assuming earlier on "sane" NIF files was broken by this custom mesh. Bloodmoon folilage looks weird. And I am sure there more issues emerge after some more exhaustive testing.
- Merging dialogues is tricky and may require more work, depending on what the dialogue subsystem can already do and what not.

Finally, there are a few points that are more complicated, and which also need to be addressed. First, it's moving references. From what I understand from previous discussions, this is a rather "hot" topic. This also touches issue 28 - whether CELLs should be (pre)loaded or on-demand. Multiple plugins introduce another issue to this happy collection of complications. In a nutshell:
- Each object placed in the world has a unique reference ID, a 32-bit integer. This ID is divided as this:
ID = pprrrrrr
pp = plugin number, *not* constant
rrrrrr = semi-random reference ID, constant
Now, without going into too many details, the "pp" is the index of the modified reference in the *local* list of parent plugins. Say, "some_plugin.esp" moves something defined in "Bloodmoon.esm". The plugin depends on the parent files "Morrowind.esm" and "Bloodmoon.esm", so the moved reference uses an ID of "02rrrrrr". However, when running in a game that also uses "Tribunal.esm", Bloodmoon may be number 3 in the list of esX files, and Morrowind changes the local "pp" value accordingly.

This is also the reason why Morrowind only supports 255 plugin files - 8 bits can only hold that much, with the currently running game/save file being plugin number 0. So getting references right can be really ugly.

So much about what I was thinking right now, now I have an actual question :D On the issue tracker, issue 28 was related to this one (but it was recently bumped to 100%), i.e. the question whether CELLs should be preloaded on boot or delayed until they are required. Moving references defined in a plugin file complicates this. If, for example, you move the Balmora Stilt Strider to Caldera, the resulting plugin will do this:
- The Stilt Strider still belongs to Balmora, i.e. the cell where it was originally defined.
- The original reference gets some new data fields, essentially saying "moved to Caldera".
- Caldera is given the information that "something was moved here", but not what it is.
Therefore, I am afraid that we will need to parse some parts of the CELL entries on boot time, just to pick out moved references, reordered by their *target* cell. A similar list may be required for "persistent" references, which are addressed by some scripts. I have not yet looked into deleted references yet (i.e. the player picks something up and drops it again).

Quite some tricky stuff, and if the plans for inroducing a separate save file format are to progress, this point (references) must be taken onto account as well.

User avatar
Zini
Posts: 5536
Joined: 06 Aug 2011, 15:16

Re: Multiple ESMs and ESP support

Post by Zini » 10 Jun 2012, 11:20

Lots of stuff and I will have to think about it a bit more before I can comment on it. A bit short on time right now, so it might take a couple of days (Sorry).

From what I see you should have a look at the developer section of our wiki again, especially the naming conventions.

About the 256-limit and the reference changes: I haven't looked at it in detail yet, my initial feelings are that we should throw out the whole concept and reengineer it from scratch.

User avatar
werdanith
Posts: 295
Joined: 26 Aug 2011, 16:18

Re: Multiple ESMs and ESP support

Post by werdanith » 11 Jun 2012, 20:08

Tried the branch on github.
The two expansions work for the most part, except in the areas around khuul. If I get anywhere near there then I can't move and I get spammed with these:

Code: Select all

loading cell -9, 17
Error in framelistener: ESM Error: Expected subrecord FRMR but got CELL
File: /home/michael/code/openmw/data4/Bloodmoon.esm
Record: CELL
Subrecord: CELL
Offset: 0xe70e8
As far as plugins go though, pretty much nothing works.
Most plugins segfault while trying to load with something akin to this:

Code: Select all

#0  0x083dcd4a in Ogre::SharedPtr<Ogre::Texture>::destroy() ()
#1  0x083dc8ec in Ogre::SharedPtr<Ogre::Texture>::release() ()
#2  0x083dc738 in Ogre::SharedPtr<Ogre::Texture>::~SharedPtr() ()
#3  0x083dc5b0 in Ogre::TexturePtr::~TexturePtr() ()
#4  0x0866c646 in OEngine::Render::OgreRenderer::~OgreRenderer() ()
#5  0x08669b9b in OMW::Engine::~Engine() ()
#6  0x086611c9 in main ()
And the ones that do work, end up looking rather scary :P
Like better bodies for example.

User avatar
Zini
Posts: 5536
Joined: 06 Aug 2011, 15:16

Re: Multiple ESMs and ESP support

Post by Zini » 11 Jun 2012, 20:20

Not surprising that most plugins don't work. The cell loading part isn't implemented yet. Still didn't found the time to look into it, but from what I remember from my first glance at the code, loading new cells added by ESX files might work. ESX files that modify existing cells probably don't.

Post Reply