Difficulty of leveled list merging in OpenMW

[Ravenwing]

So I know this isn't really on the table at any point in the near future, but I stumbled upon this in the wiki and it got me wondering:

Does openmw need an outside utility to merge leveled lists or does it do what the original engine does and only take the last changes?
If not, are there plans to implement such a feature?
OpenMW does not merge leveled lists (or any other lists for that matter, e.g. pathgrids, spell lists, etc.). Doing so cleanly is not possible, because the ESM format does not specify which list items were added, changed or deleted; it simply includes the whole list for each file that does something with that list. For a future iteration of the file format, we could eventually include this info, and then merge lists during runtime (for files that were created in this format in OpenMW-CS, anyway).

Why is this difficult to do? I could see that it would slow down the function, having to process and compare the full lists in every ESM, but aren't most computers probably fast enough to sort through a few hundred lists with a few hundred entries at this point? Even if it is slow, it only has to be performed at startup when the modlist changes. I do think this is functionality that we should include at some point, because I know this was one of the main reasons people HAD to use Wrye Mash if they cared about this stuff at all.

[Chris]

Why is this difficult to do?

Because as it mentioned, a leveled list is a record in the esm file. When a mod specifies a leveled list, it's either completely new leveled list record, or is a complete redefinition of the record (with whatever the mod desires). There's no indication of if the mod's definition needs to replace or append/edit the original list, and as the vanilla engine does a replace, that's what OpenMW has to do to work properly. Fixing it would require the record to indicate how it should be applied to the preexisting record.

[SquireNed]

Seems like something that could be added in relatively simply, though.

Simply make each list have a end delimiter that says what's going to happen: if there's none in the list, default to old behavior.

[magamo]

I don't know that I see the necessity of this aside from a minor quality of life improvement. tes3cmd and it's multipatch feature seem to work well enough with OpenMW to me, although it does not specifically work with the new multiple data directory form of installing mods. At least in Linux and Mac OSX, you can symlink all your esms and esps into a temporary directory, create a multipatch and then dump that into its own data directory to take care of this.

The author of tes3cmd is aware of some of its deficiencies when it comes to OpenMW, and seems willing to work to bring in more full compatibility in the future.

[Ravenwing]

There's no indication of if the mod's definition needs to replace or append/edit the original list

Ah that makes sense. I had assumed just do a full comparison of each list and simply mash everything together proportionally, but that doesn't account for when you're purposefully replacing and removing other records. Well at least this would be a relatively trivial functionality to add to OpenCS created content.

tes3cmd and it's multipatch feature seem to work well enough with OpenMW to me

I mean, it would be ideal to not have to use any third party software, but good to know there's something we can use if we need to.