Better handling of mod conflicts and mod merging

Feedback on past, current, and future development.
Post Reply
User avatar
Greywander
Posts: 119
Joined: 04 Dec 2014, 07:01

Better handling of mod conflicts and mod merging

Post by Greywander »

I know this has been a bit of a contentious topic in the past, and the OpenMW generally doesn't want to be responsible for things like merging leveled lists (as this would lock users into using OpenMW's specific merging method), but I was thinking about some ways that this could be improved.

Disclaimer: I haven't modded Morrowind in quite some time, so I'm going based off of my memories of modding. It's possible that some of what I'm about to suggest may have already been implemented.

The way things currently work is that you have your .esm/.openmwgame file that determines the "default" values of all the records in the game. Then you have a mod's .esp/.openmwaddon file that modifies those default values. As long as two mods don't modify the same record, then those mods are compatible. If two mods modify the same record, however, the game can't use both, and whichever one loads last is the one that gets used.

Example: If one mod gives a merchant more gold, and another mod adds new items to their stock, they are incompatible because they both modify the merchant. Even though we, as humans, can see that there isn't actually a conflict, and the two mods could be merged together just fine, all the game sees is that the merchant has been modified twice, so it can only use one of the modifications.

So first of all, what I'd like to see is moving things down to the next layer. Yes, the merchant has been modified, but what aspects of the merchant have been modified? The merchant's gold and inventory are two separate fields, each of which can be marked separately as modified or not. Thus, when the game loads the two mods, it sees that yes, the merchant has been modified by both, but one is modifying one field and the other is modifying a different field. Thus, no conflict, and the game can run both mods at the same time.

But wait, we can go deeper. What about two different mods that add (or remove) items from a merchant's stock? Again, we as humans can understand that two different mods that both add items to a merchant should be able to get along just fine. So what can we do here? My proposal is to not only mark specific fields that are modified, but also record what that modification is.

Let's say, for example, that a merchant has a default gold of 500. You make a mod that gives him 1000. Rather than the mod just giving him 1000 gold, it actually just adds +500 to his default gold. If the .esm/.openmwgame is later changed to increase his default gold to 1000, then with your mod he now has 1500. If someone else makes a mod that increases his gold from 500 to 2000, it would register as +1500, and thus, with the modified default of 1000, plus your mod for +500, plus the other person's mod for +1500, the merchant now has 3000 gold.

Actually, let's have, say, three different types of modifications:

Absolute - Works the way it currently does. Whatever you modify this field to, it is set in stone that way, regardless of any changes to the .esm/.openmwgame or any mods that loaded before this one. Mods that load after this one can still modify this field.

Additive - How I proposed above. If you add an item to a merchant's inventory, that modification is recorded as +1 of that item, and will successfully merge with other additive mods. It will also merge with absolute mods if the additive mod loads after.

Multiplicative - A bit of an odd twist on the above. This would let you, say, double or half a value, regardless of what that value might be changed to by another mod. The order of operations would be the same as the load order: if an additive mod loads first, it will add to the default value, then get multiplied, but if the multiplicative mod loads first, then it will multiply first, then add.

Another option that just occurred to me would be to be able to tag additive (or multiplicative) mods to ignore redundancies. Meaning, if two mods add the same item to a merchant, but one is marked to ignore redundancies, then only one of that item gets added.

As for leveled lists, my understanding is that most mods for leveled lists either add something to the list or take something away. Thus, using additive (possibly with ignore redundancies) would allow leveled lists to be merged without issue. This also shifts the burden of determining how to merge a leveled list, or other mod, away from the player, and away from OpenMW, and onto the creator of the mod in question.

Another nice things about this is that this would only be possible for brand new mods created with OpenMW, and thus we don't have to worry about how it will affect any existing mods. Anything with an .esp extension can just use the old method, but new mods could specify how they wanted to merge their changes.

Thoughts?
User avatar
Ravenwing
Posts: 335
Joined: 02 Jan 2016, 02:51

Re: Better handling of mod conflicts and mod merging

Post by Ravenwing »

Hmm, I haven't entirely thought all of this through, but it seems like the additive/subtractive/multiplicative system only partially solves the problem and creates new ones. I suppose it avoids catastrophic conflicts where things are completely overwritten, but it introduces lots of probably unintended behavior. I can't think of too many times where you'd want a mod to keep adding etc to a record.

What I think would solve the most problems without introducing additional weirdness is simply the finer detail records you mention. Rather than overwriting NPC X, to change his gold, we should simply overwrite his gold. If another mod adds gold, then it would overwrite. But if you want to add frost salts to his inventory, it should just add that without touching the rest. I imagine this is what Zini has in mind for the future, hence the plethora of tables introduced into the editor. The more atomic we can make edits, the less likely to introduce conflicts.

I wouldn't say there is no use for the additive/subtractive system, but it seems like a great deal of effort would have to go into this for some very niche cases.
User avatar
AnyOldName3
Posts: 2668
Joined: 26 Nov 2015, 03:25

Re: Better handling of mod conflicts and mod merging

Post by AnyOldName3 »

I've discussed changing how levelled lists work post-1.0 with Zini such that mods could just add things rather than overwrite the whole list, and he wasn't keen. I don't think we're actually planning on changing that much up.
Chris
Posts: 1625
Joined: 04 Sep 2011, 08:33

Re: Better handling of mod conflicts and mod merging

Post by Chris »

Greywander wrote: 01 Feb 2019, 05:49 As long as two mods don't modify the same record, then those mods are compatible.
This is a fundamental misunderstanding. For any useful definition of "compatible", it is possible for two mods to modify two different records, and be incompatible; that is, when used together they unbalance the game in an unintended way. If ModA decreases the price of junk items to 0, to prevent players from abusing junk to get free gold, and ModB increases the price of useful items, under the impression players can get free gold from selling junk, using the two will create a broken unbalanced game despite neither modifying the same records.
If two mods modify the same record, however, the game can't use both, and whichever one loads last is the one that gets used.
And this too. While it's true the game can't use both, sometimes you don't want it to. The entire purpose of a mod is to replace an existing record to make it do something different. If a particular value in a record is the same as vanilla while certain others are different, is it because it doesn't matter what those values are, or because they need to be like in vanilla?

In cases where you do want two changes from two mods, that's accomplished by making a third mod to overwrite the record with the two changes merged in. There's no way for the engine to know what the behavior of two distinct mods used together should be because it depends on those two particular mods; it's up to the users and modders to ensure the changes are as intended. Sometimes its easy (to the point where it could be automated), other times it's not (to the point where even people disagree on the best way to merge them, with valid arguments on all sides).
Example: If one mod gives a merchant more gold, and another mod adds new items to their stock, they are incompatible because they both modify the merchant.
I don't think that's true. To give a merchant new stock, you need to place the item in a container they own in the cell. To give them more gold, you just give them more gold on their person.
The merchant's gold and inventory are two separate fields, each of which can be marked separately as modified or not.
Pretty sure the amount of gold a merchant or NPC has is given by the number of gold_001 items in their inventory, so it's not two separate fields.

But even if it's not, there's nothing to say the mod giving the merchant new items didn't intend for them to have the same amount of gold they originally had, and giving them more could unbalance some other aspect of the mod (perhaps making it easier than intended to get the new items). Now we're starting to get into the overall intention of mods, not just the strict record modifications, and whether the two intentions can work together as-is or if it needs additional tweaking for the two changes to coexist.
Let's say, for example, that a merchant has a default gold of 500. You make a mod that gives him 1000. Rather than the mod just giving him 1000 gold, it actually just adds +500 to his default gold.
How would a modder know whether it's better to give them +500, or a flat 1000? If another mod sets the merchant's base gold to 4000, that +500 is a pittance. Or if that other mod sets the base gold to 100, that +500 is a fortune. Without knowing more about what the two mods are doing and why, adding +500 is no better than setting a flat 1000. In either case, you need to account for the other change to get the appropriate result.
User avatar
Atahualpa
Posts: 1176
Joined: 09 Feb 2016, 20:03

Re: Better handling of mod conflicts and mod merging

Post by Atahualpa »

Well, I think that many of us would agree that modifying a single entry in a record without automatically "overwriting" the whole record could be advantageous. And, yes, this could be achieved by adding a "Touch" status which marks an entry as "intentionally unmodified" - in contrast to "base" entries whose values are unimportant to the modder and can be overwritten.

Of course, we'd have to carefully think this through and likely would need to change the code quite a bit to add and handle status at entry level. - Doesn't look like a pre-1.0 feature to me.
User avatar
silentthief
Posts: 456
Joined: 18 Apr 2013, 01:20
Location: Currently traversing the Ascadian Isles

Re: Better handling of mod conflicts and mod merging

Post by silentthief »

Atahualpa wrote: 04 Feb 2019, 21:40 Well, I think that many of us would agree that modifying a single entry in a record without automatically "overwriting" the whole record could be advantageous. And, yes, this could be achieved by adding a "Touch" status which marks an entry as "intentionally unmodified" - in contrast to "base" entries whose values are unimportant to the modder and can be overwritten.

Of course, we'd have to carefully think this through and likely would need to change the code quite a bit to add and handle status at entry level. - Doesn't look like a pre-1.0 feature to me.
This has been a source of discussion for a bit lately. And even though we have had different views on how to approach the concept of mod conflict detection, Chris has a good grasp of what the issues are on this. I don't know how (or even IF) this is something that we can implement, but this would be put to post 1.0. Perhaps after we reach the 1.0 goalpost - we can get a little more input from the modding community for this issue.

ST
User avatar
AnyOldName3
Posts: 2668
Joined: 26 Nov 2015, 03:25

Re: Better handling of mod conflicts and mod merging

Post by AnyOldName3 »

Starting by talking with Mator (of Mator Smash fame) would be a good idea. He's written the best merging tool for the later Bethesda games, so should have a good idea about what can and can't be merged automatically, and what could be merged with hinting from the mod author.
claudekennilol
Posts: 92
Joined: 01 Aug 2012, 20:48

Re: Better handling of mod conflicts and mod merging

Post by claudekennilol »

I've discussed changing how levelled lists work post-1.0 with Zini such that mods could just add things rather than overwrite the whole list, and he wasn't keen. I don't think we're actually planning on changing that much up.
:(
User avatar
Greywander
Posts: 119
Joined: 04 Dec 2014, 07:01

Re: Better handling of mod conflicts and mod merging

Post by Greywander »

Ravenwing wrote: 01 Feb 2019, 17:00 Hmm, I haven't entirely thought all of this through, but it seems like the additive/subtractive/multiplicative system only partially solves the problem and creates new ones. I suppose it avoids catastrophic conflicts where things are completely overwritten, but it introduces lots of probably unintended behavior. I can't think of too many times where you'd want a mod to keep adding etc to a record.
"Unintended behavior" would be less of a problem than you might think. First, it wouldn't affect existing mods at all; only new mods that the mod author specifically and intentionally uses this feature would have it. I can see some mod authors not quite understanding what it does or how to use it, but that's less of a problem of unintended behavior and more a problem of mod authors learning to use a new tool.

As for when you'd want to keep adding to a record, I can think of a couple cases you might want to use it. If you're editing, say, the Health of an NPC then you probably don't need more than one mod doing so. Just load the mod with the HP you want for that NPC last. On the other hand, you might want to use multiple mods that each add items to a merchants inventory, so that all of those items can be for sale. Likewise, the entire reason to merge leveled lists is because you have more than one mod making changes to a leveled list.

Generally, you probably won't need mods to be additive if they're only changing a single value (like Health). It's when your changing a list of things, whether it is a leveled list or an inventory, that you'd really want things to be additive so that you could make use of all the changes made by several different mods.
But if you want to add frost salts to his inventory, it should just add that without touching the rest.
Okay, and what if I have another mod that adds fire salts or daedra hearts to his inventory? Both modify the inventory record, so even if we made things more atomic (e.g. only the inventory is marked as modified, rather than the entire merchant NPC), we can still only have one mod at a time that changes inventory.
AnyOldName3 wrote: 01 Feb 2019, 22:18 I've discussed changing how levelled lists work post-1.0 with Zini such that mods could just add things rather than overwrite the whole list, and he wasn't keen. I don't think we're actually planning on changing that much up.
I agree that we shouldn't force any changes retroactively on existing mods, as that would create a compatibility nightmare. But having more options for new mods would not be a bad thing. Even better is that it lets the mod authors decide how they want it to work, instead of having OpenMW make the decision.
Chris wrote: 02 Feb 2019, 01:28it is possible for two mods to modify two different records, and be incompatible;
Sure, but you're veering into territory that isn't really useful to this discussion. I can understand that you can have two mods that, individually, may be working as intended, but together produce a result that is unintended. I don't think OpenMW can detect that and account for it, or at least not in most cases. Ultimately, the onus is on the player to make sure they're using mods that, when they are individually working as intended, produce a result that they want for their game. My only goal here is to make sure that more mods are able to individually work as intended, instead of not working when another mod is present that modifies the same value (particularly lists such as inventories).
There's no way for the engine to know what the behavior of two distinct mods used together should be because it depends on those two particular mods; it's up to the users and modders to ensure the changes are as intended. Sometimes its easy (to the point where it could be automated), other times it's not (to the point where even people disagree on the best way to merge them, with valid arguments on all sides).
This is why I think it would be a good idea to give mod authors a tool that helps them decide how modifications from their mod should be merged with other mods that might modify the same thing. And yeah, it's hard to know how it should be merged if you're just talking in general instead of merging it with a specific mod.

Here's the thing, though: nothing actually changes, except that more mods will be compatible "out of the box". If two mods don't work correctly together, then you can still do whatever you would have done before to merge them. But now there will be more mods where you don't have to do anything because they will just work.
I don't think that's true. To give a merchant new stock, you need to place the item in a container they own in the cell. To give them more gold, you just give them more gold on their person.

Pretty sure the amount of gold a merchant or NPC has is given by the number of gold_001 items in their inventory, so it's not two separate fields.
See, this is how long it's been since I did any modding for Morrowind.
How would a modder know whether it's better to give them +500, or a flat 1000?
Maybe they don't. It's a judgement call that they would have to make, and it may take some adjusting in the modding community before they get a feel for the best way to handle things. But at least they'd be able to make that choice, and once they do figure out what the general best practices are, we're going to see a lot more mods that are fully compatible with one another that don't need any special merging procedures. For those that aren't compatible, it's business as usual and we can break out our merging tools.
AnyOldName3 wrote: 05 Feb 2019, 17:51 Starting by talking with Mator (of Mator Smash fame) would be a good idea. He's written the best merging tool for the later Bethesda games, so should have a good idea about what can and can't be merged automatically, and what could be merged with hinting from the mod author.
I'm not familiar with this person, but this sounds like an excellent idea. Having more options is good, and putting more power in the hands of mod authors is also good. I don't think we'll be able to entirely get rid of the need for some mods to be manually merged, but my hope is that we can drastically cut down on it. Not only that, but making such options available might also make manual merging easier by giving hints to the merging tool.

I'll also reiterate that I think a lot of potential issues could be accounted for by having both an "add once" and "add cumulatively" option. The "Add Once" option would be ignored if another mod added the same quantity, allowing multiple mods to make the same modification without those modifications stacking with one another. (Adding different quantities would be trickier, though, particular if one was positive and the other negative.) The "Add Cumulatively" option would allow for such modifications to be applied more than once and stack. Depending on what your mod does, you might decide that one or the other is more appropriate. In any case, this gives you more control, and helps make your mod more compatible with other mods.

Actually, the more I think about it, the more it seems like a simpler solution is to simply treat the entries on a list (such as an inventory or leveled list) as individual records, instead of treating the entire list as one record. So the number of frost salts in an NPCs inventory is one record, and the number of fire salts is a different record. The two could be edited independently from one another, and both mods would work fine together since they are modifying different records. This is basically what the "Add Once" feature would do. There might still be some use for cumulative adding, though.
Post Reply