Config/Launcher discussion (or: One guy wants some things)

[Zini]

Feels like we had this discussion at least hundred times; with the same arguments brought forth every time and no one convincing anyone. Sorry, I do not see this as a productive way to spend my time, especially since this issue/feature is still a good way off from being tackled (post 1-0) and we are going to have another discussion then anyway.

[Zini]

Okay, let's give it another bash. I am kinda stuck at the stuff I am currently working on and could use something else to take my mind of it.

It seems most of the discord comes from us looking at the problem from very different premises. Most of the people arguing here seem to want to enhance the current system (mod manager, data dirs and whatnot), while I am going for a completely different approach, because I consider the system inherited from MW as not salvageable.

Here are my two design criteria:

1. There should be only one method for the end user (gamer) to select content. The end user has a list of content, where each item has a checkbox besides it. The user checks or unchecks the checkbox and that is the end of it.
For the time being the user may also rearrange the order of content. In the very long run we should try to to minimise the need for reordering by improving our algorithms for finding a default order. But that is besides the point now.
My point is that fiddling around with data directories or resources archives has zero value for the player. He should not have to deal with this stuff. Anything we currently can do with data dirs or manual selection of archives, can also done with this single unified mechanism (provided we design it correctly).

2. The software needs to know for each resources file which content file the resources belong to. This is essential the same as #1, but from a different perspective. For the sake of simplicity I strongly suggest that each resources file is associated with exactly one content file.
The association between content file and resources file is necessary for many possible future enhancement to OpenMW-CS (automated packaging, possible improvements to collaborative content creation and so on).

The archive approach that I will present after we reach 1.0 is the easiest and most backward compatible way to realise these aspects that I have been able to come up with.

There are probably alternative approaches I haven't thought of and we should consider them when the time comes.

One alternative I could think of is to have the association done via the directory system. That would mean a per-content file directory that contains both the content file itself and all resources files. These directories would then need to contain exactly one content file.
We would then need a way to distinguish these directories from other file locations that do not fulfil these requirements. Here I see two options:
a) a specially marked data directory (file name or something). Not convinced that this is a good idea, because the data directory system really wasn't made for this task and we could definitely not keep the current cfg mechanism in its current form
b) a subdirectory within a data directory (i.e. data/somemod/). Again this directory would need to be marked in some way to distinguish it from regular subdirectories.
Personally I like the archive approach better, because it will be less work and fit more smoothly in what we already have. But maybe there are arguments to be made for the alternatives.

[AnyOldName3]

The plugin=archive solution inherits all of the problems of Skyrim's Steam Workshop, which a huge proportion of Skyrim modders completely ignore as it causes a lot of problems. Not only is it harder to control different optional components if an individual mod, it's also harder to create compatibility patches between mods, or have tools like SkyRe ReProccer or BodySlide meddle with stuff, as instead of just loading stuff from and writing to the file system, they have to go via an archive format, and a gazillion other examples.

[Chris]

1. There should be only one method for the end user (gamer) to select content. The end user has a list of content, where each item has a checkbox besides it. The user checks or unchecks the checkbox and that is the end of it.

This sounds like the way Skyrim does it with the Steam Workshop. You "subscribe" to (download) a mod, which comes as an esp/bsa pair, and it shows the esp in the launcher. You check it (automatically done when you click Subscribe on the mod page), and it it's enabled with any provided resources. To make a mod, you use the Creation Kit to package the resources with the active content file, then publish/upload it to Steam from within the CK.

The problem is, while that's "good enough" for simple esp mods with some resources, it becomes a problem when mods want to do something a bit more advanced, such as sharing resources with other mods, or providing install-time options, or prioritizing resources separately from content files, or using the resources from a mod but not its content files. All of which are things that do and will always happen due to the nature of modding, and is the reason why Nexus Mod Manager and Mod Organizer are still the most popular tools for people serious about modding.

2. The software needs to know for each resources file which content file the resources belong to. This is essential the same as #1, but from a different perspective. For the sake of simplicity I strongly suggest that each resources file is associated with exactly one content file.
The association between content file and resources file is necessary for many possible future enhancement to OpenMW-CS (automated packaging, possible improvements to collaborative content creation and so on).

Not all resources are associated with a content file. Many texture, model, and sound replacers simply provide new texture, model, or sound files to override the existing ones, and have no need for a content file. Requiring one is just extra work for no gain. They can also be packaged with standard tools like 7zip or winzip that they're used to, rather than having to use the new tools we give them to use.

Some resources may be used by more than one content file, too. Morrowind Comes Alive, for example, uses
the head and hair models from another mod. That other mod, when enabled, replaces the head and hairs of other NPCs. However, MCA doesn't need the head and hair mod enabled (which could change NPCs in an undesirable fashion), it only needs the models accessible for its own NPCs. This would require MCA to be distributed with copies of those models (which may not be allowed if the person who made them won't allow redistribution in other mods, and would also be wasteful by having multiple copies of the same thing).


I think the issue here is that there's a stark difference between game creation, such as the Example Suite or a total conversion, and every day modding. Game creation is going to want a simple, straight forward pipeline for producing and publishing content quickly and efficiently. Every day modding is going to want everything possible at its disposal to achieve a desired result, no matter how dirty it gets and especially if such a result isn't (supposed to be) possible.

[Zini]

The problem is, while that's "good enough" for simple esp mods with some resources, it becomes a problem when mods want to do something a bit more advanced

This is where we disagree.

such as sharing resources with other mods

Simple. Add an empty content file. Associate the resources with it. Then make that content file a dependency of the content file that wants to use the resources (think of Linux package management)

providing install-time options, or prioritizing resources separately from content file

Don't see where this would be affected.

using the resources from a mod but not its content files

If you only want to use it for yourself, you can just copy the resources. If you want to release to the public, you need permission from the author anyway, which would also allow copying or maybe you can convince him to factor out the resources as a separate resources pack for common use (again with an empty content file).
If we absolutely have to, we can extend the dependency record by a flag that specifies that only the resources of the dependency should be used and not the content file itself.

Not all resources are associated with a content file. Many texture, model, and sound replacers simply provide new texture, model, or sound files to override the existing ones, and have no need for a content file. Requiring one is just extra work for no gain.

I disagree strongly with this. The proposed approach will be less work, not more. An empty content file has no opportunity cost.

I think the issue here is that there's a stark difference between game creation, such as the Example Suite or a total conversion, and every day modding. Game creation is going to want a simple, straight forward pipeline for producing and publishing content quickly and efficiently. Every day modding is going to want everything possible at its disposal to achieve a desired result, no matter how dirty it gets and especially if such a result isn't (supposed to be) possible.

Yes that is the problem. Regular modding on proprietary games means hacking it together with whatever works, because the means to do it cleanly often simply do not exist. But we are dealing here with an open source game that will continue to evolve and adept to the needs of the user. Under these circumstances the mindset of getting it to work no matter how dirty is harmful IMHO and I think we should try to get away from it.

[Chris]

such as sharing resources with other mods

Simple. Add an empty content file. Associate the resources with it. Then make that content file a dependency of the content file that wants to use the resources (think of Linux package management)

Doesn't that break the "each resources file is associated with exactly one content file" rule, if two content files can be associated with one resources archive?

providing install-time options, or prioritizing resources separately from content file

Don't see where this would be affected.

If the launcher only enables content files, which implicitly enable a resource archive, how could you prioritize a resource archive separately from its content file? And since the resources archive is already there and enabled with the content file, how can you change what resources are installed, or which content files are installed? For example, to provide compatibility patches for mods you may have (leaving out the ones that you don't), or to use a different variation of texture.

If you only want to use it for yourself, you can just copy the resources. If you want to release to the public, you need permission from the author anyway, which would also allow copying or maybe you can convince him to factor out the resources as a separate resources pack for common use (again with an empty content file).
If we absolutely have to, we can extend the dependency record by a flag that specifies that only the resources of the dependency should be used and not the content file itself.

This sounds like trying to come up with a workaround to a self-imposed problem.

You need permission to distribute copies of someone else's models, which you won't always be able to get. The shutdown of Planet Elder Scrolls some years back really highlighted how many Morrowind mods there are that people still use, whose authors are unreachable (to the point where some sites took a 'mirror first, ask later' approach to saving mods from being forever lost). To say nothing of authors who simply won't give permission.

As long as a user can get a mod, you don't need permission for your mod to merely utilize its resources if they're already there (especially if people can make compatible alternatives). But requiring such resources to be distributed with your mod imposes an unnecessary burden.

Not all resources are associated with a content file. Many texture, model, and sound replacers simply provide new texture, model, or sound files to override the existing ones, and have no need for a content file. Requiring one is just extra work for no gain.

I disagree strongly with this. The proposed approach will be less work, not more. An empty content file has no opportunity cost.

Right-click on the folder(s) and pick "Compress->As ZIP Archive", done. User just has to download it to the mods folder and enable the mod.

Compared to opening OpenMW-CS, creating a new mod, adding the resources, saving the mod, then compressing the resources archive and the content file into another archive. Then the user has to download that archive, extract the resources archive and content file to wherever the launcher wants them, then enable the content file.

That's not an insignificant amount of extra work. The latter is effectively what Skyrim does, and Nexus Mod Manager and Mod Organizer go out of their way to avoid requiring it. It's wasteful (extra files that aren't otherwise needed), and confusing (can you be sure that content file isn't going to do anything to the game? users generally know that simple resource replacers will just replace the resource and not have an effect on gameplay behavior, but if simple resource replacers require a content file that could have an effect on gameplay behavior, then you're creating unnecessary doubt).

Yes that is the problem. Regular modding on proprietary games means hacking it together with whatever works, because the means to do it cleanly often simply do not exist. But we are dealing here with an open source game that will continue to evolve and adept to the needs of the user. Under these circumstances the mindset of getting it to work no matter how dirty is harmful IMHO and I think we should try to get away from it.

Right, which is why giving modders more power in how their mods are set up will help. The best way to get modders to avoid dirty hacks is to not impose unnecessary limitations on them to hack around. Having an "elegant" modding system in place is all well and good, but if that's not how modders want to do it, they're going to make dirty hacks to work around it.

[Zini]

Doesn't that break the "each resources file is associated with exactly one content file" rule, if two content files can be associated with one resources archive?

Maybe I didn't make that clear enough. The resource would still be associated with exactly one content file.

If the launcher only enables content files, which implicitly enable a resource archive, how could you prioritize a resource archive separately from its content file? And since the resources archive is already there and enabled with the content file, how can you change what resources are installed, or which content files are installed? For example, to provide compatibility patches for mods you may have (leaving out the ones that you don't), or to use a different variation of texture.

Okay, now I get it. The case of giving content and resources different priority is indeed not covered. But sounds like an edge case that probably can be worked around.
Everything else should be easily doable by utilising the available tools. For example a texture variation could go into a separate content file/resource pack that would then be loaded with higher priority.

This sounds like trying to come up with a workaround to a self-imposed problem.

No. I am trying to come up with workarounds for bad working practices that shouldn't exist in the first place (in a perfect world were MW wasn't limited by its proprietary status and where modding was done in a bit less makeshift fashion). And I hope to convince people to abandon them by given them better tools.

Compared to opening OpenMW-CS, creating a new mod, adding the resources, saving the mod, then compressing the resources archive and the content file into another archive. Then the user has to download that archive, extract the resources archive and content file to wherever the launcher wants them, then enable the content file.

That is not how I would imaging the workflow. Opening OpenMW-CS and creating a mod, yes. 3 clicks and entering a name. Implementing an easy to use import UI for resources should be easy. And everything else would then be done by the editor with basically one more click.

The best way to get modders to avoid dirty hacks is to not impose unnecessary limitations on them to hack around.

I think the current state of modding proves you wrong there.

[vakhoir]

Hi!

I know the topic is old and the issue is post-1.0, but I got frustrated with the current way to mange mods, and one way or the other I'm building something to mange it a little better for myself. So I figure as long as I'm doing that, I can at least try to build something that will be of use to someone later down the line (I just started, but I currently added a checkable data folder list that can be managed like the content files, if anyone's interested: https://gitlab.com/vakhoir/openmw/tree/ ... afolder-ui).

I initially found Zini's idea counter-intuitive, but I think I'm turning around. The only thing I'm wondering about is backwards compatibility. There seems to be quite a few resource only mods out there, it seems like if nothing else, we'd need some kind of import functionality to generate an archive+content file?

[Sagacity]

I think the best approach would be to provide the most useful flexibility to the user without making the system unnecessarily complicated. Mod Organizer and Vortex take similar, but slightly different approaches, that I believe solve the problem without being too complex for an average user to figure out; especially given a more open engine and more closely integrated manager/engine pair.

Currently I think the issue we have is that there's two competing paradigms here, that aren't necessarily mutually exclusive in my opinion.

The first paradigm, that Zini is expressing from what I can tell, is a one entry one package concept. Which makes sense. It is far simpler for one mod to centralize all of it's content into a single package, and minimizes issues associated with multiple archives/content packages coming from the same source. It also minimizes confusion when a given mod package has multiple content packages that are all needed.

This paradigm however, as with most things, has significant drawbacks. The first, and most obvious is it's inflexibility. While it would be nice if everything could all be associated with one package; this creates problems with more modular mods, and even more problems if you're making patches. Furthermore, it creates unnecessary redundancy, and theoretically prevents mixed priorities for different mods (IE, I can make asset A prioritize content package X from mod 1, but have asset B prioritize content package Z from mod 2 with Mod Organizer/Vortex, regardless of their order in relation to each other.)

The second paradigm, is one of many entries, many packages. This makes sense as well since there's no particular reason not to allow many entries or many packages, other than simplicity. It shouldn't be technically difficult, or anything like that, and it doesn't increase system strain or load in any way. Furthermore, it allows far more flexibility, and ditches any and all drawbacks of the previous paradigm. It's key issue is that it's more complex.

Again however, there are other significant drawbacks. In order for it to be more useful, it would require a more complex interface to better facilitate management of these packages and entries. Furthermore, as entry and package count increases, it becomes harder and harder to manage, making tools like Loot and Bash necessary. Similarly, things like FMOD seek to make this paradigm feasible in practice.

The key point to make here though is that these two paradigms are not mutually exclusive!

Things like FMOD and mod packages allow the second paradigm to masquerade as the first. Everything is packaged into a single archive, which is then unpacked by the mod manager and is then run somewhat like an installer, and is used to guide the user through the configuration process. This removes most of the complexity from the second paradigm, while retain it's flexibility, and even allows mod makers to explain why a user may want to disable/enable certain parts of their mod. Similarly, it facilitates patches to be packaged in with main mods.

I propose OpenMW either adopt FMOD, or make an FMOD alternative, and take Chris/AnyOldName's approach under the hood. Users that do not wish to do complex modding, and want an easily understood modding framework can have it. But those who want a more complex system can have it as well.

There's no reason, far as I can tell, to artificially restrict the flexibility of the system. There's no reason complexity should be an issue; especially since OpenMW's goal is to be an engine replacement for Morrowind; not a user interface replacement for it's launcher. More flexibility is always better; even if it is hard to understand for the end-user as far as an engine is concerned. Also remember that as a mod list increases in size, neither approach is particularly more difficult to understand. Good documentation and interfaces can also make both approaches equally feasible for end-user usage.

[vakhoir]

Originally I had the same sort of idea as you did, but after a few days of using my (admittedly incomplete) solution, I can already tell you that managing mods where content is separated from resources can get tedious. It's awesome when you're modding, and maybe you're thinking "I'll use this bit from this folder, and that bit from the other folder", but for the end user you really want to check just one box and not think if you added all the relevant data folders, and ordered them correctly.

The mixed solution might be good for the CS, but the resources should get "squashed" to an archive on save, I think,