Mod packaging - OMWPACK

Everything about development and the OpenMW source code.
User avatar
bmw
Posts: 81
Joined: 04 Jan 2019, 19:42
Contact:

Re: Mod packaging - OMWPACK

Post by bmw »

psi29a wrote: 04 Feb 2021, 15:47 Every time a mod would be changed, then it would need to be unpacked and re-tarred anyway...
That's on the author's end though, not the user's end. What I was referring to was the user's configuration of the mod (and I'm fairly sure wazabear was too). My point is that users and mod managers should not be modifying the file inside of the tarball (short of something like portmod, which could create the tarball and its contained configuration file from a mod which didn't come with one); there should be a separation of the configuration options the mod author provides in the tarball, and the configuration that the user chooses.
User avatar
wazabear
Posts: 96
Joined: 13 May 2020, 19:31
Gitlab profile: https://gitlab.com/glassmancody.info

Re: Mod packaging - OMWPACK

Post by wazabear »

I like the idea of having a separate user mod directory with configs. The spec for such a config could always be extended in the future if OpenMW decided they wanted more advanced features, and it would always be standalone/uncompressed plain text that is easily editable.

While this physically decouples config from the mod (a downside in my book) I think it adds a lot of flexibility, and importantly leaves the original mod untouched. Another upside to this is that the omwpack compression (or lack thereof) is abstracted away. Mod manager users never have to worry about re-packing the omwpack! Packaging omwpacks should always be exclusive to the mod author or advanced users.

I still strongly advocate for a native activate function to activate omwpacks, which would initialize these user config files. It could be added in components and integrated into launcher by double clicking these packs. If it's just another component tool you could even register the activator with the omwpack extension. The activation step is absolutely necessary in my opinion, you need a way for users to easily activate manually downloaded omwpack without needing an external tool or manually creating these text files.
User avatar
psi29a
Posts: 5355
Joined: 29 Sep 2011, 10:13
Location: Belgium
Gitlab profile: https://gitlab.com/psi29a/
Contact:

Re: Mod packaging - OMWPACK

Post by psi29a »

I see one big harry problem, that the mod author uses some esoteric compression in their mod. OpenMW isn't going to support everything under the sun. Would you say it is then the user's job to run a tool that un-packs/re-pack the mod into an archive that can be read by OpenMW?
User avatar
wazabear
Posts: 96
Joined: 13 May 2020, 19:31
Gitlab profile: https://gitlab.com/glassmancody.info

Re: Mod packaging - OMWPACK

Post by wazabear »

psi29a wrote: 05 Feb 2021, 09:17 I see one big harry problem, that the mod author uses some esoteric compression in their mod. OpenMW isn't going to support everything under the sun. Would you say it is then the user's job to run a tool that un-packs/re-pack the mod into an archive that can be read by OpenMW?
.

My point was that the user editing their config wouldn’t need to worry about how that omwpack is compressed (they would if the config was part of the archive). Also, I’m under the impression that these packs will have some specification, we aren’t talking about handling some diabolical rar mod or anything :D

To reinforce what bmw wrote, mod authors would be the ones unpacking/re-tarring the vast majority of time because everything that you’d ever need for any configuration of the mod is already packed, it’s up to the config to tell what OpenMW should load.
User avatar
psi29a
Posts: 5355
Joined: 29 Sep 2011, 10:13
Location: Belgium
Gitlab profile: https://gitlab.com/psi29a/
Contact:

Re: Mod packaging - OMWPACK

Post by psi29a »

Oef... then you have a lot more faith in modders than I do. Every time I look at nexus or wherever... most just don't care, I mean look at the pitiful state of licensing. It's only slightly better than it was, but most modders just don't care about that stuff so why expect them to do any more work than necessary when they can just make a zip/arc/rar/p7 and tell you the user to unpack it yourself.
User avatar
AnyOldName3
Posts: 2666
Joined: 26 Nov 2015, 03:25

Re: Mod packaging - OMWPACK

Post by AnyOldName3 »

Beside some of the technical craziness, like having several compressed streams held as two files each inside another compressed file that would typically be compressed a third time for distribution, OMOD is an interesting case study here. While it's not the case anymore, there used to be plenty of OMOD-exclusive mods, even when they were just a BSA and ESP, with the justification being that mod users kept messing up extracting a few files to a specific location. Modders were willing to use slightly more complicated tooling to ensure that if a user reported that their mod wasn't working, it wasn't that they'd installed it wrong. I've definitely encountered scripted OMODs that clearly didn't do what the author intended, and scripted mod installers are (correctly) never remembered kindly as they're not hugely safe and are really hard to add support for to new mod managers, so we shouldn't copy OMOD, but it wasn't hated when it was the state of the art.
User avatar
bmw
Posts: 81
Joined: 04 Jan 2019, 19:42
Contact:

Re: Mod packaging - OMWPACK

Post by bmw »

psi29a wrote: 05 Feb 2021, 11:37 Oef... then you have a lot more faith in modders than I do. Every time I look at nexus or wherever... most just don't care, I mean look at the pitiful state of licensing. It's only slightly better than it was, but most modders just don't care about that stuff so why expect them to do any more work than necessary when they can just make a zip/arc/rar/p7 and tell you the user to unpack it yourself.
That's why my thought was to make it useful to modders too by including configuration options that can be set up by the modders. If modders have a reason to use it, then they might actually do so. Some modders already use the BAIN format (a weak spec, but it's better than nothing), and making this be essentially an extension of that could encourage adoption.
The example AnyOldName3 gives with OMOD I think is evidence that if we provide a spec which provides consistency then modders will probably make use of it, and not requiring a scripting environment may make it survive longer.

But equally so, not everyone will (particularly not for mods which were abandoned a decade ago), hence why I think it's unlikely that this will replace mod/package managers entirely, but it would be a powerful tool that could make everyone's lives a little easier in the long run.

Configuration options would also be great from the perspective of mod/package managers, as it would mean that the package manager only has to worry about configuration that's relevant at install time, and most of it could become runtime configuration.
psi29a wrote: 05 Feb 2021, 09:17 I see one big harry problem, that the mod author uses some esoteric compression in their mod. OpenMW isn't going to support everything under the sun. Would you say it is then the user's job to run a tool that un-packs/re-pack the mod into an archive that can be read by OpenMW?
Arbitrary compression doesn't have to be a problem though, as I think the idea would be that you still might have to uncompress the tarball before installing it (the high-compression used for distribution is not really something I would think openmw would want to deal with at runtime). But it would be the only thing you need to do when installing most mods.
wazabear wrote: 04 Feb 2021, 19:24 I still strongly advocate for a native activate function to activate omwpacks, which would initialize these user config files. It could be added in components and integrated into launcher by double clicking these packs. If it's just another component tool you could even register the activator with the omwpack extension. The activation step is absolutely necessary in my opinion, you need a way for users to easily activate manually downloaded omwpack without needing an external tool or manually creating these text files.
The trouble with handling config files is that it's not quite as simple as copying the new default config into the config directory. I feel that if you want a GUI tool, you'd need to support everything through the GUI, which would mean some sort of invocation of a graphical merge tool like kdiff3 so that you can handle changes to the default configuration file template in a way that won't overwrite the user's modifications to their own file, adding more complexity that doesn't strictly need to be in openmw itself. I feel like this is more the domain of an external tool, and, especially since with a default configuration built-in modifying config files would be optional, I don't think it's unreasonable to expect people to use one if they're installing many mods which would be difficult to deal with manually (though I may be biased).
User avatar
wazabear
Posts: 96
Joined: 13 May 2020, 19:31
Gitlab profile: https://gitlab.com/glassmancody.info

Re: Mod packaging - OMWPACK

Post by wazabear »

bmw wrote: 05 Feb 2021, 15:49 The trouble with handling config files is that it's not quite as simple as copying the new default config into the config directory. I feel that if you want a GUI tool, you'd need to support everything through the GUI, which would mean some sort of invocation of a graphical merge tool like kdiff3 so that you can handle changes to the default configuration file template in a way that won't overwrite the user's modifications to their own file, adding more complexity that doesn't strictly need to be in openmw itself.
By activate I mean dead simple overwrite everything type of activate with a file overwrite warning when applicable. If all the packs follow an extended BAIN format (as an example), you'd simply have to check a few boxes for what data folders and ESPs you wanted to use. This is only the case if the OMWPACK format has a strict structure. If the structure is dependent on a config then yeah, this would complicate things.

As for modders not following the spec for these packs, this is certainly true and happens every so often. As others mentioned, we see that in the past these horrendous formats such as FOMOD worked most of the time, mainly because they provided something where there was nothing. If the omwpack is simple and familiar (again like BAIN) I don't see an issue here, especially if the format doesn't include any scripting. Sure there will be some mods with errors, but I like to believe most modders will be willing to fix their broken deliverables.

Some people failing to create compatible packs and then refusing to fix them later when the inevitable users come with the reports is something we shouldn't even think about honestly. They are surely in the minority.
User avatar
AnyOldName3
Posts: 2666
Joined: 26 Nov 2015, 03:25

Re: Mod packaging - OMWPACK

Post by AnyOldName3 »

An XML FOMOD with a wizard installer is much nicer than a BAIN with a wizard installer. There are a few issues where it's off-spec as the spec was written for Fallout Mod Manager and then the canonical implementation people targeted became Nexus Mod Manager, so Mod Organizer, Vortex and Kortex all have to copy some of NMM's spec misinterpretations, but the scripting language for BAIN wizards is nightmarish.
User avatar
bmw
Posts: 81
Joined: 04 Jan 2019, 19:42
Contact:

Re: Mod packaging - OMWPACK

Post by bmw »

wazabear wrote: 05 Feb 2021, 18:54 By activate I mean dead simple overwrite everything type of activate with a file overwrite warning when applicable. If all the packs follow an extended BAIN format (as an example), you'd simply have to check a few boxes for what data folders and ESPs you wanted to use. This is only the case if the OMWPACK format has a strict structure. If the structure is dependent on a config then yeah, this would complicate things.
Are you talking about "activate" to install the mod? I thought we were talking about packs which would be installed as-is (again, you still might have to uncompress them first) and have subcomponents loaded conditionally. More likely we're discussing both and everyone's using language that's ambiguous enough that it's not always clear what is being referred to.

An overview of the two options as I see them at the moment:

Installer Spec:
  • Provides a standard for how mods present themselves to a mod/package manager, or a standalone installer (which could be included in openmw). A GUI could let users manually choose components to install, or a CLI tool like portmod could use its own configuration to provide default options and remember user choices.
  • Very little change would be necessary on the OpenMW side, other than if an installer/activator is to be included. There would be no change to how mods are loaded at runtime.
  • Means that you have to interact with the installer every time you install/update a mod. For CLI package managers this isn't a huge problem, as it can be done at the same time as downloading and unpacking it, but if you're doing things manually you'd need to click through the GUI every single time.
  • Context-aware installation is possible, but is significantly more complex since you won't be able to tell which mods haven't been installed yet (one of the reasons why package managers exist).
  • Spec is hard to enforce, as there might be many different installer tools which might implement it differently (like FOMOD)
  • Easily could be used as a spec for mods designed for the vanilla Morrowind engine (or other games).
Loader Spec:
  • Provides a standard for how installed mods are loaded by openmw (and can equally serve as the distribution format, as long as it's flexible enough).
  • If done correctly the speed penalty would be trivially small (e.g. for context-aware loading it just means that you need to load all the mod metadata before starting to load mod contents so that you know which other mods are being used).
  • OpenMW would need to support the new interface for loading mods
  • Means that installing mods is as simple as copying the pack to wherever you usually keep them and add it to the list in openmw.cfg. Because of default settings and context-aware loading, no configuration is required unless you want non-default features of a mod.
  • Context-aware loading means that patches are enabled based on the mods used at runtime, not when the mod was installed. As such you could run openmw using an arbitrary subset of your installed mods without compatibility problems (assuming that, if some patches are distributed separately, you always select all such patches, since they'll only be loaded if they're needed anyway).
  • Spec would be easy to enforce, as OpenMW is the de facto arbiter of how loading works.
  • Ports for the vanilla engine would be more difficult, given that this needs a VFS to work. Unpacking the tarballs plus a tool to set up an overlayfs could work, but I have no idea how feasible it would be on Windows. Sadly this might impact adoption, as without a solution for the original morrowind engine, authors may not want to use the format. In a pinch, an installer could be written to turn this type of package into something the original engine can use, but you would lose many of the advantages of this format.
Preferences? I still like the loader spec, though I'll admit that adoption of such a spec would be much easier if it would work equally well with both OpenMW and the original engine.
A spec that works as both a Loader spec and an Installer spec (mainly for supporting the original engine) is also a possibility, but context-aware installation is more complex than most of the mod managers that exist for morrowind at the moment can handle (most are basically just install wizards with a vfs, and have no way of automatically re-installing previously installed mods to fix new compatibility issues). Admittedly they could still use such a spec fairly easily, they just couldn't take full advantage of it without significant changes. Plus half-hearted attempts to get existing mod managers to install mods using the spec could cause issues if the spec is not being rigidly followed (noting that if mod managers don't fully support it then it's more likely that mod authors won't take full advantage of it in their mods).
Post Reply