Mod packaging - OMWPACK

Everything about development and the OpenMW source code.
User avatar
wazabear
Posts: 96
Joined: 13 May 2020, 19:31
Gitlab profile: https://gitlab.com/glassmancody.info

Mod packaging - OMWPACK

Post by wazabear »

Opening up this thread to aggregate everyone's thoughts on the matter instead of being scattered all over discord.

Inspiration

I've been thinking about the next big thing OpenMW can bring to the table, there is a lot! One feature in particular that I think would be immensely powerful is a dedicated format for mod packaging. This idea has been brought up in many a casual conversation but the talk never seems to have any direction or intent. A standardized mod packaging system is something everyone wants, yet I don't see a plan for it in the foreseeable future, if anything it can't hurt to discuss possible implementations. From now on I will reference such a package as an omwpack, from this doc.

A mod package needs to be both friendly to modders and friendly to the people installing the mods. The current system in place is friendly to neither if only for the fact that there is zero standardization.

Overview

Essentially, I envision mods to be standalone files, think BSAs on steroids. Every mod is its own omwpack and is loaded in the standard OpenMW way. These would of course need some precedence over loose files such as BSAs, plugins, and data directories but this is a minor issue at best.

Code: Select all

mod=rainbows.omwpack

Core Features

The following are some features which I think would be absolutely necessary to have for any initial design. The list is short, but I think I touch on the core aspects of such a system.
  • Context aware activation. When activating an omwpack it should be fully aware of the users load order. Here would be the place to spit warnings such as overwriting existing files (potentially from other omwpack), errors regarding missing dependencies, etc.
  • Configurable installation. Mods have options, so there needs to be a way to provide options. The option to blacklist arbitrary files is an important feature as well. With pluginless replacers especially it is sometimes desirable to not want to use a handful of specific files. This is not so simple as selecting from a checklist as options are conditional on previous choices (see GiTD). Thus, a wizard of some kind is mandatory (GUI, CLI, or both). I believe this part might be a bit controversial as you'd need some manner of writing these conditionals, presumably in a standardized script (potentially use existing langauge with restricted API). There are a few formats such as OBMM and FOMOD which allow wizard-like installations through programs such as Wrye and MO2.
  • Stateful Installation and Re/Un-installation. The side-effect of a configurable installation is needing to track extra state to know how to either reinstall or uninstall the mod package. I don't think it's a good idea to install contents of omwpacks somewhere else. With that method it means you need to keep a copy of the original omwpack around if you want to reinstall. Reinstalling is incredibly useful when trying out new options or needing to check a new compatibility patch.
You may note the absence of automatic load ordering, this is intentional. While I think it's very important to be able to auto-sort your load order to some degree (to deal with the most obvious culprits), I think it should be left to a separate tool. Whether this is just another app in the components directory or an "auto-sort" button on the launcher is not really relevant. Tools such as MLOX and LOOT attempt to do this with varying degrees of success. Additionally, manual ordering of omwpacks which contain only loose files is a mandatory feature as it is purely up to the user to decide which files they would want to overwrite (there is no wrong order).

Example Structure

Putting everything together I'd imagine an omwpack to be a simple archive with the following structure:
  • meta.cfg : Version, author, contact info, etc...
  • wizard.script : Install script of some sort, supporting conditional options
  • data/ : Plugins and standard loose files such as sounds, meshes, textures, lua, etc...
  • .install : May or may not be part of archive, determines how OpenMW loads the omwpack on game launch. Following this structure, it would be generated from the wizard.script when run. Necessary for uninstall or reinstall.

You'd expect to be able to do the packaging entirely in OpenMW-CS. That is, boot up a new project and have everything you need in an isolated environment for packaging. With the current state of OpenMW-CS I do not see this as an option, at least for now. There is no reason, however, that we cannot go ahead and have our own package structures as integration can always be added. I may be off here but I don't believe OpenMW-CS should ever be an essential ingredient for the creation of an omwpack.

Mod Repository

I see this as an end-goal, opening up the OpenMW mod store browser and clicking the install button on all your favorite mods. While this is something that might happen far into the future it will be much simpler to do if we already have a mature packaging system.


A note on compatibility: I think the great thing here is that we don't have to worry at all about backwards comparability, the standard way of loading mods is here to stay for foreseeable future and adding a new packaging system will not break this.
User avatar
psi29a
Posts: 5355
Joined: 29 Sep 2011, 10:13
Location: Belgium
Gitlab profile: https://gitlab.com/psi29a/
Contact:

Re: Mod packaging - OMWPACK

Post by psi29a »

Zini had the idea that it would somehow be possible in the future to use a package manager like interface (cli and/or gui) to install mods.

While not core to OpenMW itself, it could be another program that handled this for you. Similiar to 'apt' in Debian/Ubuntu or 'pacman', 'emerge' from gentoo or even 'brew' from MacOS.

I thought perhaps people would write "recipes" for mods. Some could 'just work' out of the box, others might have additional configuration steps, so you could script it those steps on top of the mod.

The problem, I see and others see), is then OpenMW then becomes known for also being 'the' mod manager... another area where people are sharply critical.

If someone outside of OpenMW wanted to do this, why the hell not? Python or Go... get 'er done, show a PoC, get attention and contributors.

Keep in mind though, OpenMW does not want to deal in content hosting that isn't OUR content. So we will not host anything.
User avatar
wazabear
Posts: 96
Joined: 13 May 2020, 19:31
Gitlab profile: https://gitlab.com/glassmancody.info

Re: Mod packaging - OMWPACK

Post by wazabear »

Hi, thanks for the reply!

So sounds like the plan is to never have support for self contained mods then?

Creating a new recipe type and 3rd party format that will later just be transformed to OpenMW’s way of handling mods(multi-data paths) would very much be reinventing the wheel. Install scripts and wizards already exist, and despite their ugliness, are still widely used and supported on popular mod managers.

I remember you saying that Zini envisioned the VFS as just a stopgap and mods would eventually be loaded via this new archive type. In my opinion, such a format would be quite useless if it’s just an archive of a data folder as it adds no benefits over standard data= lines.
User avatar
AnyOldName3
Posts: 2666
Joined: 26 Nov 2015, 03:25

Re: Mod packaging - OMWPACK

Post by AnyOldName3 »

I recommend reading the post-1.0 document Zini wrote as, while we've abandoned much of it, it does have some valid thoughts about mod packaging. A few things that are interesting to think about include:
  • There's not much justification for an ESP and a BSA to be separate files, or at least there's not much justification for putting an ESP into the BSA being banned. It's easier for users if one file = one mod.
  • It would be nice if there was a better way to express dependencies between mods.
  • Things go faster with content-aware archives. For example, instead of effectively being a bunch of DDS files in a zip, Fallout 4's texture BA2 contains data equivalent to the DDS header and some offsets into the actual texture data. That means you can load a particular mip level really quickly, which helps a lot if you're limited by VRAM or disk speed and want to page things in and out regularly.
  • Particularly on Windows, seeking within a big file is much cheaper than opening individual loose files, so when you're actually playing, loose files are best avoided, and it would be best to have the package format be the big file rather than unzip it to lots of little ones.
Fair warning - I've not read the first post because I have an ear infection, and that's more text than I'm willing to deal with.
mistermoonshine
Posts: 30
Joined: 03 Apr 2018, 20:35
Location: Ireland

Re: Mod packaging - OMWPACK

Post by mistermoonshine »

Mod Organizer 2 already does basically all of that. No need to reinvent the wheel. It could have better OpenMW integration though.
User avatar
wazabear
Posts: 96
Joined: 13 May 2020, 19:31
Gitlab profile: https://gitlab.com/glassmancody.info

Re: Mod packaging - OMWPACK

Post by wazabear »

mistermoonshine wrote: 03 Feb 2021, 01:08 Mod Organizer 2 already does basically all of that. No need to reinvent the wheel. It could have better OpenMW integration though.
MO2 is indeed incredibly useful, it's what I am using right now via wine as I think it's the best we got right now. It is not really relevant here though, same with Portmod. Any and all such solutions are essentially workarounds to a convoluted modding system that BGS titles follow. OpenMW has the privilege of being able to use it's own format, and if it's remotely robust enough it would make mod managers redundant. This theoretical omwpack wouldn't turn OpenMW into a mod manager, it would just make managing mods so simply that you wouldn't need one.

I've been modding BGS titles for over a decade and I can say with confidence there way is pretty terrible.
darkbasic
Posts: 153
Joined: 18 Apr 2016, 15:45
Contact:

Re: Mod packaging - OMWPACK

Post by darkbasic »

mistermoonshine wrote: 03 Feb 2021, 01:08 Mod Organizer 2 already does basically all of that. No need to reinvent the wheel. It could have better OpenMW integration though.
Where is my Linux ppc64le binary? I run OpenMW on a Power 9 machine.
darkbasic
Posts: 153
Joined: 18 Apr 2016, 15:45
Contact:

Re: Mod packaging - OMWPACK

Post by darkbasic »

wazabear wrote: 03 Feb 2021, 01:39This theoretical omwpack wouldn't turn OpenMW into a mod manager, it would just make managing mods so simply that you wouldn't need one.
It won't be that simple, the main reason being that you can't expect mod authors to provide exhaustive dependencies, provide compatibility patches for every single mod, etc. The fact that many of them don't even grant you distribution permissions means that if they abandon them you can't even fix the metadata. If mod authors did their job correctly the modding would already be way easier. portmod already fixes all of this, plus it allows you to install collections of hundreds of mods with a single command. Of course a better format would be welcome, but I think that data and metadata must stay separate because you simply can't rely on mod authors, plus you still need the ability to apply external patches. In the end it will make writing portmod pybuilds a tiny little easier, but nothing more.
User avatar
psi29a
Posts: 5355
Joined: 29 Sep 2011, 10:13
Location: Belgium
Gitlab profile: https://gitlab.com/psi29a/
Contact:

Re: Mod packaging - OMWPACK

Post by psi29a »

Could portmod devs be convinced to to roll an openmw compatible 'pack'? I mean, a zip archive is better than many loose files, or even better... since openmw supports it, lz4 archive? Plain text meta-data op front to make it easily parse-able, then the content in lz4.

I'm confident that if this (or equivalent) becomes the defacto way of doing things that it won't be long before it's hacked into Morrowind.exe via a 3rd party tool because it would be just too useful.
Post Reply