Mod packaging - OMWPACK

Everything about development and the OpenMW source code.
darkbasic
Posts: 153
Joined: 18 Apr 2016, 15:45
Contact:

Re: Mod packaging - OMWPACK

Post by darkbasic »

psi29a wrote: 03 Feb 2021, 10:23 Could portmod devs be convinced to to roll an openmw compatible 'pack'? I mean, a zip archive is better than many loose files, or even better... since openmw supports it, lz4 archive? Plain text meta-data op front to make it easily parse-able, then the content in lz4.
portmod is all about openmw compatibility, so I guess it will follow wherever openmw goes. The only problem I see with archives is that you will loose one of the most important features: use flags. Each pybuild comes with a list of use flags which allow to enable or disable certain aspects of the mod. For example let's say that I have a mod which retextures all races and I like only the argonian ones: I could disable the use flags for all the races but the argonian one. Pybuilds don't have to have such an insane amount of customizability if it's not necessary, but if I'm building my own collection of mods and I need an existing one to be more customizable I can simply pull request its pybuild (or override it with a local version) to provide a greater degree of flexibility, all without touching the original mod. I know it may sound a bit confusing to everyone who never used Gentoo (its package manager has been the inspiration for portmod), but this is what allows the greatest degree of customizability and user friendliness.
If we're going to use archives then we will need some way to tell openmw "please discard those files inside the archive" to achieve the same effect.
Anyway you're best talking with bmw, who is the main author: I'll PM him about this topic. By the way portmod v2 is just around the corner ;)
Last edited by darkbasic on 03 Feb 2021, 13:02, edited 1 time in total.
darkbasic
Posts: 153
Joined: 18 Apr 2016, 15:45
Contact:

Re: Mod packaging - OMWPACK

Post by darkbasic »

By the way I personally think that archives are a thing of the past: it should be the job of the file system to transparently compress data.
This is the way a modern package manager handles compression while never using archives: https://www.phoronix.com/scan.php?page= ... -Fedora-34
User avatar
bmw
Posts: 81
Joined: 04 Jan 2019, 19:42
Contact:

Re: Mod packaging - OMWPACK

Post by bmw »

darkbasic wrote: 03 Feb 2021, 10:59 If we're going to use archives then we will need some way to tell openmw "please discard those files inside the archive" to achieve the same effect.
Portmod could always re-compress mods into archives at installation, allowing them to still be modified when installing. Perhaps more importantly, portmod could repackage mods which aren't using this format so that they are all installed this way.
darkbasic wrote: 03 Feb 2021, 11:14 By the way I personally think that archives are a thing of the past: it should be the job of the file system to transparently compress data.
This is the way a modern package manager handles compression while never using archives: https://www.phoronix.com/scan.php?page= ... -Fedora-34
Isn't this more about removing the costly unpacking step? They're still using archives, they're just bridging the archive and the filesystem to make installation faster. Even so, good luck getting Windows to support this sort of thing (while I recall there is a btrfs driver for Windows, I don't imagine we can rely on the presence of a btrfs filesystem).
psi29a wrote: 03 Feb 2021, 10:23 Could portmod devs be convinced to to roll an openmw compatible 'pack'? I mean, a zip archive is better than many loose files, or even better... since openmw supports it, lz4 archive? Plain text meta-data op front to make it easily parse-able, then the content in lz4.
When you say roll, do you mean design, or use an existing one? I see that openmw supports lz4 compressed bsas, but I'm not aware of any good cli-friendly tools to create them (or even uncompressed BSAs for that matter). Plus, does openmw support plugins inside of archives? With both of those things, it would be pretty easy to have portmod install each package as a single compressed BSA.
In terms of other archive formats, it would certainly be easy to create zip archives, but I'm not actually aware of any archive formats, save for the lz4-compressed bsas, that use per-file lz4 encoding. You might be stuck with something like tar.lz4, where you'd have to uncompress the whole thing to access any files, and I don't imagine that's helpful, particularly for large archives.

With all the talk of single-archive mods and supporting the configuration options many mods offer, It seems to me like two different formats are needed, one which includes installation, and one which is the installed format. For simple mods without options they could be the same, but if you want arbitrary configuration (the complex examples I usually look at being omwllf generating its plugin and project atlas generating textures) there needs to be an intermediate stage. "Context aware activation" can only go so far.

That being said, I think openmw's plugin format could go a lot further in terms of context aware activation. Admittedly I'm mostly thinking about KSP's programmable plugins here, but even with a more restricted format it might be possible to have plugins which can change their behaviour depending on which other plugins are loaded (and maybe there could be some way of linking other assets to the plugins such that the plugin can tell the engine to ignore them or not). Basically, instead of the current method of installation, where you have a bunch of different versions, or optional subdirectories, you could have that be a single plugin, and a single set of assets, which change how they are loaded based on a settings file (for purely optional options) and their environment (i.e. other plugins and their settings). Between this simple type of configuration, and mods which don't need configuration, I think it would be feasible to support a data format which could be installed without modification for virtually all mods (again, this wouldn't help for mods which generate content, but those are in the extreme minority).
Of course there's a trade-off here between the current complex installation and making loading mods more complex. There's going to be complexity somewhere, and there's certainly an argument towards doing as little as possible at runtime so that openmw loads quickly, since you'll hopefully launch openmw many more times than you'll install mods. I think this could be restricted though so that it has a limited impact on performance: e.g. if all it supports are conditional records, then everything could still be loaded in a single linear pass (unfortunately it's at least a little more complicated than that, as there would need to be some way of having records in a plugin override plugins loaded later, but a mechanism for that could even eventually remove the need for a pre-defined load order (which would also be complicated)).
User avatar
psi29a
Posts: 5355
Joined: 29 Sep 2011, 10:13
Location: Belgium
Gitlab profile: https://gitlab.com/psi29a/
Contact:

Re: Mod packaging - OMWPACK

Post by psi29a »

with deb, it is a formalised package with metadata.

Simply, there is a file structure, including the binary data itself.. with pre and post-run hooks. Building the deb package is up to package maintainers, not modders themselves, but they could be both. They describe the rules necessary to make the package. The package is then easily downloadable and structured so that it could be read on-the-fly by openmw. Anyway, I think this is over-engineering the problem.

I'd be happy with a tar ball, just one file, no compression. Just that inside the tarball is the mod itself (unpacked but pristine) with a metadata file describing the mod's contents and how it is to be loaded. KISS: keep it simple stupid
darkbasic
Posts: 153
Joined: 18 Apr 2016, 15:45
Contact:

Re: Mod packaging - OMWPACK

Post by darkbasic »

bmw wrote: 03 Feb 2021, 16:42 Isn't this more about removing the costly unpacking step? They're still using archives, they're just bridging the archive and the filesystem to make installation faster.
Not only so, but also reducing the amount of wasted space on disk and the cost of copying the data. On the server the rpms are stored uncompressed on a compressed COW filesystem (which could be btrfs, zfs, xfs or even bcachefs), the uncompressed rpm is sent through a compressed http connection to save bandwidth, then saved uncompressed in a compressed COW fs on the client. This is where the magic happens: instead of copying the files from the rpm to the desired locations (/usr, /bin, etc..) they get reflinked from the rpm. So you don't have the expensive decompression phase, you don't waste the space to store the rpm because its content is reflinked to where its files belong and you don't even have the somehow expensive phase of copying data from the uncompressed rpm to the destination path (reflinks are almost free). Plus whenever you have to update a package you could simply download the files which have been updated instead of the whole rpm or having to compute the delta archive from the server.

bmw wrote: 03 Feb 2021, 16:42Even so, good luck getting Windows to support this sort of thing (while I recall there is a btrfs driver for Windows, I don't imagine we can rely on the presence of a btrfs filesystem).
You just need a COW filesystem: Linux has at least 4 options, MacOS X has one but unfortunately there is none on Windows AFAIK, so it's not something we can really pursue unfortunately.
User avatar
bmw
Posts: 81
Joined: 04 Jan 2019, 19:42
Contact:

Re: Mod packaging - OMWPACK

Post by bmw »

psi29a wrote: 03 Feb 2021, 18:05 with deb, it is a formalised package with metadata.

Simply, there is a file structure, including the binary data itself.. with pre and post-run hooks. Building the deb package is up to package maintainers, not modders themselves, but they could be both. They describe the rules necessary to make the package. The package is then easily downloadable and structured so that it could be read on-the-fly by openmw. Anyway, I think this is over-engineering the problem.
It's also worth noting that deb files with scripts require a consistent environment to execute in (and the scripts also mean that openmw wouldn't be able to just read the archive), which is fine when you're targetting a single Linux distribution (as long as the spec is very precise), but is much harder when you want to run on an arbitrary system. Portmod already runs into this problem, and while there are solutions (see here), it's always going to be difficult to guarantee long-term stability (of course when you've got a source-based package manager, long term stability is easier, as you can update the metadata without needing to modify the original, but that's not what this is about).
psi29a wrote: 03 Feb 2021, 18:05 I'd be happy with a tar ball, just one file, no compression. Just that inside the tarball is the mod itself (unpacked but pristine) with a metadata file describing the mod's contents and how it is to be loaded. KISS: keep it simple stupid
Starting with support for uncompressed archives would probably be simplest, and support for compression can always be added later. My question then is what sort of metadata file do you want, and how complex should it be?
Are you talking about a post-configuration archive (i.e. something that could be created by portmod so that OpenMW can deal with individual files instead of directories, but does little outside of that), or an archive that can be created by mod developers and that has a way to support configuration?
In the case of the former, sure, portmod can spit out tar files, but I don't think that really tackles the issue wazabear brought up, though it would be a step in the right direction (it would provide a standard for non-configurable mods at least).
In the case of the latter, should that metadata file have the ability to use context from other mods so that it can determine which patches or features to use? If not, it doesn't seem very useful compared to the current system, as it seems to me that you'd be limited to just listing the plugins contained in the tarball, and listing optional directories/files without any context allowing them to be selected automatically. If so, then it seems very similar to what I described being potentially supported inside of plugins, and sure, having it outside of plugins could be a good alternative, but it's fundamentally the same thing (and to be clear, I'm fine with either, though I think in both cases it would be necessary to also have some sort of user configuration files to expose a way of configuring pure options).
darkbasic wrote: 03 Feb 2021, 18:43 You just need a COW filesystem: Linux has at least 4 options, MacOS X has one but unfortunately there is none on Windows AFAIK, so it's not something we can really pursue unfortunately.
It's certainly a useful idea, and even the idea of providing such a thing for the systems and derivable mods which support it is an attractive one (even just supporting delta updates would be useful, and that wouldn't depend on the filesystem), but I think it's probably too much work and requires too much custom infrastructure to make up for the savings (for us anyway; debian is a much larger and older project).
User avatar
psi29a
Posts: 5355
Joined: 29 Sep 2011, 10:13
Location: Belgium
Gitlab profile: https://gitlab.com/psi29a/
Contact:

Re: Mod packaging - OMWPACK

Post by psi29a »

My idea, a tar ball is just a collection of files in one 'ball' archive, so it would be trivial to add each ball to the VFS. So the mod itself would be unpacked (because they are usually shipped as either zip, rar, p7 or whatever) inside the tar ball.

https://en.wikipedia.org/wiki/Tar_(computing) (have a read as it goes into detail of the format, including header information)

The tarball could be later compressed, *.tar.bz2 , *.tar.gz, *.tar.xz, *.tar.p7 and even *.tar.lz (whatever works).

The meta-data would, in the form of a file, could just be simple csv or ini based thing that describes the mod so that OpenMW can run it. Are there requirements? Is there an ESM/ESP to load? Does it need Lua? Does it need something else? So if you don't have a setting turned on in OpenMW, that there could be a warning as to why a mod isn't loaded.

Feel free to brainstorm about the type of things that would be best to go in such a file.
User avatar
wazabear
Posts: 96
Joined: 13 May 2020, 19:31
Gitlab profile: https://gitlab.com/glassmancody.info

Re: Mod packaging - OMWPACK

Post by wazabear »

I do see the appeal in keeping it as simple as possible. While it doesn’t touch on some of the key problems I wanted to solve I think it would still be a huge step in the right direction with minimal changes to source.

Now, about the metadata file (which i’ll refer to config in this context). I am strongly for adding some kind of configuration. This could be as simple as an ordered list of files. Someone correct me if i’m wrong here, but I don’t believe that should add any considerable overhead when the engine loads it.

We could even use BAIN format where you have multiple directories with an integer prefix, each representing a top level Data Directory. Many folders can share a prefix, but at most one folder is chosen from each prefix. Really though, you could use any names you want for your data directories.

I believe something like this is a good middle ground here as it allows configurations to change by only editing the one file. Mod managers, for example, would never have to worry about copying or restructuring a directory, only editing this config. Another bonus to this is it would make porting existing mods which already use this format almost trivial. You could even add a blacklist for individual files, all of this adds very little complexity to the format with a huge benefit in my eyes.


TLDR: put data= content= and blacklisted-file= entries as part of the metadata/config file with paths relative to archive itself.
User avatar
bmw
Posts: 81
Joined: 04 Jan 2019, 19:42
Contact:

Re: Mod packaging - OMWPACK

Post by bmw »

wazabear wrote: 04 Feb 2021, 10:55 I believe something like this is a good middle ground here as it allows configurations to change by only editing the one file. Mod managers, for example, would never have to worry about copying or restructuring a directory, only editing this config. Another bonus to this is it would make porting existing mods which already use this format almost trivial. You could even add a blacklist for individual files, all of this adds very little complexity to the format with a huge benefit in my eyes.
You wouldn't want the configurable file to be inside the tarball though. If you did that then you would have to edit it again every time you updated a mod.
If you had an external configuration (which ideally wouldn't change significantly between releases) then you could install minor updates without having to worry about the mod's configuration. Think of it like Linux's /etc directory, which doesn't get overwritten on updates, though package managers may provide an easy way to manage changes to the default configuration files.

I'm also still in favour of having a system which can enable patches based on other available mods.
Consider, for example, the following hypothetical toml metadata file (based on this, except just the main file for simplicity):

Code: Select all

name = "rr-better-meshes"
version = "1.4.0"
description = ...
license = ...
authors = ...

[data]
"00 - Main Files" = {},
"01 - Optional - Optimized Melchior Dunmer Lanterns" = {
    patches = "dunmer-lantern-replacer"
} 
"02 - Optional - Animation Fix for Dwemer Mesh Improvement mod" = {
    patches = "dwemer-mesh-improvement"
}
"03 - Optional - Smaller 0.5 Potions" = {
    config = "smaller-potions"
}
"04 - Optional - Skooma Pipe 1" = {
    config = {skooma-pipe = 1}
}
"05 - Optional - Skooma Pipe 2" = {
    config = {skooma-pipe = 2}
}
"06 - Optional - Skooma Pipe 3" = {
    config = {skooma-pipe = 3}
}
"07 - Russian - Apparatus Fix" = {
    l10n = "ru"
}

[default-config]
smaller-potions = false
skooma-pipe = 1
A user's config file could then enable options in the following manner (and distributing a default configuration with comments inside the tarball would be recommended).
Path could be ~/.config/openmw/mods/rr-better-meshes.toml

Code: Select all

smaller-potions = true
skooma-pipe = 2
Also note the special l10n configuration type for enabling localizations automatically based on the user's locale.

The key here, is that out of 7 optional directories there's only two different settings that the user would want to change (neither of which are required to use the mod).
Any packs which declare themselves to be "dunmer-lantern-replacer" or "dwemer-mesh-improvement" would cause the patches to be enabled and loaded after those packs. While a centralized (or decentralized) naming authority would be useful, it's not necessary, and hopefully having to maintain a metadata file like this will encourage consistent naming.
User avatar
psi29a
Posts: 5355
Joined: 29 Sep 2011, 10:13
Location: Belgium
Gitlab profile: https://gitlab.com/psi29a/
Contact:

Re: Mod packaging - OMWPACK

Post by psi29a »

bmw wrote: 04 Feb 2021, 14:38 You wouldn't want the configurable file to be inside the tarball though.
Every time a mod would be changed, then it would need to be unpacked and re-tarred anyway...

Unless you move away from archive idea all together and just have a config file that points to where the mod is located, download it, extract it into a tarball with all the necessary bits in place so that it could be readily used by openmw. Kind of like how gentoo 'emerge' works... reading a recipe on how to build and install an application.
Post Reply