Portmod - Mod Manager

Learn about OpenMW mod compatibility, check for problematic mods, discuss issues, and give us feedback about your experience with modded OpenMW.
User avatar
bmw
Posts: 18
Joined: 04 Jan 2019, 19:42
Contact:

Portmod - Mod Manager

Post by bmw » 09 Apr 2019, 04:10

Portmod
https://gitlab.com/portmod/portmod

Previously known as OpenMMM v2.0, Portmod is an in-development command line mod manager that automatically downloads, configures and installs mods. The original thread for OpenMMM can be found here: viewtopic.php?f=40&t=5672

While I would like to use this opportunity to announce that version 2.0 is complete, unfortunately that is not yet the case. However after some recent work, I have reached the important milestone of having completed the planned changes to the pybuild and repository formats. The remaininng features are mostly optional and non-breaking additions to the pybuild spec (texture sizes and binary patches), as well as a couple of user tools for migration. I've also taken the opportunity to merge the dev changes into master since the master branch has long since ceased to represent the current form of the project.

Large amounts of credit goes to Gentoo’s Portage project (https://wiki.gentoo.org/wiki/Project:Portage), which this is heavily based on. While I considered forking Portage, I decided instead on building from scratch a version supporting a subset of Portage’s features (though I have on occasion borrowed code), due to the size and complexity of the Portage codebase and the fact that many of its features will not be used in Portmod.

Mod descriptions, and information Portmod uses to install them, are stored in a repository located here. The repository contains one or more build files for each mod, each in a pybuild format, a variation on Portage’s ebuild format that is expressed as python code, as well as files storing repository metadata.

Current State:
The project is more or less functional, albeit lacking in mods to actually install. There may be bugs (particularly with new installations as I test that less often), but it is relatively feature-rich already in terms of Portmod itself.
See the README for a brief overview of its usage; I plan to document more detailed usage information in the project wiki, but that has not yet been done.

There are two main goals prior to the release of v2.0:
1. Fill the repository. I plan to add all mods that I can find some reference to working with OpenMW. Help would certainly be appreciated, and I will put together a guide to writing pybuilds soon. There are also currently a few missing features that will hinder adding certain mods to the repo that need to be implemented prior to 2.0.
2. Make migrating a mod setup to Portmod as hassle-free as possible. One way this will be done is to have a tool that will automatically detect manually installed mods and search for them in the repo, prompting the user to migrate them. Another is to have an archive importer that will detect, rename and move archives based on their hash so that you don't have to re-download the source archives for all the mods (particularly useful given the below note regarding Nexus Mods). Any other suggestions would be appreciated, but I think this more or less should cover it.

Reporting Bugs
Please report any bugs you find on the GitLab issue tracker.
https://gitlab.com/portmod/portmod/issues
You can also email [email protected] if you don't have a gitlab account.

News of Note:
ModdingOpenMW.com:
I’ve been discussing with with ModdingOpenMWdotcom the possibility of working together. The combination of mod installation through portmod and exploration of available mods through moddingopenmw.com could make for a very capable and usable result.

Nexusmods.com:
It turns out that Nexus Mods only allows direct download through their API if you’re a premium user. Lacking a way to test this functionality, and preferring to avoid spending time on a feature that will only be usable for the subset of people who are premium nexusmods users, the following three things will be done for mods on NexusMods:

1. If the mod is available elsewhere with direct download, we’ll use that.
2. If the mod not available elsewhere, but its license permits redistribution, I will create a mirror somewhere and make the download available through the mirror (I will need to add support for download mirrors, but it’s not a particularly complex feature).
3. If the mod is not available elsewhere and its license does not permit redistribution, portmod will prompt the user to manually download the mod into the cache directory, and then will continue installation as usual.

Preferential Features:
I mentioned in a previous update that I’m planning functionality to handle mod ordering choices for when a user wants to prefer a certain mod over another conflicting mod (a situation where there is no “right” way to do it, essentially).
I’ve given further thought to this and came up with the idea of using a flag system, lets call them “feature” flags (admittedly the term is overloaded enough already, but I can’t think of a better name at the moment), where, for each feature flag that occurs in multiple mods you have installed, you are required to decide which mod you want to handle that feature.

For example: you might have two magic mods installed with the following flags:
mod1: magic_effects magic_spells
mod2: magic_effects magic_spells magic_potions

Suppose you prefer the changes to the effects that mod1 uses over mod2’s changes, but prefer the spells from mod2. Then you could select magic_effects: mod1, magic_spells: mod2, and magic_potions would automatically be set to mod2, since it’s the only mod that has it.

An important point here is that this is more than just choosing the ordering of the mods in the load order. Assuming that the mods support it, you could select preferences for individual features, rather than selecting preferences for all the features supported by a mod. I.e., as in the example above, you can select a feature from one mod and a feature from a different mod, even if both mods contain both features.
Unfortunately while this is relatively easy when it comes to assets (you can just remove the files that are not needed), picking and choosing features from a mod would be more difficult when those features are locked inside plugins, and short of writing mod-specific patches, the best solution would be to indicate that the mod can only provide certain features at the same time and throw an error if a certain configuration cannot be resolved.

An example of a potential use for this feature (again, way too overloaded of a term) would be to allow a texture mod to specify which things it re-textures and letting the user easily choose between multiple different texture options that are provided by different mods, for a particular thing. You could actually just install a bunch of different texture mods and go through each of the options, switching between enabled textures with just a couple of commands.

It’s worth noting that this, while not being too difficult to implement in portmod, would be quite time-consuming to implement at the repository level, since every mod that uses it would need to be set up to declare which parts of it belong to which feature. That being said, rather than micromanaging every little feature, it could just be used for large things, or as a way of resolving major conflicts, and then most mods would just ignore the subsystem entirely.
Also note that this feature system is not yet implemented and I do not plan on implementing it for v2.0. Eventually something like it will be incorporated though.

darkbasic
Posts: 87
Joined: 18 Apr 2016, 15:45
Contact:

Re: Portmod - Mod Manager

Post by darkbasic » 09 Apr 2019, 12:03

Wow, a mod package manager inspired by Gentoo's ebuilds is like a dream come true :shock:
Last edited by darkbasic on 09 Apr 2019, 12:46, edited 1 time in total.

darkbasic
Posts: 87
Joined: 18 Apr 2016, 15:45
Contact:

Re: Portmod - Mod Manager

Post by darkbasic » 09 Apr 2019, 12:05

Also, do you plan to support use cases like "if mod A is installed then I need to apply patch C to mod B"?

User avatar
bmw
Posts: 18
Joined: 04 Jan 2019, 19:42
Contact:

Re: Portmod - Mod Manager

Post by bmw » 09 Apr 2019, 15:35

darkbasic wrote:
09 Apr 2019, 12:05
Also, do you plan to support use cases like "if mod A is installed then I need to apply patch C to mod B"?
Definitely. In fact, it's already partially supported using use flags. For example, the master-index tribunal journal fix is included in the master index pybuild and is automatically applied if the tribunal global use flag is enabled. I say that this is partial support as doing this with use flags is for patches that are optional (as technically the master-index is still usable if tribunal is installed but the patch is not). On the other hand, for patches that resolve conflicts that would otherwise break things, you would mark the two mods as conflicting and mark the patch as resolving the conflict. Conflicts are already implemented, but marking directories/files as resolving conflicts is not yet.

Edit: I also feel I should mention that in general I'm using the convention that patches are included with the mods they patch, rather than separately in the repository, except for the case of patches that are not for specific mods such as omwllf.

ModdingOpenMWdotcom
Posts: 29
Joined: 12 Jul 2018, 04:25
Contact:

Re: Portmod - Mod Manager

Post by ModdingOpenMWdotcom » 10 Apr 2019, 18:40

darkbasic wrote:
09 Apr 2019, 12:03
Wow, a mod package manager inspired by Gentoo's ebuilds is like a dream come true :shock:
Agreed! Huge thanks to Ben for his awesome work on this. I look forward to growing that repository.

User avatar
Jemolk
Posts: 180
Joined: 20 Mar 2017, 02:12
Location: Sadrith Mora

Re: Portmod - Mod Manager

Post by Jemolk » 10 Apr 2019, 18:49

bmw wrote:
09 Apr 2019, 04:10
Preferential Features:
I mentioned in a previous update that I’m planning functionality to handle mod ordering choices for when a user wants to prefer a certain mod over another conflicting mod (a situation where there is no “right” way to do it, essentially).
I’ve given further thought to this and came up with the idea of using a flag system, lets call them “feature” flags (admittedly the term is overloaded enough already, but I can’t think of a better name at the moment), where, for each feature flag that occurs in multiple mods you have installed, you are required to decide which mod you want to handle that feature.

For example: you might have two magic mods installed with the following flags:
mod1: magic_effects magic_spells
mod2: magic_effects magic_spells magic_potions
[...]
Unfortunately while this is relatively easy when it comes to assets (you can just remove the files that are not needed), picking and choosing features from a mod would be more difficult when those features are locked inside plugins, and short of writing mod-specific patches, the best solution would be to indicate that the mod can only provide certain features at the same time and throw an error if a certain configuration cannot be resolved.
(Emphasis mine)
Wrye Bash's Bashed Patch feature could provide ideas for implementation here. It basically creates a plugin that merges features from various mods into one file so that they don't conflict, and you can control what gets imported with bash tags, which are basically exactly the same idea as the flags you're talking about and get placed in the mod's file header, and with check boxes on the GUI screen when you build the Bashed Patch. Generally it's used for merging changes to the same record from multiple mods when they're tinkering with different subrecords, but it can also be used to selectively have portions of one mod override another that's technically loaded later. Wrye Bash was originally an Oblivion utility, upgraded over Wrye Mash, the Morrowind utility written by Wrye, which doesn't, IIRC, have the same patching functionality.

User avatar
AnyOldName3
Posts: 1478
Joined: 26 Nov 2015, 03:25

Re: Portmod - Mod Manager

Post by AnyOldName3 » 10 Apr 2019, 20:25

Mator Smash is now the recommended utility instead of Wrye Bash for making that kind of patch for the games it supports. It might be getting Morrowind support soon now TES3Edit exists/is about to exist.
AnyOldName3, Master of Shadows

User avatar
bmw
Posts: 18
Joined: 04 Jan 2019, 19:42
Contact:

Re: Portmod - Mod Manager

Post by bmw » 10 Apr 2019, 21:47

I hadn't actually made the connection to the Bashed Patch, thanks, that's a good point.
It would still be a lot of work to integrate such a thing though, given the lack of Morrowind support and the fact that neither Wrye Bash nor Mator Smash seem to be designed to integrate with other software. I could probably hack out the important parts though, or just use them as a basis for a implementing whatever ends up going into Portmod. Either way, I think it's currently at the bottom of my list of things to do, which admittedly does mean there's plenty of time to discuss the idea and figure out exactly the best way to implement it.

BaronPampa
Posts: 16
Joined: 04 Aug 2017, 12:50

Re: Portmod - Mod Manager

Post by BaronPampa » 13 Apr 2019, 13:37

Seems fantastic. I'd love the ability to get massive mod compilations installed with one script, instead of doing all the manual work myself. Also, I've got a list of some mods which provide permission to redistribute:

Morrowind Optimization Patch
Correct Meshes
Correct UV Mudcrabs
Ingredients Mesh Replacer
RR Mod Series - Better Meshes
Better Meshes plus Optimization
PikachunoTMs Mesh Artifact Improvements and Replacers
HD Vanilla Textures

You can download all of them here: https://www.mediafire.com/file/1br1gbvb ... M.7z/file-
In each folder there's a "permission.txt." file, which contains a quote from the author and link to prove that redistribution is allowed.

User avatar
Jemolk
Posts: 180
Joined: 20 Mar 2017, 02:12
Location: Sadrith Mora

Re: Portmod - Mod Manager

Post by Jemolk » 14 Apr 2019, 00:11

AnyOldName3 wrote:
10 Apr 2019, 20:25
Mator Smash is now the recommended utility instead of Wrye Bash for making that kind of patch for the games it supports.
Only because Wrye Bash doesn't support all the relevant records for those games. Mator Smash doesn't support merging mods into the patch entirely as Wrye Bash does, nor does it have the tweaks functionality of Wrye Bash. It's not that Mator Smash is better than Wrye Bash, but that Mator Smash supports newer files than Wrye Bash but with less features and capabilities.

Post Reply