Portmod - Mod Manager

Learn about OpenMW mod compatibility, check for problematic mods, discuss issues, and give us feedback about your experience with modded OpenMW.
User avatar
bmw
Posts: 23
Joined: 04 Jan 2019, 19:42
Contact:

Re: Portmod - Mod Manager

Post by bmw » 02 May 2019, 23:03

I’ve been working a lot on Portmod, and one of the few remaining features is patch support.
To recap, the idea was to use binary patches to handle in-place modifications to arbitrary files for purposes such as cleaning plugins, applying the modifications made by the TR-patcher and fixing bump maps on NIF files.

Unfortunately, looking further into the idea, binary patches really don’t seem to be a good tool for this. Unlike text patches, binary patches are extremely opaque, both due to the fact that the diff contained by the patch is binary, and the fact that none of the options I’ve seen (xdelta, bsdiff) support any sort of comments/annotations inside the files themselves.

We could go back to the idea of just having portmod run these scripts at runtime, possibly even including them in the repository, however we can’t include their dependencies in the repository (i.e. perl for tes3cmd, java for TR-patcher (admittedly common, though as usual I have idea as to the state of such dependencies on Windows), and pyffi for fix_nif (available via pip, but not through package managers)). I don't particularly like either the idea of including various tools in the mod repo, given that there are a lot of platform-specific issues with respect to installing them, or the idea of just having them be dependencies of Portmod, given that many mods won't need them and the dependency list just keeps growing. I think I have come up with a reasonable way of doing it, but I'm not entirely satisfied with it.

An alternative I’m considering, is creating patch files specifically formatted for plugin files. Basically a file containing a sequence of record/subrecord remove and add/modify commands, with text diffs for the contents of those records. Sort of like the diffs that esmtool can already produce, though without including the full details of removed records and having some nice wrapper information for descriptive purposes and to allow the patches to be self applying in the way that regular patches are. This wouldn't fix the bump map issue, but does reduce the number of special cases, especially given that pyffi and fix_nif are both implemented in python and would work nicely with a pip install of portmod.

The idea of plugin patches does bring us back to the issue of a native plugin library, previously discussed on the old thread with respect to omwllf. One thing that did occur to me is that we already have such a library within OpenMW itself, it’s just that OpenMW doesn’t currently provide the library and the relevant interfaces (or at least doesn't install it). I imagine that setting up such a library is probably not terribly high priority for the dev team at the moment, but it would be rather helpful. That being said, given that esplugin is already very close to providing all the necessary features, I’m really not sure whether writing a Rust interface for such a C++ “libopenmw” would be more or less work than just making the necessary changes to esplugin.

I think I’ll leave the issue of patches for a bit and start working on expanding the repository, as almost all other work on portmod 2.0 has been completed.

Also, thanks BaronPampa for the list of re-distributable mods. I'll probably add those to the repo first as I like mods with permissive licenses.

User avatar
Jemolk
Posts: 189
Joined: 20 Mar 2017, 02:12
Location: Sadrith Mora

Re: Portmod - Mod Manager

Post by Jemolk » 05 Jun 2019, 00:43

Just a heads up -- for resolving mod conflicts, we have a potential solution upcoming now. TES3Merge is being worked on by NullCascade, and he's said he will accept pull requests to improve it/make it run on Linux.

User avatar
AnyOldName3
Posts: 1561
Joined: 26 Nov 2015, 03:25

Re: Portmod - Mod Manager

Post by AnyOldName3 » 05 Jun 2019, 01:22

I don't understand how it can look so much like xEdit if it's written in C# (xEdit is in Delphi) and supposedly a different program. Maybe the screenshot isn't a screenshot of the tool, but instead of its results loaded into TES3Edit?
AnyOldName3, Master of Shadows

User avatar
Capostrophic
Posts: 659
Joined: 22 Feb 2016, 20:32

Re: Portmod - Mod Manager

Post by Capostrophic » 05 Jun 2019, 09:17

It is. The tool itself seems to be a command-line application with no input required from the user.

User avatar
bmw
Posts: 23
Joined: 04 Jan 2019, 19:42
Contact:

Re: Portmod - Mod Manager

Post by bmw » 07 Jun 2019, 17:04

Interesting... I can't say that I love C#, but any command line program is better than a gui (at least with respect to portmod compatibility). I will watch its development with interest.

I've quickly tried it out and it works with mono on Linux if you replace the backslash path separators in code with forward-slashes (using mono's fork of msbuild to compile and noting that you have to install ini-parser into a packages directory within the source tree). It's lacking awareness of the openmw vfs, or indeed any sort of configuration as to where plugins can be found, and while I could work around that (I could write a pybuild today that copies all plugins into a temp directory that pretends to be a Morrowind install and compiles and runs tes3merge using mono if the platform is not windows), I'd rather get better support included upstream.

On an unrelated note, I've recently completed batch-importing a large number of mods, which I've put in a new testing repo. There are about 370 mods in total (almost every mod on modding-openmw.com, plus a few extra), and it is likely that many of the build files work in their current form (though all are missing at the very least license information), however complex mods with options and compatibility patches will need manual adjustments to the build files. I've started the long process of manually fixing them up so they can be included in the main mod repo.

rdcl170
Posts: 3
Joined: 27 Jul 2019, 20:56

Re: Portmod - Mod Manager

Post by rdcl170 » 31 Jul 2019, 04:24

This is looking great. Just what I need

User avatar
bmw
Posts: 23
Joined: 04 Jan 2019, 19:42
Contact:

Re: Portmod - Mod Manager

Post by bmw » 01 Aug 2019, 02:27

I've been a bit busy with other things recently, but have managed to merge a couple of major features into Portmod, and there is some other news worth mentioning too.

tldr:
  • If the idea that Portmod essentially downloads python scripts from the internet and runs them to install mods worries you, you can rest somewhat easier as we now sandbox all pybuilds to ensure they don’t do anything nasty to your system.
  • Portmod now supports automatically updating fallback entries in the config, as well as settings.cfg. Soon there will be tes3mp server-side support in some form.
  • Portmod 2.0 alpha will be released soon-ish.
Security
At the core of Portmod’s functionality is the ability to use python files, downloaded from an online repo, to make the changes necessary to install mods. This gives it tremendous power, as it can make arbitrary changes to get the mod to work, however at the same time it also means that there are a lot of things that it can do that aren’t related to setting up the mod, such as making potentially dangerous changes to your filesystem.

It’s worth noting that pybuilds available from the main repo are not going to do anything nasty to your computer by design, however there are other reasons to be worried about this sort of thing:
  1. Mistakes certainly do happen. A simple mistake in a path could be the difference between getting rid of a few unnecessary files in a mod and deleting the user’s home directory.
  2. Malicious third parties could gain access to the repo, or, noting that Portmod supports adding arbitrary third-party repos, could set up a seemingly benign repo that has a more sinister purpose.
To prevent this sort of thing from happening, we create a sandboxed environment for pybuilds to work in that allows them to make changes necessary to set up the mods, but prevents dangerous modifications to the user’s system (primarily by restricting filesystem access and network access).

See here and here for details about the implementation, but in a nutshell we use RestrictedPython to limit what can be done on the python side and a sandbox tool such as bubblewrap (on Linux anyway) to limit what can be done when executing system commands.

Sadly on other platforms open source sandbox tools seem to be more limited. I am not aware of any open-source tools for either Windows or OSX, however OSX does have a builtin sandbox-exec command, and given that OSX users trust Apple enough to use OSX, they are presumably also okay with trusting sandbox-exec. Windows on the other hand is as terrible as its reputation from a security point of view. There is apparently a built-in sandbox introduced recently to newer versions, but it seems to just wrap a virtual machine, I can’t find any documentation, it doesn’t appear to be available from the command line, and it’s hardly as ubiquitous as sandbox-exec on OSX. Instead the solution for now is that Windows users will need to define their own sandbox command (See the Sandbox wiki page for details).
There are proprietary sandbox tools that exist, but I’m not comfortable endorsing those as I’m not fond of the ethics of proprietary software and we can’t know if they are actually secure. If anyone knows of an open-source windows sandbox tool, please let me know.

It’s also worth noting that the sandbox has introduced a significant slowdown to loading pybuilds, since part of the process includes cloning all imported python modules (python doesn’t really have a concept of read-only objects, so we use duplication to isolate them instead). I will revisit optimizing this later.

Config Files
Portmod now supports updating arbitrary config files. This made it easy to implement updating fallback entries in openmw.cfg and settings.cfg changes, and means that supporting server-side tes3mp is now more or less possible.

Currently the issue for tes3mp is a matter of tes3mp’s packaging (or lack thereof). If it was manually installed, we probably have write access to CoreScripts, but can’t automatically detect where it is (we could have the user specify), while if it is installed by a package manager (extremely uncommon at the moment), we probably don’t have write access to CoreScripts. Tes3mp-server.cfg allows specifying the entire CoreScripts directory location, but while Portmod could, it doesn’t really need to manage the entirety of it, just scripts/customScripts.lua and data/requiredDataFiles.json. Probably the best way to handle this until I hear back from upstream is to require the user to specify the location of their CoreScripts directory and raise an exception if we don’t have write access.

This also raises the issue of supporting openmw and tes3mp side-by-side, or even multiple openmw/tes3mp installations, something that isn’t currently possible, but probably should be. An option at the moment (probably most convenient for Linux users) would be to create a separate user for each installation, but I imagine that there might be interest in a solution that is more integrated into Portmod.

2.0 alpha
I’d mentioned a release candidate a while back, and while I’m not even committing to something as close to a stable release as a release candidate yet, I am planning on releasing an alpha version very soon, primarily as a way of indicating that Portmod is more or less feature-complete. The mod repo is sparse yet, but there is also better documentation for adding mods on the Wiki for those of you interested in helping. There are also still some missing features that I plan to implement before the full 2.0 release, however those are mostly smaller, non-breaking changes that are just necessary for mods not yet included in the repo, the big stuff has now been done.

I'll probably have the alpha release out relatively soon, possibly as early as this weekend. There won't be significant changes from the current development version, mainly this will just be stamping the version number onto it and releasing it on pypi (there's also a small bug I need to fix first too).

User avatar
AnyOldName3
Posts: 1561
Joined: 26 Nov 2015, 03:25

Re: Portmod - Mod Manager

Post by AnyOldName3 » 01 Aug 2019, 21:01

If the idea that Portmod essentially downloads python scripts from the internet and runs them to install mods worries you, you can rest somewhat easier as we now sandbox all pybuilds to ensure they don’t do anything nasty to your system.
Honestly, I still see this as a half-solution. Security is only one of the reasons I'm against having scripts for mod installation, and while I don't know of a single case of an executable mod installer doing anything malicious, I do have examples of it causing genuine problems for other reasons.

The key thing is that while a mod package format having something like a FOMOD XML can be implemented into future mod managers relatively simply (and interaction with manager features the format wasn't aware of can be more easily sorted), anything executable needs far more work. This can be seen with FOMODs and OMODs - XML FOMODs are supported in nearly every mod manager people actually use, but C# FOMODs aren't as widely supported, with Mod Organizer going as far as building Nexus Mod Manager as a DLL and calling functions in it instead of implementing support natively. OMODs are even worse, with some managers supporting unscripted ones, but only the original OBMM which invented the format supporting scripted ones.

Even if Portmod sounds like the ultimate solution to every problem now, it's not impossible that something better will exist in the future, and scripts make it much harder for that future thing to support mods packaged in your format.
AnyOldName3, Master of Shadows

User avatar
bmw
Posts: 23
Joined: 04 Jan 2019, 19:42
Contact:

Re: Portmod - Mod Manager

Post by bmw » 03 Aug 2019, 01:16

AnyOldName3 wrote:
01 Aug 2019, 21:01
The key thing is that while a mod package format having something like a FOMOD XML can be implemented into future mod managers relatively simply (and interaction with manager features the format wasn't aware of can be more easily sorted), anything executable needs far more work. This can be seen with FOMODs and OMODs - XML FOMODs are supported in nearly every mod manager people actually use, but C# FOMODs aren't as widely supported, with Mod Organizer going as far as building Nexus Mod Manager as a DLL and calling functions in it instead of implementing support natively. OMODs are even worse, with some managers supporting unscripted ones, but only the original OBMM which invented the format supporting scripted ones.

Even if Portmod sounds like the ultimate solution to every problem now, it's not impossible that something better will exist in the future, and scripts make it much harder for that future thing to support mods packaged in your format.
What Portmod does is fundamentally different from installers. Yes, what’s actually happening is similar, in that Portmod uses a script to do the installation, but the similarities end there. There are no mods that are packaged using this format, it exists entirely separate from the mods and is in the hands of the community as a whole rather than individual modders. I’ve released all the build files in a public repo under the GPL and openly invited the community to contribute, so it’s not like we’ll ever be locked into anything, like what happens with abandoned mods that are released without a permissive license and use an outdated packaging format.

Now this doesn’t mean that forward compatibility isn’t a concern, but, similarly to Linux package managers, it’s not necessary that new package managers support older formats, though there are examples of this and it may be useful to avoid having to create a new repo from scratch. Admittedly writing full support for Portmod’s build files in other software from scratch would be rather complex, and completely automatic migration to a new format would also not be very simple, so I agree that it’s not exactly an ideal solution in this respect.
The main thing that I think will help with forward-compatibility with other systems is that Portmod is highly scriptable and it is relatively easy to pull information out of the mod repo due to the data being generally very well structured and there being many functions built into portmod that allow easy access to the mod metadata (and more user-facing ones coming soon).

The only other similar mod manager I can think of to compare to is CKAN, and you are right, a system like CKAN’s json files would be much easier to migrate from, or for future systems to parse, but the poor consistency and varied requirements among morrowind mods means that such a rigid system would need to be insanely complex to be able to support everything. That being said, for simpler mods that don’t need scripting I could generate something akin to CKAN’s json files relatively easily due to how scriptable the system is, and how much of the build files isn’t just scripts. Migration would also require much less work than has been necessary to create the Portmod build files in the first place, given how much well structured information is (and will be) available through Portmod, while on the other hand there was/is no pre-existing data source detailed enough to make use of (modding-openmw.com is good, but has too little detail and is not structured enough for to make use of directly for this purpose, though I have been using what information I can).

User avatar
bmw
Posts: 23
Joined: 04 Jan 2019, 19:42
Contact:

Re: Portmod - Mod Manager

Post by bmw » 12 Aug 2019, 04:03

Portmod 2.0 alpha is out!

It took a little longer than I expected as I ended up doing a bit of a bug hunt, but it should be a lot more stable than before and test coverage is now above 50%.

The following packages are now available through https://pypi.org (i.e. can be installed via pip and similar python package tools):
  1. portmod - https://pypi.org/project/portmod/
  2. portmod-migrate - https://pypi.org/project/portmodmigrate/ - Tool for migrating from manually installed mods to Portmod. Previously part of importmod.
  3. importmod - https://pypi.org/project/importmod/ - Tool for facilitating the creation of new pybuilds. Largely undocumented at the moment.
The Portmod Wiki has also received some attention, and has more information related to installation, configuration and creating pybuilds and repos.

Future Plans:

My main focus will be continuing to add to the OpenMW Mods repository. If you want to help with this, as there are many mods to add, there is a guide on the wiki that gives a brief overview of how to contribute.

I’m also planning on creating an update detection tool so that we get informed automatically when a mod in the mod repo is out of date. This would use tools such as the nexus mods api and webpage scraping to determine if the mod has updated, and then automatically create an issue on gitlab and assign it to the mod’s maintainer.

Speaking of maintainers, if there are any people willing to maintain any of the orphaned mods (most of them) in the repo, that would be great. Basically you would be the first point of contact with respect to mod-specific issues (for the mod(s) you maintain) and would be responsible for updating the build files when new versions are released. This is particularly wanted for mods that are being actively developed. See Mod Maintainers for more information.

I’m also planning on moving the file conflict gui, that is currently in a small hastily-written python script using ncurses, into omwcmd. This will also facilitate a ui for the next feature planned for omwcmd, plugin conflict detection, something necessary to implement the overlap rule that LOOT uses for sorting, as well as being useful as a manual tool for determining plugin compatibility.

Then of course, there are a few issues remaining for Portmod itself, and generally working on improving testing, stability and documentation before the 2.0 stable release.

Post Reply