OpenMW Mod Manager

Post about your mods, learn about OpenMW mod compatibility, check for problematic mods, discuss issues, and give us feedback about your experience with modded OpenMW.
User avatar
bmw
Posts: 81
Joined: 04 Jan 2019, 19:42
Contact:

OpenMW Mod Manager

Post by bmw »

OpenMMM
Faced with the task of installing the large list of OpenMW-compatible mods on https://modding-openmw.com/ (because why wouldn't you play OpenMW with hundreds of mods installed?), I did not like the idea of doing everything manually. Unfortunately there seems to be a lack of both native Linux mod managers, as well as mod managers that actually install mods the OpenMW way.
So starting with a small script I found on Github (https://github.com/Korons/openMMM/, author never responded to my pull request), I slowly built on it until I had a tool that could in a single command install any mod on the list with minimal manual configuration.

See the readme on Gitlab for more details and installation instructions.
https://gitlab.com/bmwinger/openMMM

Features
  • Mods can be installed directly from archives. OpenMMM automatically detects the location of the data files directory within the archive, as well as optional directories (providing a prompt if multiple are found)
  • Automatically updates the openmw.cfg file, adding the data directories, registering bsas, adding esps, esms, omwaddons etc.
  • Automatically detects and renames normal and specular textures files to use the _n and _spec suffixes respectively.
  • If mlox is in your path, OpenMMM can use it to update the load order (currently just appends omwaddon files onto the end of the list)
Future Plans
(feedback would be appreciated)
  • Conflict detection and mod install order editor (dcv combined with openmw-conflicts in the dev branch can be used for conflict detection, plus automatic data directory ordering has been implemented in dev) - Because OpenMW supports multiple data directories this is less tied to the mod installer than in vanilla, so this doesn't necessarily have to be included as part of the mod manager, but there is also no reason why it couldn't. As I don't plan on creating a GUI for the mod manager, this will at the very least use a separate interface.
  • Mod repository (In development in the dev branch of the repository) - This would solve a lot of problems, such as mod specific patches which OpenMMM currently has no way of supporting, as well as enabling a fast way of checking for updates and single-command download and install in place of the current two-step process. Supporting Nexus would be more difficult than locations where mods can be downloaded without authentication, but as Nexus Mod Manager is open source it should be feasible.
  • CLI version of the Tamriel Rebuilt Patcher (mostly done Now available here), so that it can be run automatically by OpenMMM. This would benefit hugely from a repository as otherwise OpenMMM would just have to run it on everything or guess based on the name of the esps.
Cross-Platform Support
Currently OpenMMM only supports Linux, however this does not have to be the case. As it is written in python, and most of the code should run independent of platform, it could certainly be rewritten to run on Windows and OSX.
This is a project that I mainly hacked together as I was installing mods, and as I exclusively use Linux I generally did not spend time trying to ensure that everything would work on other platforms. If someone would be willing to work on getting it running on other platforms that would be ideal, but if there is sufficient interest and someone willing to, at the least, test it on other platforms, I could work on removing the Linux-specific code.

Downloading
The latest release (v1.0.1) can be downloaded here:
https://gitlab.com/bmwinger/openMMM/tags
A package can also be found in the Arch User Repository, for those Arch Linux users out there.

Reporting Bugs
Please report any bugs you find on the GitLab issue tracker.
https://gitlab.com/bmwinger/openMMM/issues
You can also email [email protected] if you don't have a gitlab account.
Last edited by bmw on 20 Feb 2019, 16:22, edited 2 times in total.
User avatar
Enkida
Posts: 51
Joined: 27 Nov 2018, 20:57

Re: OpenMW Mod Manager

Post by Enkida »

Thanks for all your hard work! This sounds like something I could've used last year :lol: I do hope someone will convert it to work with Windows at some point in time. :)
jmelesky
Posts: 47
Joined: 02 Mar 2017, 20:52

Re: OpenMW Mod Manager

Post by jmelesky »

Nice! Came here to comment about integrating with omwllf, and then saw it was already integrated!

Thanks for working on this, and for leveraging existing work :)
User avatar
AnyOldName3
Posts: 2668
Joined: 26 Nov 2015, 03:25

Re: OpenMW Mod Manager

Post by AnyOldName3 »

Enkida wrote: 05 Jan 2019, 16:22 Thanks for all your hard work! This sounds like something I could've used last year :lol: I do hope someone will convert it to work with Windows at some point in time. :)
Mod Organizer already does all of this on Windows with a plugin, and actually lets you use non-VFS-aware tools like MLOX with the VFS.

Maybe we should mention this somewhere...
User avatar
bmw
Posts: 81
Joined: 04 Jan 2019, 19:42
Contact:

Re: OpenMW Mod Manager

Post by bmw »

jmelesky wrote: 06 Jan 2019, 01:13 Nice! Came here to comment about integrating with omwllf, and then saw it was already integrated!

Thanks for working on this, and for leveraging existing work :)
You're welcome. I always prefer to avoid recreating something if it already exists. That being said, and as AnyOldName3 pointed out, mod organizer does apparently do this stuff already on Windows, but I imagine that getting that to work on Linux would take significantly longer (its not even fully supported with Wine) and, in any case, I much prefer command line tools to clunky GUIs (among other things, they can integrate much more easily).

Actually I was wondering if, to better integrate omwllf, it would be possible to do some sort of incremental levelled list updates. I've noticed that at the moment omwllf takes quite some time to run when there are many mods installed; would it be possible to have omwllf update a previously created levelled list file and only process the mods that have changed instead of creating an entirely new one? I haven't looked looked too much into how omwllf works (I'll probably do so later), but running a fast update every time a mod is installed might be preferable to remembering to run a slow update afterwards, especially if you're adding mods one at a time.
User avatar
psi29a
Posts: 5356
Joined: 29 Sep 2011, 10:13
Location: Belgium
Gitlab profile: https://gitlab.com/psi29a/
Contact:

Re: OpenMW Mod Manager

Post by psi29a »

Isn't mod organizer better supported with more contributors? I believe it would be in everyone's best interest to help port this to Linux (and other platforms).
Lowtek
Posts: 2
Joined: 06 Jan 2019, 22:38

Re: OpenMW Mod Manager

Post by Lowtek »

Hi there OpenMW Forum,

running nightlies on Mint/Ubuntu/Gnu/Linux/ :) Nice to see others with a similar setup.

Mod management is a struggle: My primary problem is keeping meta-information (comments, compatibility, urls) together. I am using a centralized file to store my load order with comments and grep it into an openmw.cfg template, so i might give your script a test drive. The "unpack and interactive install" looks promising, as does the directory scanning and checking.

One feature i noticed: "Attempts to fix incorrectly named normal and specular texture files"
Warning: Mods may contain such files for the vanilla engine. There is a major difference in how these are used and renaming them is not a valid solution! These mods may include .nif Meshes in which those textures are named, maybe on a very specific part of that mesh. The correct approach is to remove those textures from the mesh. You may also want to make screenshots without the _n.dds and with it and have a look how openmw handles a vanilla engine compatible "bump map" it now uses as a "normal map" after you cleaned the mesh. I haven't experimented with mods that include_reflection.dds files or other specular maps, yet.

There is a nice tool i found called NIF Texture Stripper by Ryan Cecil / Era Scarecrow v15 (Gplv3) written in C. Wonderful little thing: it's primary purpose is to remove absolute texture paths from meshes (detect and remove a bug from a nif), but it does this by crawling directories and listing all textures used in all the meshes, which is a very nice feature.
Especially since its --no-save switch dumps the list of used textures (and planned changes) without changing the nif.

I patched it up a bit (for purists)
Spoiler: Show
and refactored it a bit more to make the code better understandable. Next should be parseable output. The simplicity of this tool is beautiful, call it on a mesh or mod dir and get a tree of meshes and their referenced textures.

For anything more complex NifSkope runs native
Spoiler: Show
but i've yet to see a "remove texture from nif" command line tool. Stupid bump maps.

A larger toolkit by abot is called "Morrowind Resources Scanner" and also runs under linux. It isn't openmw ready (uses morrowind.ini, mse.ini and a single 'data files' folder) but can make an educated guess about used, unused and missing data files by scanning bsa, esp and nif files. Sadly its total added functionality is less than it's parts: there is scanning bsas and scanning nifs with regex and command code for tes3cmd to extract data dependencies from esps in this thing :shock: yet the "unused assets" list should be taken with a grain of salt.

A more conservative tool is openmw-modchecker.py by M'aiq The Admin. It's debug mode produces an almost parseable "who overwrites who" for the files in whitelisted subdirectories of 'data' dirs gathered from openmw.cfg

abot has made more tools that run on linux, most notably MMOG the Morrowind Merged Objects Generator, again a perl controller for tes3cmd to solve object conflicts.

As well as "unique toolkit" which calls tes3cmd to count usage of things marked unique.

"tes3cmd clean *.esp" should not be missed in any pipeline, yet again there are rumors about mods that must stay unclean, so we are back to "metadata management". This tool also needs a compatibility update, it requires symlinks to all used esms and a generated morrowind.ini in its path for its multipatch --cellnames and overdial functions. (It's perl btw). The "header can also be useful to check for warnings and to --synchronize when they are solved.
I am not sure if multipatch --fogbug --summons-persist is relevant for openMW, and for --merge-lists there is an alternative with omwllf.py

Last but not least we have mlox, which has strong opinions about load order of esps as well as where and how the users should note their own preferences (mlox_user.txt) so mlox can do the reordering. You should have a look at it, if you want to inject mods interactivly in the load order.

Imho the vital part of mod management is to know and integrate these tools.
There is a ModOrganizer-to-OpenMW Plugin from AnyOldName3 (git repo) which i didn't yet try and WyreMash Polemos fork, whose git repo is outdated and does not yet include the OpenMW features.

So.. welcome to the world of linux openmw mod management.

In my opinion we need small tools that do their job well and can be piped together.

Mod management is all about knowledge. It needs a representation of what a mod is and how it relates to others Mods in terms of "openmw.cfg order", "mlox recommendations", "mlox user preference", "asset overwrites", "patches", "alternatives", "compatibility merges", "folder structure closeness", "tags". "user comments", and "urls". In an optimal case it could import mod recommendations list and metadata from nexus, modhistory or other repositories. Tools that help with gathering and storing that information are always nice to have.

An interactive archive installer on cli you say? That sounds interesting, i loose time mangling archives. Will give it a try.
In the "CLI version of the Tamriel Rebuilt Patcher" i am also interested. Since finding outdated master references can be scripted, batch patching would be possible :D
User avatar
bmw
Posts: 81
Joined: 04 Jan 2019, 19:42
Contact:

Re: OpenMW Mod Manager

Post by bmw »

psi29a wrote: 06 Jan 2019, 21:09 Isn't mod organizer better supported with more contributors? I believe it would be in everyone's best interest to help port this to Linux (and other platforms).
Lowtek wrote: 07 Jan 2019, 07:10 In my opinion we need small tools that do their job well and can be piped together.
Unfortunately it's not really as simple as focusing on bringing Mod Organizer to Linux. I'm hardly an expert on Mod Organizer, but I've taken a short look at the code and my initial conclusion is that bringing it to Linux would take a large amount of work, that is, more than it was to make this tool.
The bigger reason though is that Mod Organizer isn't really the sort of thing that Linux users such as myself actually want. As Lowtek puts it, we want small, single-purpose tools that run on the command line and work with each other, rather than large, bloated, monolithic, GUI programs like Mod Organizer.
Lowtek wrote: 07 Jan 2019, 07:10 Warning: Mods may contain such files for the vanilla engine. There is a major difference in how these are used and renaming them is not a valid solution!
Yes, I know that renaming isn't really a valid solution. I was mainly going from the tips on modding-openmw. I'm interested to hear that Nifscope can compile on Linux; I wonder what the possibility of getting a cli interface for that is? Unfortunately I've had no luck getting it to build on either my Arch or Gentoo machines. I'll take another look later.
Lowtek wrote: 07 Jan 2019, 07:10 My primary problem is keeping meta-information (comments, compatibility, urls) together.
This is why I think we need to build a mod repository. It would be really useful to have a centralized database containing compatibility information (dependencies, conflicts etc.) as well as changes necessary to get mods working properly with OpenMW.
Lowtek wrote: 07 Jan 2019, 07:10 In an optimal case it could import mod recommendations list and metadata from nexus, modhistory or other repositories.
I'm not really sure if this is an optimal case, as these recommendations and metadata are, from what I've seen, generally incomplete and hard to parse automatically. They would be very useful for including in the aforementioned mod repo though.

Thanks for all the links and info Lowtek! Other than mlox and omwllf (already integrated into OpenMMM) and Nifscope (heard of, haven't been able to use), I hadn't heard of those previously.
Basically my hope is that OpenMMM can be the tool that handles installing mods and selectively applying these other tools to get a working configuration with as little user input as possible (user input will of course be necessary for the configuration of options).
User avatar
psi29a
Posts: 5356
Joined: 29 Sep 2011, 10:13
Location: Belgium
Gitlab profile: https://gitlab.com/psi29a/
Contact:

Re: OpenMW Mod Manager

Post by psi29a »

I put a lot time into porting NifScope to Linux, give it a try. :)

The beta versions should work out of the box.

(Hint, also a member of the niftools team)
jmelesky
Posts: 47
Joined: 02 Mar 2017, 20:52

Re: OpenMW Mod Manager

Post by jmelesky »

bmw wrote: 06 Jan 2019, 02:44 Actually I was wondering if, to better integrate omwllf, it would be possible to do some sort of incremental levelled list updates. I've noticed that at the moment omwllf takes quite some time to run when there are many mods installed; would it be possible to have omwllf update a previously created levelled list file and only process the mods that have changed instead of creating an entirely new one? I haven't looked looked too much into how omwllf works (I'll probably do so later), but running a fast update every time a mod is installed might be preferable to remembering to run a slow update afterwards, especially if you're adding mods one at a time.
Currently, omwllf does record the mods used in the "DESC" part of the resulting mod. Unfortunately, that "DESC" can only be 256 characters long (according to the old TES3 spec), so that info is very frequently truncated. It also doesn't keep track of mod version (or checksum, or whatnot).

Realistically, I don't think that's possible without writing a status file out somewhere else, which would complicate things.

That said, it should be possible to speed it up, though my likely next step would be a rewrite in not-python (and also not-c++ -- sorry, all).

What kind of times are you seeing, and with what length of mod lists? I guess I'm also curious what the more common use case is: add a bunch of mods occasionally, or add mods one at a time frequently? If it's the former, I'm inclined away from spending much time optimizing.
Post Reply