OpenMW Mod Manager

Post about your mods, learn about OpenMW mod compatibility, check for problematic mods, discuss issues, and give us feedback about your experience with modded OpenMW.
User avatar
bmw
Posts: 81
Joined: 04 Jan 2019, 19:42
Contact:

Re: OpenMW Mod Manager

Post by bmw »

psi29a wrote: 07 Jan 2019, 19:18 I put a lot time into porting NifScope to Linux, give it a try. :)

The beta versions should work out of the box.

(Hint, also a member of the niftools team)
It was the same issue as https://github.com/niftools/nifskope/pull/147: I'm using gcc 8.2 and Qt5 5.10. It might be useful to list dependencies in the readme.
I've got it working now using the change in the pull request.
jmelesky wrote: 07 Jan 2019, 21:21 What kind of times are you seeing, and with what length of mod lists? I guess I'm also curious what the more common use case is: add a bunch of mods occasionally, or add mods one at a time frequently? If it's the former, I'm inclined away from spending much time optimizing.
With 185 files being processed I'm getting a runtime of about 79 seconds. Not terrible if you only run it once after installing a bunch of mods, but in the context of installing them one at a time (if, for example, you wanted to test compatibility of each one before proceeding), it's pretty slow.
By comparison just running a sha512sum on all those files takes about 1.7 seconds (note ssd, and relevance below).
jmelesky wrote: 07 Jan 2019, 21:21 That said, it should be possible to speed it up, though my likely next step would be a rewrite in not-python (and also not-c++ -- sorry, all).
The trouble with converting it to not-python (also not fond of c++. Rust maybe? I like Rust), is that even if the result is an order of magnitude faster, it still may not scale well, seeing as the issue is caused by lots of mods. Then again, I ran a profile of it and the bulk of the work is being done in readSubRecord, almost certainly the string slices, since apparently Python does-slice by-copy, i.e. ever time you make a slice of a string in readSubRecord it's creating a new string.
Using a bytearray and consuming the processed elements rather than returning a slice of the remaining data got the total time down to 25 seconds (16 when compiled with cython). Still not amazing, but much better for the trivial change. I also checked and the two resulting files were identical save for the list of mods in the DESC, which had a different order.
Not-python could admittedly do better still by using references instead of slices for all the operations in readSubRecord, but it would be hard to tell how much better without trying it.

Relevant lines from the profile (97.996 seconds when profiling):

Code: Select all

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
 14377555   75.323    0.000   81.510    0.000 omwllf.py:103(readSubRecord)
   220616   11.396    0.000   96.510    0.000 omwllf.py:111(readRecords)
      185    1.343    0.007   97.854    0.529 omwllf.py:138(getRecords) 
After the modification (37.913 seconds when profiling):

Code: Select all

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
 14377555   17.016    0.000   24.080    0.000 omwllf.py:103(readSubRecord)
   220616    8.728    0.000   36.079    0.000 omwllf.py:112(readRecords)
      185    1.353    0.007   37.432    0.202 omwllf.py:138(getRecords) 
Virtually all the time is still being spent in getRecords, but substantially less in total.
jmelesky wrote: 07 Jan 2019, 21:21 Realistically, I don't think that's possible without writing a status file out somewhere else, which would complicate things.
Basically this is what I would suggest to improve on the above change without rewriting in a different language. Write a file containing the files used to build the merged levelled lists that contains their names and hashes (e.g. with sha512sum) and compare with the files you collect every time the mod is run. Then you'd know if a file has been removed, modified or added and can update the merged file to reflect that. What might also be worth trying is writing a cache of the records in a more python friendly format for each file (marshal is apparently quite fast) and import those for the files that haven't changed, given that reading all the records is the bottleneck.

Still though, you might be right that using a language other than python (specifically, where we can avoid costly copying of strings) is the best way forward (if any); the time for everything other than getRecords is very small, and efficiently written the overhead for that shouldn't be much more than the i/o cost (which is also necessary if caching the records).

Pull request for the small change from above incoming.
User avatar
bmw
Posts: 81
Joined: 04 Jan 2019, 19:42
Contact:

Re: OpenMW Mod Manager

Post by bmw »

The dev branch of OpenMMM now contains work that I've been doing on version 2.0, a complete rework of the original that uses a mod repository to allow for completely automated installation, with configuration mainly taking place on a global level. There will also be capability for local configuration of mods that have non-compatibility-related options (though any configuration shared between mods, e.g. texture size, I will try to eventually implement a global way of configuring).

The code can be found here: https://gitlab.com/bmwinger/openMMM/tree/dev
The repository can be found here: https://gitlab.com/bmwinger/openmmr

The repo is built around what I'm calling the pybuild format, a package format that I've designed that is based on Gentoo's Ebuild format. A major difference is that pybuilds are valid python source files, which sped up development considerably as it there was no need to write a parser.

There are currently two pybuilds in the repository.
abandoned-flat-2.0 - a simple pybuild
herbalism-for-purists-1.22 - a more complex pybuild that patches version 1.22 on top of version 1.21. Also specifies ESPs in such a way that the correct ones can automatically be enabled depending on what combination of bloodmoon and tribunal you have (not actually implemented yet, but will be soon Done!).

Both of these can be installed with the following command:

Code: Select all

./openmmm.py --sync abandoned-flat herbalism-for-purists
The '--sync' updates the local version of the repository and is not required to install mods.

They can be uninstalled by providing the '-c' flag

Code: Select all

./openmmm.py -c abandoned-flat herbalism-for-purists
Also note that until you edit the openmmm.cfg file it will complain if you try to install any mods that depend on morrowind (both of them). You just need to add the USE flag 'morrowind' to the config (optionally 'bloodmoon' and 'tribunal', but those won't do anything yet).
When you first run openmmm it will create a config file and tell you where it is located.

You want your config file to look something like this:

Code: Select all

[general]
USE=morrowind bloodmoon tribunal
ARCH=openmw
(ARCH is also unimplemented, but will eventually allow you to alternatively select tes3mp, with mods, as appropriate, being marked as being unstable or non-functional with tes3mp).

This version of OpenMMM should also work on other platforms.
Edit: Found the documentation for paths, and dev is now updated to use those paths. I would still like someone to verify that these paths do actually work. If they don't and OpenMMM tries to use the wrong path, you can specify the correct one in openmmm.cfg using 'OPENMW_CONFIG=path_to_config'.

In addition to patool (a dependency already in 1.0) and appdirs, colorama is also used for cross-platform console colour support, and will also need to be installed. There now is a pip-format requirements.txt file in the repo, so you can

Code: Select all

pip install -r requirements.txt
if you want to pull in the dependencies automatically.

Note that the dev branch has not been thoroughly tested. It probably won't mess up your openmw install, but note that if you already had one of the mods installed beforehand, then uninstalling using OpenMMM will remove references to the esps and bsas in those mods from the openmw.cfg file. Otherwise, you should be able to try it out beside an existing openmw configuration without any problems.

Next, in addition to finishing up the missing features from v2.0, I plan on working on scripts to automatically generate pybuild files using, in part, the autodiscovery code from openmmm v1.0. The idea is that when adding mods to the repository we can start with the autogenerated pybuilds, which in many cases will only need the descriptive fields filled out, and configure them manually if necessary.

Update: Basic importing support has been added. Mods can be imported by creating a file with a each line formatted like: 'category/modname-version download_url' and passing it to openmmm via the --import flag. The generated pybuilds will be automatically placed in a user-overlay repo within the openmmm local directory (where the openmmr local repo and mods are by default).
Note that tes3cmd must be in your path for the automatic esp dependency detection to work (dependencies must also be already in the repo or imported beforehand).
User avatar
MinerMan60101
Posts: 24
Joined: 09 Sep 2017, 15:40
Location: California: U.S.A.

Re: OpenMW Mod Manager

Post by MinerMan60101 »

So, does the integrated filepatcher look for mods dependent on TR_Data.esm and patch them?
User avatar
bmw
Posts: 81
Joined: 04 Jan 2019, 19:42
Contact:

Re: OpenMW Mod Manager

Post by bmw »

MinerMan60101 wrote: 20 Jan 2019, 22:34 So, does the integrated filepatcher look for mods dependent on TR_Data.esm and patch them?
The TR filepatcher isn't integrated (yet), though I have created a cli version of it so that it can be (linked to in the first post).
The plan is that in OpenMMM v2 you can specify in the pybuild the files that need to be patched in the prepare function.

e.g. the following code, if added to a pybuild, would run the patcher whenever the mod is installed. All that needs to be added is the tr-patcher function, which can be easy implemented by calling the cli version of the patcher.

Code: Select all

def prepare(self):
    tr_patcher(os.path.join(SOURCE_DIR, CM, "EspName.esp"))
This can be added for mods that are known to need it, however automatically detecting such files would be useful for the pybuild generator and reduce the work required to add such mods.
I don't actually know if there is a definite way of auto-detecting such mods, though I would be glad to learn of one. Are you asking because every mod that depends on TR_Data.esm requires patching? Are there possibly any exceptions to this rule?
The pybuild generator already detects masters for the purpose of detecting dependencies. It would be relatively simple to have it add calls to tr-patcher for mods that rely on TR_Data.esm.

Taking a look at the FilePatcher code in tr-patcher (I may have written a cli, but I didn't take that deep of a look at the code), I notice that it removes 'TR_Data.esm', 'PC_Data.esm' and 'Skyrim_Data.esm' from the masters list. Would this be a more complete list of masters, or would mods that depend on the others also depend on 'TR_Data.esm', making them redundant?
User avatar
bmw
Posts: 81
Joined: 04 Jan 2019, 19:42
Contact:

Re: OpenMW Mod Manager

Post by bmw »

A number of updates to report:

Roadmap
First of all, OpenMMM v2.0 is mostly done, with several smaller things left to implement. The current state can be viewed on the v2.0 milestone on gitlab: https://gitlab.com/bmwinger/openMMM/milestones/3. Note that there are many things that were implemented that weren’t included in the issue tracker, so the number of things that have been completed is lower than it should be, however I think that I have more or less all the remaining features written up as issues now.
Once the remaining features are done I’ll make a release candidate and start work on building up the repository (a large task in itself) and populating a wiki describing the various features. After adding a reasonable number of common mods known to work with OpenMW and when I’ve used the release candidate enough to ensure that it’s reasonably capable, I’ll declare v2.0 stable and release it on the python package index. I’m not personally going to maintain packages for every Linux distribution, hombrew etc., but by having it there it will be widely available.

dcv
I've written a script called the Directory Conflict Viewer: https://gitlab.com/bmwinger/dcv. It’s a curses-based command line program that can interactively view file differences between a list of directories. The dev branch of openmmm also contains an openmw-conflicts script which feeds the data directories listed in openmw.cfg to dcv. It currently doesn’t check whether or not conflicting files are actually the same, but I’ll probably implement that once I start using it more often.

omwcmd
I’ve also created omwcmd: https://gitlab.com/bmwinger/omwcmd. It’s a command line program written in Rust that uses esplugin (https://github.com/Ortham/esplugin), a native library for reading elder scrolls plugin files. At the moment its only features are to print out the masters of a specified file, but I will happily take feature requests if there is anything else you want from such a tool. Basically I implemented this so that I did not have to deal with the extremely slow speed of tes3cmd, as getting masters from parsing `tes3cmd dump` was taking a very long time, particularly for large plugins. I’m thinking that omwllf could possibly be re-implemented as part of it so that its levelled list files could be generated more quickly.

Sorting Plugins and Data Directories
With respect to OpenMMM itself, I’ve recently implemented automatic data directory and plugin sorting. Since they both use the same mechanism I decided to implement both rather than continuing to use mlox for plugin sorting. This gives more control over the sorting rules and I would also prefer to integrate ordering information into the repository than to use an external list like mlox does. That goes for both mlox’s base rules and the user rules, as in this system there aren’t currently user rules, though I probably will add a way of adjusting the priority of esps for certain cases.
Ordering is done using both the masters of an esp and a mod’s TIER field, which specifies either a number, or the special tier ‘z’ (the default), which indicates priority in the plugin order and data directory order. I.e. when sorting plugins, masters will always be enforced, but the TIER distinguishes the ordering of mods once their masters are already in the list. This is similar to mlox’s nearstart and nearend rules, but allows for more levels of complexity (Tier 0 would be nearstart and Tier z would be nearend).
Data directories work similarly, but use dependencies rather than masters.

Basic outline of tiers:
Tier 0 – base mods such as Morrowind itself
Tier 1 – mods that directly modify Tier 0 mods but little else, such texture and mesh replacers
Tier 2 – large mods that have many dependencies such as Tamriel Rebuilt (not sure if this is really necessary)
Tiers 3+ – reserved for later use
Tier z – all other mods (default if no tier is specified)

The basic idea I had for sorting, without something akin to mlox’s userlist, is that sorting rules fall into two categories: those that are always necessary, i.e. rules that if not enforced would break things, and rules that choose between two different implementations of something (i.e. two mods that both modify a certain record but are otherwise compatible with each other). The former can be stored in the pybuild files for those mods, while the latter could be encoded using flags of some sort that the user sets.
I can’t think of an example for Morrowind, but in Oblivion, Less Annoying Magic Experience and Supreme Magica can be used together, and you get spells from both, but the order of the esps determines which mod’s changes to the spell system are applied. The idea for openmmm is that you would set either a “supreme-magica” flag or a “lame” flag, and that would reduce the priority of whichever mod’s flag is set (i.e. it would be lower in the list).
If you know of mods that would not be covered by these categories, or that you think would otherwise not work properly with this sort of system, please let me know.

Please note that the dev branch is still quite unstable, as I sometimes make changes that cause the program to break when encountering files from previous versions of OpenMMM. You are certainly free to try it out, but you may encounter issues from time to time. However once 2.0 is complete, changes will be made in a way that ensures things don’t break, using migration code if necessary.

Naming
I’m also considering renaming the project, as I’ve never really liked the name OpenMMM. I’m not great at naming things and building this off an existing project allowed me to avoid having to come up with a name, however if such a change is to be made I would prefer to do so prior to releasing 2.0. One possible name that I’d entertained was Teleport, to keep with the naming scheme of Portage (and BSD Ports, that Portage was named after), while having a distinct Fantasy twist, but it’s too common of a name and is generally already taken. I’m gravitating towards modports (sort of like MacPorts) or portmod. Probably the latter.
Any suggestions would be appreciated.

Finally, it turns out that GitLab supports creating issues by email. I’ve added the addresses to the original post on this thread. Feel free to use those for submitting bug reports and feature requests.

Edit: Bump Maps and Patches
I forgot to mention that I also figured out a way of automatically fixing issues with bump maps: fix them manually and distribute binary patches in the repository using something like bsdiff (which was chosen since it's fairly widely available and also apparently can be built for Windows with a small tweak). This could also be used to create and apply patches for files needing patching with the tr-patcher, making tr-patcher no longer be a necessary runtime dependency unless you're importing mods.
jmelesky
Posts: 47
Joined: 02 Mar 2017, 20:52

Re: OpenMW Mod Manager

Post by jmelesky »

bmw wrote: 27 Feb 2019, 04:02 I’m thinking that omwllf could possibly be re-implemented as part of it so that its levelled list files could be generated more quickly.
I've considered RIIR for omwllf, so that sounds fine to me.

Oh, and I've got your PR, just haven't had a chance to examine closely and merge -- work has been swamping me lately.
User avatar
bmw
Posts: 81
Joined: 04 Jan 2019, 19:42
Contact:

Re: OpenMW Mod Manager

Post by bmw »

jmelesky wrote: 12 Mar 2019, 21:22 I've considered RIIR for omwllf, so that sounds fine to me.

Oh, and I've got your PR, just haven't had a chance to examine closely and merge -- work has been swamping me lately.
I actually looked a little closer into esplugin, and it turns out that it's a little less complete that I had thought. Particularly, it doesn't seem to support reading/writing arbitrary records, something that would be necessary for omwllf. It mainly just supports, as admittedly the author says in the readme, a few specific features for the benefit of libloadorder and LOOT.

It's still the most capable native esp library, however a Rust port of omwllf would require getting support for reading all the records in an esp into esplugin. I'd be happy to make the necessary changes to esplugin myself and submit a PR for them, however given that I have other priorities at the moment, such a port is still pretty low priority.
jmelesky
Posts: 47
Joined: 02 Mar 2017, 20:52

Re: OpenMW Mod Manager

Post by jmelesky »

bmw wrote: 19 Mar 2019, 22:24 I actually looked a little closer into esplugin, and it turns out that it's a little less complete that I had thought. Particularly, it doesn't seem to support reading/writing arbitrary records, something that would be necessary for omwllf. It mainly just supports, as admittedly the author says in the readme, a few specific features for the benefit of libloadorder and LOOT.
I noticed that, too. Fortunately, I've already done that work in python, so a rewrite likely still wouldn't be too hard, but I'd want to do it the "right" way: contribute to esplugin or write a new library, then base the new code off of that crate.

Like you, time is tight for me, so I'm glad that it works right now.
User avatar
bmw
Posts: 81
Joined: 04 Jan 2019, 19:42
Contact:

Re: OpenMW Mod Manager

Post by bmw »

jmelesky wrote: 20 Mar 2019, 18:46 I noticed that, too. Fortunately, I've already done that work in python, so a rewrite likely still wouldn't be too hard, but I'd want to do it the "right" way: contribute to esplugin or write a new library, then base the new code off of that crate.

Like you, time is tight for me, so I'm glad that it works right now.
Indeed. I also realise that I've said a lot criticizing the speed of omwllf, but I should really emphasize that I think it's great and thanks so much for writing it!

I've recently actually put in a little work on updating how Portmod handles omwllf. Basically I've created a pybuild for it and implemented a "rebuild" feature that automatically adds it to the @rebuild set whenever a mod containing a plugin is installed, removed or re-installed. It will then warn you whenever anything's in the rebuild set, indicating that you can rebuild those mods using `mwmerge @rebuild`. That way it won't automatically rebuild OMWLLF.omwaddon every time anything's changed, but it will still make it very clear when it needs to be rebuilt.
I also wrapped the output in a progressbar, partly because the progressbar gives a clear indication of how many plugins need to be processed, and partly because the messages omwllf outputs at the end don't really apply to portmod as it does those things automatically.

I've also officially renamed the project to Portmod. Check out the new thread here!
Post Reply