OpenMW Mod Manager

Learn about OpenMW mod compatibility, check for problematic mods, discuss issues, and give us feedback about your experience with modded OpenMW.
User avatar
bmw
Posts: 8
Joined: 04 Jan 2019, 19:42
Github profile: https://gitlab.com/bmwinger
Contact:

Re: OpenMW Mod Manager

Post by bmw » 08 Jan 2019, 04:34

psi29a wrote:
07 Jan 2019, 19:18
I put a lot time into porting NifScope to Linux, give it a try. :)

The beta versions should work out of the box.

(Hint, also a member of the niftools team)
It was the same issue as https://github.com/niftools/nifskope/pull/147: I'm using gcc 8.2 and Qt5 5.10. It might be useful to list dependencies in the readme.
I've got it working now using the change in the pull request.
jmelesky wrote:
07 Jan 2019, 21:21
What kind of times are you seeing, and with what length of mod lists? I guess I'm also curious what the more common use case is: add a bunch of mods occasionally, or add mods one at a time frequently? If it's the former, I'm inclined away from spending much time optimizing.
With 185 files being processed I'm getting a runtime of about 79 seconds. Not terrible if you only run it once after installing a bunch of mods, but in the context of installing them one at a time (if, for example, you wanted to test compatibility of each one before proceeding), it's pretty slow.
By comparison just running a sha512sum on all those files takes about 1.7 seconds (note ssd, and relevance below).
jmelesky wrote:
07 Jan 2019, 21:21
That said, it should be possible to speed it up, though my likely next step would be a rewrite in not-python (and also not-c++ -- sorry, all).
The trouble with converting it to not-python (also not fond of c++. Rust maybe? I like Rust), is that even if the result is an order of magnitude faster, it still may not scale well, seeing as the issue is caused by lots of mods. Then again, I ran a profile of it and the bulk of the work is being done in readSubRecord, almost certainly the string slices, since apparently Python does-slice by-copy, i.e. ever time you make a slice of a string in readSubRecord it's creating a new string.
Using a bytearray and consuming the processed elements rather than returning a slice of the remaining data got the total time down to 25 seconds (16 when compiled with cython). Still not amazing, but much better for the trivial change. I also checked and the two resulting files were identical save for the list of mods in the DESC, which had a different order.
Not-python could admittedly do better still by using references instead of slices for all the operations in readSubRecord, but it would be hard to tell how much better without trying it.

Relevant lines from the profile (97.996 seconds when profiling):

Code: Select all

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
 14377555   75.323    0.000   81.510    0.000 omwllf.py:103(readSubRecord)
   220616   11.396    0.000   96.510    0.000 omwllf.py:111(readRecords)
      185    1.343    0.007   97.854    0.529 omwllf.py:138(getRecords) 
After the modification (37.913 seconds when profiling):

Code: Select all

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
 14377555   17.016    0.000   24.080    0.000 omwllf.py:103(readSubRecord)
   220616    8.728    0.000   36.079    0.000 omwllf.py:112(readRecords)
      185    1.353    0.007   37.432    0.202 omwllf.py:138(getRecords) 
Virtually all the time is still being spent in getRecords, but substantially less in total.
jmelesky wrote:
07 Jan 2019, 21:21
Realistically, I don't think that's possible without writing a status file out somewhere else, which would complicate things.
Basically this is what I would suggest to improve on the above change without rewriting in a different language. Write a file containing the files used to build the merged levelled lists that contains their names and hashes (e.g. with sha512sum) and compare with the files you collect every time the mod is run. Then you'd know if a file has been removed, modified or added and can update the merged file to reflect that. What might also be worth trying is writing a cache of the records in a more python friendly format for each file (marshal is apparently quite fast) and import those for the files that haven't changed, given that reading all the records is the bottleneck.

Still though, you might be right that using a language other than python (specifically, where we can avoid costly copying of strings) is the best way forward (if any); the time for everything other than getRecords is very small, and efficiently written the overhead for that shouldn't be much more than the i/o cost (which is also necessary if caching the records).

Pull request for the small change from above incoming.

User avatar
bmw
Posts: 8
Joined: 04 Jan 2019, 19:42
Github profile: https://gitlab.com/bmwinger
Contact:

Re: OpenMW Mod Manager

Post by bmw » 13 Jan 2019, 20:31

The dev branch of OpenMMM now contains work that I've been doing on version 2.0, a complete rework of the original that uses a mod repository to allow for completely automated installation, with configuration mainly taking place on a global level. There will also be capability for local configuration of mods that have non-compatibility-related options (though any configuration shared between mods, e.g. texture size, I will try to eventually implement a global way of configuring).

The code can be found here: https://gitlab.com/bmwinger/openMMM/tree/dev
The repository can be found here: https://gitlab.com/bmwinger/openmmr

The repo is built around what I'm calling the pybuild format, a package format that I've designed that is based on Gentoo's Ebuild format. A major difference is that pybuilds are valid python source files, which sped up development considerably as it there was no need to write a parser.

There are currently two pybuilds in the repository.
abandoned-flat-2.0 - a simple pybuild
herbalism-for-purists-1.22 - a more complex pybuild that patches version 1.22 on top of version 1.21. Also specifies ESPs in such a way that the correct ones can automatically be enabled depending on what combination of bloodmoon and tribunal you have (not actually implemented yet, but will be soon Done!).

Both of these can be installed with the following command:

Code: Select all

./openmmm.py --sync abandoned-flat herbalism-for-purists
The '--sync' updates the local version of the repository and is not required to install mods.

They can be uninstalled by providing the '-c' flag

Code: Select all

./openmmm.py -c abandoned-flat herbalism-for-purists
Also note that until you edit the openmmm.cfg file it will complain if you try to install any mods that depend on morrowind (both of them). You just need to add the USE flag 'morrowind' to the config (optionally 'bloodmoon' and 'tribunal', but those won't do anything yet).
When you first run openmmm it will create a config file and tell you where it is located.

You want your config file to look something like this:

Code: Select all

[general]
USE=morrowind bloodmoon tribunal
ARCH=openmw
(ARCH is also unimplemented, but will eventually allow you to alternatively select tes3mp, with mods, as appropriate, being marked as being unstable or non-functional with tes3mp).

This version of OpenMMM should also work on other platforms.
Edit: Found the documentation for paths, and dev is now updated to use those paths. I would still like someone to verify that these paths do actually work. If they don't and OpenMMM tries to use the wrong path, you can specify the correct one in openmmm.cfg using 'OPENMW_CONFIG=path_to_config'.

In addition to patool (a dependency already in 1.0) and appdirs, colorama is also used for cross-platform console colour support, and will also need to be installed. There now is a pip-format requirements.txt file in the repo, so you can

Code: Select all

pip install -r requirements.txt
if you want to pull in the dependencies automatically.

Note that the dev branch has not been thoroughly tested. It probably won't mess up your openmw install, but note that if you already had one of the mods installed beforehand, then uninstalling using OpenMMM will remove references to the esps and bsas in those mods from the openmw.cfg file. Otherwise, you should be able to try it out beside an existing openmw configuration without any problems.

Next, in addition to finishing up the missing features from v2.0, I plan on working on scripts to automatically generate pybuild files using, in part, the autodiscovery code from openmmm v1.0. The idea is that when adding mods to the repository we can start with the autogenerated pybuilds, which in many cases will only need the descriptive fields filled out, and configure them manually if necessary.

Update: Basic importing support has been added. Mods can be imported by creating a file with a each line formatted like: 'category/modname-version download_url' and passing it to openmmm via the --import flag. The generated pybuilds will be automatically placed in a user-overlay repo within the openmmm local directory (where the openmmr local repo and mods are by default).
Note that tes3cmd must be in your path for the automatic esp dependency detection to work (dependencies must also be already in the repo or imported beforehand).

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests