Portmod - Mod Manager

Learn about OpenMW mod compatibility, check for problematic mods, discuss issues, and give us feedback about your experience with modded OpenMW.
darkbasic
Posts: 99
Joined: 18 Apr 2016, 15:45
Contact:

Re: Portmod - Mod Manager

Post by darkbasic » 04 Sep 2019, 08:55

Awesome, thanks for your work :)

darkbasic
Posts: 99
Joined: 18 Apr 2016, 15:45
Contact:

Re: Portmod - Mod Manager

Post by darkbasic » 05 Sep 2019, 10:12

I'm currently testing it, may I suggest adding sets for the http://modding-openmw.com mod groups, like "Total Overhaul" and "Expanded Vanilla"? :)

User avatar
bmw
Posts: 28
Joined: 04 Jan 2019, 19:42
Contact:

Re: Portmod - Mod Manager

Post by bmw » 20 Oct 2019, 03:36

darkbasic wrote:
05 Sep 2019, 10:12
I'm currently testing it, may I suggest adding sets for the http://modding-openmw.com mod groups, like "Total Overhaul" and "Expanded Vanilla"? :)
Sorry for the late response, I've been rather busy and haven't had much spare time.
Yes I definitely will add such sets, in one form or another. It occurred to me when thinking about this, that, particularly for large collections of mods that could change significantly over time, we really need some way of handling updates for people who don't want the changes to break their save files. There's already the feature, or at least the idea, as it's unfortunately not yet implemented, to roll back changes to your mod configuration, but it would be useful to have a pre-emptive way of avoiding breaking changes. I've put together some ideas here, such as making it possible to just do minor updates (with the option to ignore this for mods without plugins).

I think I'd mentioned something about using sets rather than metapackages/metamods previously, but it occurs to me that metamods would be much more powerful, mainly since we could take advantage of both use flags and versioning, which sets lack (and with versioning they could take advantage of the above safe update mechanisms, and it would be possible to continue to provide support for old versions of the metamod). The only advantage sets have is simplicity.

I've been working on adding all the "Expanded Vanilla" mods from modding-openmw, and am somewhere around two thirds of the way done, maybe closer to three quarters; once I'm done I'll create a metamod for them. Total Overhaul will be a lot more work, but it will be done eventually.

siemreeap
Posts: 2
Joined: 23 Oct 2019, 20:37

Re: Portmod - Mod Manager

Post by siemreeap » 23 Oct 2019, 20:56

This question will be trivial to anyone who has experience with Python. It's not to an ignorant gamer like me. The wiki on gitlab says:
The first thing you should do after installing is to synchronize the mod repository. This can be done through portmod via omwmerge --sync.
How exactly do I run portmod or the omwmerge command? I ask here because I can't find any better place.

User avatar
bmw
Posts: 28
Joined: 04 Jan 2019, 19:42
Contact:

Re: Portmod - Mod Manager

Post by bmw » 23 Oct 2019, 23:02

The only interface is via the command line, so you need to open a terminal to interact with it (or command prompt, or whatever you want to call it).

Importantly, you almost certainly want to make sure that the directory it's installed in is in your PATH environment variable (otherwise the commands would only work by calling the executable by its path). A simple way to check for this would be just to try running `omwmerge` in a terminal and see if the executable is found. If it is, then you don't need to do anything, otherwise, you need to modify your PATH. If you're on Windows and don't know how to modify your PATH environment variable, I'm hoping you can look that up, as I'm not familiar with the process and there should be plenty of guides out there.
The location of the install directory will vary depending on how you installed it and your platform. If you used `pip install --user ...` then it will either be in `~/.local/bin` on Linux and OSX, or `C:\Users\Username\AppData\Roaming\Python36\Scripts` on Windows (this presumably varies depending on the version of python you're using) (details from here). Without `--user` it probably should install it to a location that is already in your path, but I'm not precisely sure where except on Linux (which should be `/usr/bin`)

It should then be able to be run by opening a terminal and entering the appropriate command.

I'll add this information to the wiki too, as I'm sure you won't be the only person with this question, or a similar one.

siemreeap
Posts: 2
Joined: 23 Oct 2019, 20:37

Re: Portmod - Mod Manager

Post by siemreeap » 24 Oct 2019, 19:51

bmw wrote:
23 Oct 2019, 23:02
The only interface is via the command line, so you need to open a terminal to interact with it (or command prompt, or whatever you want to call it).

Importantly, you almost certainly want to make sure that the directory it's installed in is in your PATH environment variable (otherwise the commands would only work by calling the executable by its path). A simple way to check for this would be just to try running `omwmerge` in a terminal and see if the executable is found. If it is, then you don't need to do anything, otherwise, you need to modify your PATH. If you're on Windows and don't know how to modify your PATH environment variable, I'm hoping you can look that up, as I'm not familiar with the process and there should be plenty of guides out there.
The location of the install directory will vary depending on how you installed it and your platform. If you used `pip install --user ...` then it will either be in `~/.local/bin` on Linux and OSX, or `C:\Users\Username\AppData\Roaming\Python36\Scripts` on Windows (this presumably varies depending on the version of python you're using) (details from here). Without `--user` it probably should install it to a location that is already in your path, but I'm not precisely sure where except on Linux (which should be `/usr/bin`)

It should then be able to be run by opening a terminal and entering the appropriate command.

I'll add this information to the wiki too, as I'm sure you won't be the only person with this question, or a similar one.
Thank you. I should have said it but I am running Kubuntu 18.04. portmod got installed in ~/.local/lib/python3.6/. However it started working after I restarted my PC and I didn't need to change PATH at all.

edit: it seems that omwmerge informs you before installing some mods (like Patch for Purists) that they require you to download the .7z manually and place it in ~/.cache/portmod/downloads/ . This warning doesn't display when installing Correct UV Mudcrabs which won't successfully install otherwise.

Jimmon
Posts: 4
Joined: 30 Jan 2018, 16:34

Re: Portmod - Mod Manager

Post by Jimmon » 07 Nov 2019, 23:19

what version of python are you developing this on? i tried python 3.6 and i keep getting warnings like
WARNING: cannot import name 'getegid'
WARNING: Could not load pybuild "morrowind-1.6.1820-r1"
are these warnings normal? or do i need a different version of python?
--edit 1 --
it would also appear i'm unable to download most mods in the repo

User avatar
bmw
Posts: 28
Joined: 04 Jan 2019, 19:42
Contact:

Re: Portmod - Mod Manager

Post by bmw » 16 Nov 2019, 03:33

Well, Linux python, but the aim is for it to work on all platforms (python >= 3.6 is required). Your issue appears to be one with the sandbox, meaning it affects any mod, hence why all mods fail to install, and is due to an oversight on my part.

Basically I assumed the existence of several parts of the os module that only exist on Unix systems, i.e. not on Windows (which I assume you must be using). This is actually in the python documentation, but I must have missed it originally. The underlying issue is that we only run automated tests on Linux, as GitLab's infrastructure doesn't support CI on other platforms as far as I know. Combined with the fact that I personally don't own any machines running anything other than Linux, Windows testing is done entirely by users such as yourself who notice issues and report them.

I've merged a fix into master that should fix this. You'll need to install the latest development version if want to test the change. It's possible that there are other issues though, I think I got all the stuff in the os module not available on Windows, but I could have also missed things elsewhere.

Also, @siemreeap in case you didn't see, I fixed the issue with correct-uv-mudcrabs, thanks for pointing that out.

walkingwounded
Posts: 2
Joined: 02 Dec 2019, 10:51

Re: Portmod - Mod Manager

Post by walkingwounded » 02 Dec 2019, 11:40

I tried to get it running under under windows 10 64bit with python 3.8.0 and also 3.7.5 (both amd64 and x86 versions) without success.

I can run these command before I hit a wall:
pip install pip3 install git+https://gitlab.com/portmod/portmod
pip install portmodmigrate
omwmerge --sync
omwselect profile set 4
-note: have the base plus both expansions installed plus the latest official patch for BM (retail)

When I run "omwmerge --update --deep --newuse @world" i get different errors
-3.8.0 = nothing (as in "it returns without errors)
-but "omwmerge -s" results in os import failures reported for features not found in windows (PRIO_PROCESS, etc) throws errors for each mod scanned (sorry, lost my notes, best I can recall is it ending with ...[ARCH])
-3.7.5 = same import errors as for 3.8.0 plus these two:
--WARNING: Could not load pybuild "morrowind-1.6.1820-r1"
--ERROR: Unable to find mod for atom base/morrowind

Further: using python 3.7.5 and 3.8.0
-If I comment out lines for missing features in C:\Program Files\Python37\Lib\site-packages\portmod\safemodules\os\__init__.py
code lines:
26: #PRIO_PROCESS,
27: #PRIO_PGRP,
28: #PRIO_USER,
29: #uname,
40: #sync,
93: #chown..
99: #mkinfo..
100: #pathconf...
111: #statvfs...

-I can can get it down to this (which is as far as I can go with my google skillset)
WARNING: Unable to load restricted module winreg
WARNING: Could not load pybuild "morrowind-1.6.1820-r1"
ERROR: Unable to find mod for atom base/morrowind

I am guessing there is a registry read to find the game installation path. In my case "c:\morrowind".

I'm going to try with the last of the 3.6 releases (3.6.8 I think) and see what happens. May be a deprecation or regression issue.

I understand your access to windows platforms may be limited, but if you are proficient with virtualization you could fire up a windows 10 trial VM and use it for testing until it expires, then rinse and repeat. I think you can slmgr.exe to extend the time (google it).

walkingwounded
Posts: 2
Joined: 02 Dec 2019, 10:51

Re: Portmod - Mod Manager

Post by walkingwounded » 02 Dec 2019, 13:03

Update: (I hope this helps)
(elevated administrator cmd "terminal")
only py 3.6.8 installed

Outputs:

>py -V
Python 3.6.8

>pip3 install git+https://gitlab.com/portmod/portmod
-no errors
>pip3 install portmodmigrate
-no errors


>pip list
Package Version
---------------- --------------------
appdirs 1.4.3
attrs 19.3.0
black 19.10b0
Click 7.0
colorama 0.4.1
fuzzywuzzy 0.17.0
gitdb2 2.0.6
GitPython 3.0.5
pathspec 0.6.0
patool 1.12
pip 19.3.1
portmod 2.0a4.dev19+gf53e12b
portmodmigrate 0.1.2
progressbar2 3.47.0
python-utils 2.3.0
pywin32 227
PyYAML 5.1.2
regex 2019.11.1
RestrictedPython 5.0
setuptools 40.6.2
six 1.13.0
smmap2 2.0.5
toml 0.10.0
typed-ast 1.4.0

*Note: same issue with the 2.0a3 release version

>omwmerge --sync
Syncing repo openmw...
Done syncing repo openmw.

>omwselect profile set 4
(no errors)

>omwmerge --update --deep --newuse @world
WARNING: cannot import name 'PRIO_PROCESS'
WARNING: Could not load pybuild "morrowind-1.6.1820-r1"
ERROR: Unable to find mod for atom base/morrowind

-after masking out unsupported platform features in __init.py__ (C:\Program Files\Python36\Lib\site-packages\portmod\safemodules\os\__init__.py)
>omwmerge --update --deep --newuse @world
WARNING: Unable to load restricted module winreg
WARNING: Could not load pybuild "morrowind-1.6.1820-r1"
ERROR: Unable to find mod for atom base/morrowind


Thoughts:
-I really wanted this to work (currently I'm using MO2 with the export plugin, but it keeps dropping the .omwaddon files and some mods do not like that at all)
-the wiki says to find a command line sandbox tool, but no suggestions (I get it, open source all the things), even a single "this is an example, but up to you if you trust them or not" might help. Windows users consider closed source to be 'normal', and are not allergic to it. Probably not be related to the issue.
-this is all WAY out of my league/skillset
-consider bypassing the security/sandbox stuff for windows with big old "at your own risk, are you really sure you trust it?" warnings. I get and TRULY appreciate the reason, but square pegs do not fit round holes (unless you use a bigger hammer)
-perhaps embed 'sanity checks' in the package/mod processing stage
-perhaps a windows 'read/write path whitelist' file and validate reads and writes against it (placed in the "my games\openmw" user folder in case users have edge cases)? might be slow, but for one-time hits not terrible, things will only need to be written to a few places (\download, staging/temp and \mods folders, "<docs path>\my games\openmw", maybe a few more). Alert the user for anything 'unexpected' (or anything outside of the %userprofile% path).
-perhaps if reading and writing can be monitored just dump reads/writes to an audit log that is launched at the end? won't prevent what happened but if you can see it you can report it

Post Reply