Portmod - Mod Manager

Post about your mods, learn about OpenMW mod compatibility, check for problematic mods, discuss issues, and give us feedback about your experience with modded OpenMW.
ezze
Posts: 513
Joined: 21 Nov 2013, 13:20

Re: Portmod - Mod Manager

Post by ezze »

I think I stumbled upon another misbehavior. Basically, if one moves a local repository, updates the configuration to point to the new directory. omwmerge breaks for it is unable to find the files. It seems the full path is cached or something like that.

Besides, would't be better to have all textual files in one directory so one can easily keep track of everything. The program data seems a bit scattered (config, local, repo...)
User avatar
bmw
Posts: 81
Joined: 04 Jan 2019, 19:42
Contact:

Re: Portmod - Mod Manager

Post by bmw »

ezze wrote: 02 Oct 2020, 01:17 For textures pack I think the rough "bigger before smaller" convention would cover most cases. After all, if you put a small texture pack it is more likely to cover a detail you want to see.
That should already be possible using the existing tier system. If you set a low-priority tier for large texture packs, then everything else will use the high-priority default and override them.
ezze wrote: 02 Oct 2020, 02:31 I think I stumbled upon another misbehavior. Basically, if one moves a local repository, updates the configuration to point to the new directory. omwmerge breaks for it is unable to find the files. It seems the full path is cached or something like that.
Good catch. It looks like the cache invalidation code is currently assuming that the old files still exist. Using relative paths might also be better there so that the cache isn't invalidated when a repository is moved.
ezze wrote: 02 Oct 2020, 02:31 Besides, would't be better to have all textual files in one directory so one can easily keep track of everything. The program data seems a bit scattered (config, local, repo...)
Other than having the config files (which are meant to be edited by users) separate from the data/local files (which aren't), it's not particularly scattered. The metadata repositories can be separate, but by default they are stored within the data files. The only other thing is the cache files, which need to be separate so that it's clear that they aren't a necessary part of the system (useful, but not necessary).
mkienenb
Posts: 6
Joined: 16 Jan 2021, 21:13

Re: Portmod - Mod Manager

Post by mkienenb »

I also finally worked through the hoops to get this closer to working, although some of my final clues came from reading ezze's posts.

It'd be more helpful if the wiki gave a complete working example. Here's the ones I used.

Code: Select all

# select default repo
portmod openmw select repo list
portmod openmw select repo add 0

# select GOTY data files
portmod openmw select repo list
portmod openmw select profile set 4

portmod openmw merge --update --deep --newuse @world

# edit  ~/.config/portmod to define MORROWIND_DIR

# finally figured out a command to list which "patterns" can be installed
portmod openmw search meta | more

# tried to install this one (wanted total-overhaul, but that one is missing)
portmod -v --debug openmw merge i-heart-vanilla-directors-cut
I'm not a python programmer, so it's unclear why this is failing.

Code: Select all

Traceback (most recent call last):
  File "<string>", line 21, in <module>
  File "/usr/lib64/python3.6/site-packages/portmod/wrapper.py", line 252, in main
    args.func(args)
  File "/usr/lib64/python3.6/site-packages/portmod/wrapper.py", line 99, in wrapper_pybuild
    pkg_func(pkg, "pkg_pretend")
  File "/usr/lib64/python3.6/site-packages/portmod/wrapper.py", line 85, in pkg_func
    return func()
  File "/home/minecraft/.local/share/portmod/repos/openmw/common/system/system-1.0.pybuild", line 35, in pkg_pretend
    self.warn(string)
  File "/usr/lib64/python3.6/site-packages/pybuild_/pybuild.py", line 868, in warn
    os.makedirs(warnings_dir, exist_ok=True)
  File "/usr/lib64/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/tmp/portmod/warnings'
Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/portmod/_cli/merge.py", line 141, in merge_main
    emptytree=args.emptytree,
  File "/usr/lib64/python3.6/site-packages/portmod/merge.py", line 304, in configure
    save_state=True,
  File "/usr/lib64/python3.6/site-packages/portmod/loader.py", line 368, in _sandbox_execute_pybuild
    installed=installed,
  File "/usr/lib64/python3.6/site-packages/portmod/loader.py", line 304, in _sandbox_execute
    ) from None
portmod.loader.SandboxedError: /home/minecraft/.local/share/portmod/repos/openmw/sys-bin/tes3cmd/tes3cmd-0.40_pre2.pybuild pretend failed!
ERROR: /home/minecraft/.local/share/portmod/repos/openmw/sys-bin/tes3cmd/tes3cmd-0.40_pre2.pybuild pretend failed!
/tmp/portmod exists and is writable. The filesystem is definitely not read-only.

Code: Select all

minecraft@linux-n7ak:~/Games> ls -ld /tmp/portmod/
drwxr-xr-x 5 minecraft users 4096 Jan 16 14:27 /tmp/portmod/

minecraft@linux-n7ak:~/Games> ls -l /tmp/portmod/
drwxr-xr-x 3 minecraft users 4096 Jan 16 14:19 modules
drwxr-xr-x 2 minecraft users 4096 Jan 16 14:26 pybuild
drwxr-xr-x 5 minecraft users 4096 Jan 16 14:32 sys-bin
So I created /tmp/portmod/warnings as a directory manually and re-ran.

Code: Select all

FileNotFoundError: [Errno 2] No such file or directory: '/tmp/portmod/warnings/sys-bin/tes3cmd-0.40_pre2'

portmod.loader.SandboxedError: /home/minecraft/.local/share/portmod/repos/openmw/sys-bin/tes3cmd/tes3cmd-0.40_pre2.pybuild pretend failed!
ERROR: /home/minecraft/.local/share/portmod/repos/openmw/sys-bin/tes3cmd/tes3cmd-0.40_pre2.pybuild pretend failed!

Code: Select all

mkdir /tmp/portmod/warnings/sys-bin
mkdir /tmp/portmod/warnings/sys-bin/tes3cmd-0.40_pre2

Code: Select all

IsADirectoryError: [Errno 21] Is a directory: '/tmp/portmod/warnings/sys-bin/tes3cmd-0.40_pre2'

Code: Select all

minecraft@linux-n7ak:~/Games> rm /tmp/portmod/warnings/sys-bin/tes3cmd-0.40_pre2
A search turns up:

Code: Select all

tes3cmd | Tamriel Rebuilt.  Tes3cmd is a command-line tool used for editing, inspecting and cleaning mods. Download ...
So I dropped back to i-heart-vanilla:

Code: Select all

OSError: [Errno 30] Read-only file system: '/tmp/portmod/warnings/sys-bin/virtualenv-0'
A search turns up:

Code: Select all

virtualenv is a tool to create isolated Python environments.
At this point, I'm at a loss on how to continue debugging.
User avatar
bmw
Posts: 81
Joined: 04 Jan 2019, 19:42
Contact:

Re: Portmod - Mod Manager

Post by bmw »

mkienenb wrote: 16 Jan 2021, 21:53 I also finally worked through the hoops to get this closer to working, although some of my final clues came from reading ezze's posts.

It'd be more helpful if the wiki gave a complete working example.
I will add one soon.
mkienenb wrote: 16 Jan 2021, 21:53 At this point, I'm at a loss on how to continue debugging.
The main thing I think you're missing when debugging is that Portmod sandboxes package scripts (there's a wiki page that gives a brief overview), and the only write permissions given at the point when the exception is raised are to a particular temporary directory, which it turns out is not the one which is used by the code which stores warnings so that they can be displayed later and not lost in the output (there was a significant change to how the permissions work since when the code to produce the warning was written, and I guess this was missed).

Obviously it will need to be fixed, and I'll get that done shortly, but if you want a quick workaround you should be able to bypass the warning. Basically the "sys-bin" packages just check to see if an executable (the one in the name) is already available on the system, and if not print a warning to tell you which executable you need. For tes3cmd you can even use "bin/tes3cmd", which will install it, but for virtualenv you'll need to get it from your package manager.
mkienenb
Posts: 6
Joined: 16 Jan 2021, 21:13

Re: Portmod - Mod Manager

Post by mkienenb »

Thanks for your quick reply!

I was able to progress the i-heart-vanilla-directors-cut using these commands under OpenSUSE Leap 15.2 -- it's still going.

Code: Select all

portmod openmw merge bin/tes3cmd
sudo zypper install paython3-virtualenv
sudo zypper install unrar
I also saw someone on reddit had the same issue 3 days ago and pointed them back to this thread.

Another person there mentioned that they had installed the meta-momw package for total overhaul. That's the one meta package I don't see listed (and the one I most wanted to try). Am I missing it? Is it in a different repo or does it have a different name?
mkienenb
Posts: 6
Joined: 16 Jan 2021, 21:13

Re: Portmod - Mod Manager

Post by mkienenb »

Also, I am willing to help update the wiki if you want to give me access.

And the install hit this error:

Code: Select all

Installing collected packages: fontTools
Successfully installed fontTools-4.18.2
>>> Preparing source in /tmp/portmod/modules/openmw-fonts-0.2.3/work/openmw-fonts-0.2.3/openmw-fonts-0.2.3 ...
>>> Source Prepared
>>> Installing modules/openmw-fonts-0.2.3 into /tmp/portmod/modules/openmw-fonts-0.2.3/image
Traceback (most recent call last):
  File "<string>", line 21, in <module>
  File "/usr/lib64/python3.6/site-packages/portmod/wrapper.py", line 252, in main
    args.func(args)
  File "/usr/lib64/python3.6/site-packages/portmod/wrapper.py", line 95, in wrapper_pybuild
    pkg_func(pkg, "src_install")
  File "/usr/lib64/python3.6/site-packages/portmod/wrapper.py", line 85, in pkg_func
    return func()
  File "/home/minecraft/.local/share/portmod/repos/openmw/common/exec/exec-1.0.pybuild", line 23, in src_install
    patch_dir(os.path.join(self.WORKDIR, self.S), self.D)
  File "/usr/lib64/python3.6/site-packages/portmod/fs/util.py", line 108, in patch_dir
    raise Error(errors)
shutil.Error: [('/tmp/portmod/modules/openmw-fonts-0.2.3/work/openmw-fonts-0.2.3/openmw-fonts-0.2.3/.venv/lib64', '/tmp/portmod/modules/openmw-fonts-0.2.3/image/.venv/lib64', "[Errno 2] No such file or directory: '/tmp/portmod/modules/openmw-fonts-0.2.3/work/openmw-fonts-0.2.3/openmw-fonts-0.2.3/.venv/lib64'")]

Code: Select all

/tmp/portmod/modules/openmw-fonts-0.2.3/image/.venv:
total 20
drwxr-xr-x 5 minecraft users 4096 Jan 16 21:58 .
drwxr-xr-x 4 minecraft users 4096 Jan 16 21:58 ..
drwxr-xr-x 2 minecraft users 4096 Jan 16 21:58 bin
drwxr-xr-x 3 minecraft users 4096 Jan 16 21:58 lib
drwxr-xr-x 3 minecraft users 4096 Jan 16 21:58 share
lib but no lib64 directory there
mkienenb
Posts: 6
Joined: 16 Jan 2021, 21:13

Re: Portmod - Mod Manager

Post by mkienenb »

I'm not all that familiar with python, but it looks like the lib64 issue is related to this:

https://github.com/pypa/virtualenv/issues/1751
lib64 directory is no longer symlinked to lib? #1751

when using virtualenv==16.7.10 or simply python -m venv, the lib64 directory was a symlink to the lib directory as expected. However, in virtualenv==20.0.14, this is no longer the case. Here, lib64 is created as a directory.
My distro's version of virtualenv is version 16.1.0-lp152.2.4. Is there a simple way to upgrade it outside of my distro package manager?
mkienenb
Posts: 6
Joined: 16 Jan 2021, 21:13

Re: Portmod - Mod Manager

Post by mkienenb »

Nevermind.

I remembered that the wiki said to use "pip" to update portmod, and sure enough that also works for upgrading virtualenv.

Code: Select all

sudo pip install virtualenv --upgrade
The processes marches onward....
mkienenb
Posts: 6
Joined: 16 Jan 2021, 21:13

Re: Portmod - Mod Manager

Post by mkienenb »

It looks like I'm now close to getting i-heart-vanilla-directors-cut working.

Code: Select all

minecraft@linux-n7ak:~/Games> portmod openmw merge i-heart-vanilla-directors-cut
Calculating Dependencies...
Done!
These are the packages to be installed, in order:
[N] patches/omwllf-1.0-r1 
[N] meta-momw/i-heart-vanilla-0_p20200809 
[N] meta-momw/i-heart-vanilla-directors-cut-0_p20200809 
Total: 3 packages (0 updates, 3 new, 0 reinstalls, 0 removals),
Size of downloads: 0 MiB

Code: Select all

>>> Preparing source in /tmp/portmod/patches/omwllf-1.0/work/omwllf-1.0 ...
Generating OMWLLF.omwaddon...
  File "omwllf-1.0/omwllf.py", line 131
    yield record
SyntaxError: 'return' with argument inside generator
Traceback (most recent call last):
  File "<string>", line 21, in <module>
  File "/usr/lib64/python3.6/site-packages/portmod/wrapper.py", line 252, in main
    args.func(args)
  File "/usr/lib64/python3.6/site-packages/portmod/wrapper.py", line 93, in wrapper_pybuild
    pkg_func(pkg, "src_prepare")
  File "/usr/lib64/python3.6/site-packages/portmod/wrapper.py", line 85, in pkg_func
    return func()
  File "/home/minecraft/.local/share/portmod/repos/openmw/patches/omwllf/omwllf-1.0-r1.pybuild", line 29, in src_prepare
    f"{python} {archive_path}/omwllf.py -m OMWLLF.omwaddon -d {archive_path}"
  File "/usr/lib64/python3.6/site-packages/portmod/execute.py", line 68, in execute
    proc = subprocess.run(cmd, check=check, stdout=output, stderr=error)
  File "/usr/lib64/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/usr/bin/python', 'omwllf-1.0/omwllf.py', '-m', 'OMWLLF.omwaddon', '-d', 'omwllf-1.0']' returned non-zero exit status 1.
Searching the omwlif issue tracker turns up:

https://github.com/jmelesky/omwllf/issu ... -520172510
Often times a system ends up with a python2 and a python3, and the python will point to one or the other. You can try python3 omwllf.py, or you can just make sure that omwllf.py is executable and run it directly (./omwllf.py, for example) -- it should invoke with the python3 on the system if it exists.

But it does need version 3.3 or later.
And that's my issue:

Code: Select all

minecraft@linux-n7ak:/etc/alternatives> ls -l /usr/bin/python*
lrwxrwxrwx 1 root root     9 Dec  8 04:57 /usr/bin/python -> python2.7
lrwxrwxrwx 1 root root     9 Dec  8 04:57 /usr/bin/python2 -> python2.7
-rwxr-xr-x 1 root root  6304 Dec  8 04:58 /usr/bin/python2.7
lrwxrwxrwx 1 root root     9 Dec 28 10:16 /usr/bin/python3 -> python3.6
-rwxr-xr-x 2 root root 10472 Dec 28 10:16 /usr/bin/python3.6
-rwxr-xr-x 2 root root 10472 Dec 28 10:16 /usr/bin/python3.6m
I'm trying to find a workaround, but is there a way to insure that python3 is being used for portmod? I wonder if some of the other errors I've hit are also related to this problem? Or maybe there's a way to change the exec fork call in python to run omwlif.py directly? Can't remember if that's valid as the last time I messed with C fork/exec calls was 25 years ago.

I was able to use this workaround for OpenSUSE Leap 15.2:

Code: Select all

git clone https://github.com/pyenv/pyenv.git ~/.pyenv
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n  eval "$(pyenv init -)"\nfi' >> ~/.bashrc

# Install the missing headers needed by Python modules

sudo zypper install readline-devel sqlite3-devel libbz2-devel zlib-devel libopenssl-devel

# Install the desired version of Python with pyenv

pyenv install 3.6.3

# Install virtualenv -- this was already done above

sudo zypper install python3-virtualenv

# Create a virtual environment for Python, for example

mkdir ~/pythons
cd ~/pythons
virtualenv -p ~/.pyenv/versions/3.6.3/bin/python3.6 python3.6.3
source ./python3.6.3/bin/activate
And the portmod merge now completes. Just to be sure:

Code: Select all

portmod openmw merge i-heart-vanilla-directors-cut # rebuild warnings
portmod openmw merge @rebuild assets-misc/project-atlas
portmod openmw merge @rebuild patches/omwllf
portmod openmw merge i-heart-vanilla-directors-cut # clean output
User avatar
bmw
Posts: 81
Joined: 04 Jan 2019, 19:42
Contact:

Re: Portmod - Mod Manager

Post by bmw »

Okay, the above discussion of bugs seems to have moved over to GitLab.

As for the wiki, I'm not super fond of the way GitLab has set up its wiki permissions, but I also haven't had much success finding a free wiki hosting service that doesn't require requesting to join the project before making/suggesting changes. As a compromise, I've bypassed the weird way GitLab handles wikis by setting up a mirror of the wiki repository: https://gitlab.com/portmod/portmod-wiki . If you make merge requests to that, it will also update https://gitlab.com/portmod/portmod/-/wikis/
Not that I understand why GitLab doesn't allow a full project page for wikis (at least optionally). There is a git repository behind the wiki, but it can only be accessed by developers of the parent repository.

And the total overhaul package isn't complete yet, though I think it's equivalent to the combination of meta-momw/expanded-vanilla and meta-momw/graphics-overhaul (the latter of which is not complete), plus one other mod, so if you want to get as close as possible for now, you can install those.
Post Reply