Portmod - Mod Manager

Post about your mods, learn about OpenMW mod compatibility, check for problematic mods, discuss issues, and give us feedback about your experience with modded OpenMW.
User avatar
bmw
Posts: 81
Joined: 04 Jan 2019, 19:42
Contact:

Re: Portmod - Mod Manager

Post by bmw »

walkingwounded wrote: 02 Dec 2019, 11:40 -If I comment out lines for missing features in C:\Program Files\Python37\Lib\site-packages\portmod\safemodules\os\__init__.py
code lines:
26: #PRIO_PROCESS,
27: #PRIO_PGRP,
28: #PRIO_USER,
29: #uname,
40: #sync,
93: #chown..
99: #mkinfo..
100: #pathconf...
111: #statvfs...
...
I understand your access to windows platforms may be limited, but if you are proficient with virtualization you could fire up a windows 10 trial VM and use it for testing until it expires, then rinse and repeat. I think you can slmgr.exe to extend the time (google it).
Easy to fix, thanks. And yes, having a way of testing on Windows would be extremely useful. I might try a VM, but I'd rather have a way that can be used for automated testing. I just took a look at getting it working with Wine again, and while the last time I tried I had issues getting git to work, it's working now. Unfortunately while running git manually works, importing GitPython causes an error when running the git command, I don't really know why.
I also just saw that AppVeyor does Windows CI, so maybe I'll try setting that up at some point.
walkingwounded wrote: 02 Dec 2019, 13:03 -after masking out unsupported platform features in __init.py__ (C:\Program Files\Python36\Lib\site-packages\portmod\safemodules\os\__init__.py)
>omwmerge --update --deep --newuse @world
WARNING: Unable to load restricted module winreg
WARNING: Could not load pybuild "morrowind-1.6.1820-r1"
ERROR: Unable to find mod for atom base/morrowind
This is an issue with the RestrictedPython sandbox environment. I forgot that the morrowind build file was accessing the registry, so at the moment the winreg module is being blocked. Probably the best fix (I'll try to throw something together, though I don't have a lot of time at the moment) is to move the read_reg function defined in the morrowind pybuild into portmod and add it to the sandbox whitelist. There really isn't a need for arbitrary registry access anyway.
walkingwounded wrote: 02 Dec 2019, 13:03 -the wiki says to find a command line sandbox tool, but no suggestions (I get it, open source all the things), even a single "this is an example, but up to you if you trust them or not" might help. Windows users consider closed source to be 'normal', and are not allergic to it. Probably not be related to the issue.
No suggestions were provided basically because I didn't know of any good tool at the time that I would want to endorse (basically, as you say, "open source all the things"). Fortunately, that appears to be changing. Sandboxie has been made freeware and the makers have announced plans to release the source code. When I get the chance, I'll take a look at it and try to figure out how to get it to work, but it should be able to be integrated like bubblewrap and sandbox-exec are on Linux/OSX so that all you need to do is install it.
walkingwounded wrote: 02 Dec 2019, 13:03 -consider bypassing the security/sandbox stuff for windows with big old "at your own risk, are you really sure you trust it?" warnings. I get and TRULY appreciate the reason, but square pegs do not fit round holes (unless you use a bigger hammer)
You basically can already. I think that if you leave "SANDBOX_COMMAND" empty it should just execute the commands without a binary sandbox. The python side of the sandbox on the other hand should work, it's just that apparently I made a number of mistakes when setting up the sandbox.
walkingwounded wrote: 02 Dec 2019, 13:03 -perhaps a windows 'read/write path whitelist' file and validate reads and writes against it (placed in the "my games\openmw" user folder in case users have edge cases)? might be slow, but for one-time hits not terrible, things will only need to be written to a few places (\download, staging/temp and \mods folders, "<docs path>\my games\openmw", maybe a few more). Alert the user for anything 'unexpected' (or anything outside of the %userprofile% path).
-perhaps if reading and writing can be monitored just dump reads/writes to an audit log that is launched at the end? won't prevent what happened but if you can see it you can report it
This is actually already being done on the python side of things, or at least the path whitelisting (there's no log). The separate sandbox tool is necessary so that we can execute non-python programs as part of the build process. A good example where executing arbitrary code like this is useful is project atlas (which I haven't actually finished adding to the repo) which comes with scripts that can be used to generate atlases (or at least it comes with windows scripts, but there are bash scripts available elsewhere). I've got a WIP project atlas pybuild that will automatically generate atlases using textures from the mods you have installed.

Thanks for the feedback and debugging work!
Mertz
Posts: 1
Joined: 21 Dec 2019, 12:32

Re: Portmod - Mod Manager

Post by Mertz »

Jemolk wrote: 14 Apr 2019, 00:11
AnyOldName3 wrote: 10 Apr 2019, 20:25 Mator Smash is now the recommended utility instead of Wrye Bash for making that kind of patch for the games it supports.
Only because Wrye Bash doesn't support all the relevant records for those games. Mator Smash doesn't support merging mods into the patch entirely as Wrye Bash does, nor does it have the tweaks functionality of Wrye Bash. It's not that Mator Smash is better than Wrye Bash, but that Mator Smash supports newer files than Wrye Bash but with less features and capabilities.
That used to be true but a lot has changed since Mator Smash's last version in mid-2018. Wrye Bash was last updated on 19 Dec 2019 and has had a ton of updates since then.

Wrye Bash is open source with multiple talented developers that also work on the Unofficial Patches and xEdit teams (and vice versa). the Wrye Mash project is being tested with OpenMW as well.
User avatar
Jemolk
Posts: 237
Joined: 20 Mar 2017, 02:12
Location: Sadrith Mora

Re: Portmod - Mod Manager

Post by Jemolk »

Mertz wrote: 21 Dec 2019, 12:55
Jemolk wrote: 14 Apr 2019, 00:11
AnyOldName3 wrote: 10 Apr 2019, 20:25 Mator Smash is now the recommended utility instead of Wrye Bash for making that kind of patch for the games it supports.
Only because Wrye Bash doesn't support all the relevant records for those games. Mator Smash doesn't support merging mods into the patch entirely as Wrye Bash does, nor does it have the tweaks functionality of Wrye Bash. It's not that Mator Smash is better than Wrye Bash, but that Mator Smash supports newer files than Wrye Bash but with less features and capabilities.
That used to be true but a lot has changed since Mator Smash's last version in mid-2018. Wrye Bash was last updated on 19 Dec 2019 and has had a ton of updates since then.

Wrye Bash is open source with multiple talented developers that also work on the Unofficial Patches and xEdit teams (and vice versa). the Wrye Mash project is being tested with OpenMW as well.
Good to know. I haven't been following the modding scene for games newer than Oblivion, for which I favored Wrye Bash anyway, in ages. Wrye's tools have always impressed me. About as lightweight in terms of load as GUI tools get, and they do basically everything you want or need. Just need to get Mash working on Linux now and I'll be happy. Wish I had time to figure out enough to contribute to that.
User avatar
bmw
Posts: 81
Joined: 04 Jan 2019, 19:42
Contact:

Re: Portmod - Mod Manager

Post by bmw »

I've just released portmod 2.0_alpha6, which notably includes full Windows support. Thanks goes to walkingwounded for setting me down that road. I've slowly fixed a variety of Windows problems over the past couple months, starting with with what walkingwounded found, followed by what I found using Windows CI, which is now set up via AppVeyor, and finally the remaining issues that I discovered by running Portmod on Windows via a virtual machine. There may still be other platform-specific bugs, but it should generally work now, as opposed to being horribly broken.

Sandboxie is now used as the windows sandbox tool, though sadly it has not yet been released as open source. This causes a minor issue as there's no way of installing it in the Appveyor CI environment due to it only being available via a GUI installer, and the EULA is too strict to allow me to repackage it. There are also a couple of other technical issues (detailed here if you're interested), but it more or less appears to be working.

Full details of the update can be found on the releases page, and I've also started keeping a changelog in the repository.
Also, if installing Portmod on Windows, please take note of the Windows-specific installation instructions, as certain directories, such as the sandboxie installation directory, need to be added to your PATH environment variable.
User avatar
AnyOldName3
Posts: 2668
Joined: 26 Nov 2015, 03:25

Re: Portmod - Mod Manager

Post by AnyOldName3 »

If you can actually get the link for the Sandboxie installer, the script from this chocolatey package should still work: https://chocolatey.org/packages/sandboxie

It looks like it just passes /S and /install on the command line, though.
User avatar
bmw
Posts: 81
Joined: 04 Jan 2019, 19:42
Contact:

Re: Portmod - Mod Manager

Post by bmw »

Thanks, it's useful to know that the installer can also be run headless (I don't get why such software that takes command line arguments doesn't also provide a help message of any sort). Unfortunately there isn't a direct download, and it appears that Chocolatey is running into the same problem from their test log, where it looks like they've set it up to use a temporary download link.

Strangely you are forced to agree not to use the software for anything related to "chemical, biological or nuclear weapons" before you can download it, and you even need to provide your name, email, job, country and state/province (or more accurately, a name, email, job, country and state/province). I can only speculate why, and this isn't really the place for it (they say its due to government export regulations, but that doesn't explain why this software in particular is subject to these regulations), but the result is that there is no direct link. Assuming that the open source version is actually open source according to the open source definition, then there shouldn't be any such issues once that version is available.

I also did not know that there was a general-purpose package manager for Windows. There's hope for the platform yet (personally, I'm hoping Microsoft might take notice of a recent petition from the FSF, but I highly doubt it).
User avatar
AnyOldName3
Posts: 2668
Joined: 26 Nov 2015, 03:25

Re: Portmod - Mod Manager

Post by AnyOldName3 »

I think if they divert engineers to making 7 open-source it'll make it less likely that they'll make the latest version open-source before they get bored of the concept. 7 has got a lot more crap code in it than 10, so it's better to push for a still-maintained version to be opened instead.

As for Chocolatey, it's been around for a while. Windows 10 had a package manager on launch day that could be configured to use Chocolatey as a source, but it didn't let you update things back then and Chocolatey did, so no one really used it for that.

Most Windows installers need a /? to print their help message. If you just double click them in explorer, they won't get any arguments, and that's what people will do if they just want to run the GUI, so they'll just run the GUI if there're no arguments. You can only specify arguments when running from a command line (ish), so if it sees some, it'll consider printing stuff.
User avatar
bmw
Posts: 81
Joined: 04 Jan 2019, 19:42
Contact:

Re: Portmod - Mod Manager

Post by bmw »

AnyOldName3 wrote: 30 Jan 2020, 02:30 I think if they divert engineers to making 7 open-source it'll make it less likely that they'll make the latest version open-source before they get bored of the concept. 7 has got a lot more crap code in it than 10, so it's better to push for a still-maintained version to be opened instead.
You're not wrong, but at the same time I feel the bigger issue at hand is that M$ isn't quite so enamoured with the Open Source movement as they'd like us to believe (let alone Free Software), mainly since W10 seems to reflect a desire for more control over their users, which a Free/Open Source version of Windows would not give them. I think the FSF's point was more that they could at least release old code that isn't going to directly compete with the latest version. Realistically, I see such a thing mostly benefiting projects like Wine and ReactOS, rather than giving us a maintained and usable Open Source version of Windows (which it might also do, but it would require a lot of effort to clean up, and something like ReactOS is probably a better route anyway).
There isn't really any expectation that they'll release nicely refactored code, but I suppose the code in its current form might reflect poorly on them, and there is probably a lot that they wouldn't want anyone to see. I also doubt that W10 has less poor code; I feel it's much more likely that there's just been a lot more bloat added and certain key components reworked, but that's mostly speculation and I'm hardly unbiased.
AnyOldName3 wrote: 30 Jan 2020, 02:30 As for Chocolatey, it's been around for a while. Windows 10 had a package manager on launch day that could be configured to use Chocolatey as a source, but it didn't let you update things back then and Chocolatey did, so no one really used it for that.
I had heard about the Windows 10 package manager, but from what I recall it was more aimed at C# dependencies and tools than at user-facing applications, but I may be confusing it with NuGet.
I don't follow Windows software news, so I'm not that surprised that I haven't heard of Chocolatey until now.
AnyOldName3 wrote: 30 Jan 2020, 02:30 Most Windows installers need a /? to print their help message. If you just double click them in explorer, they won't get any arguments, and that's what people will do if they just want to run the GUI, so they'll just run the GUI if there're no arguments. You can only specify arguments when running from a command line (ish), so if it sees some, it'll consider printing stuff.
I tried that on the Sandboxie installer, as well as "/help", "--help", etc., but it never displayed anything.
User avatar
AnyOldName3
Posts: 2668
Joined: 26 Nov 2015, 03:25

Re: Portmod - Mod Manager

Post by AnyOldName3 »

8, 8.1 and 10 all gave significant speed improvements over their predecessors, and that was because a lot of things got rewritten to be less terrible. Lots of hacks and workarounds that were kept around for backwards compatibility reasons got moved out of the main implementations and into separate ones that only get used in compatibility mode. Windows has loads of cases of bugfixes being fenced off behind checks that the application doesn't rely on the bug being present, or jumps being placed at certain addresses because something people used in the 90s called a non-exported function in a DLL based on the offset it had on the machine it was written on.

Currently, I'm leaning towards the idea that M$ are so tired of all the legacy code they've got that they'd like to abandon it, replace it with a fresh version, and make it other people's responsibility to deal with maintenance. That means making it so that other people can see and change the source code. The thing that makes me think this is going to keep going is that they've opened up things like Detours. With .NET, PowerShell and WSL, there's the obvious argument that they're trying to get Linux users to get their feet wet with Microsoft tools, but I don't think there's any situation where Detours being MIT-licenced makes using Windows more attractive to anyone who wasn't already stuck with it.



Regarding package mangers, Nuget is sort-of designed to be general-purpose, so nearly everything else (including Chocolatey, OneGet and the thing PowerShell uses for modules) is based on it. However, the only thing it does out-of-the-box is .NET (and sometimes C++) dependencies.
User avatar
bmw
Posts: 81
Joined: 04 Jan 2019, 19:42
Contact:

Re: Portmod - Mod Manager

Post by bmw »

Interesting. Frankly most of my knowledge is second-hand and anecdotal, and I've noticed that speed improvements have coincided with increased RAM on new machines, the widespread use of SSDs, and Windows including a few tricks such as hybrid boot and compressed memory, which together could probably account for perceived speed improvements.

I suppose it's good to know that M$ doesn't purely add bloat and surveillance software to their OS.
AnyOldName3 wrote: 30 Jan 2020, 21:25 Currently, I'm leaning towards the idea that M$ are so tired of all the legacy code they've got that they'd like to abandon it, replace it with a fresh version, and make it other people's responsibility to deal with maintenance. That means making it so that other people can see and change the source code. The thing that makes me think this is going to keep going is that they've opened up things like Detours.
Basically they've realised that maintaining support for old interfaces so that binaries will work for more than 20 years isn't worth it? I think the biggest issue they would run into here is that they might also need to change the entire Windows software ecosystem to get this to work. As long as people keep distributing purely binary software, they're going to have to keep supporting old binary interfaces. A purely FLOSS ecosystem avoids this issue, but as Linux finds, despite being mostly FLOSS, that as long as there are binary blobs being distributed you still need to string along old versions of libraries if you want to maintain support (e.g. Steam runtime). Then again, I guess that by releasing their runtime libraries under free licenses (and detours, for old versions that aren't free), they can get software developers to just bundle as many libraries as necessary to keep the software working while Windows APIs change.
Post Reply