[PRE-REL] For ALOT of scripts.

Post about your mods, learn about OpenMW mod compatibility, check for problematic mods, discuss issues, and give us feedback about your experience with modded OpenMW.
kaekaze
Posts: 6
Joined: 07 Aug 2022, 18:05

[PRE-REL] For ALOT of scripts.

Post by kaekaze »

TLDR: How do I minimize maintenance for online contributions while living primarily off-the-grid?

Now the long version:

For years I've been modding Morrowind but it has been over a decade since I've released ANYTHING. Frankly this is unacceptable given the contributions I'm taking to playing with.

Here is an extremely simple example of a script that could help a lot of people and that many of my mods are based on.

Code: Select all

BEGIN kBSFScrGlbCalendarDayUpdate

;Similar to DaysPassed but based on calendar math instead of days which could be missed.

float fResult

set fResult to Year * 365.0

if ( Month == 1 )
	set fResult to fResult + 31.0
elseif ( Month == 2 )
	set fResult to fResult + 59.0
elseif ( Month == 3 )
	set fResult to fResult + 90.0
elseif ( Month == 4 )
	set fResult to fResult + 120.0
elseif ( Month == 5 )
	set fResult to fResult + 151.0
elseif ( Month == 6 )
	set fResult to fResult + 181.0
elseif ( Month == 7 )
	set fResult to fResult + 212.0
elseif ( Month == 8 )
	set fResult to fResult + 243.0
elseif ( Month == 9 )
	set fResult to fResult + 273.0
elseif ( Month == 10 )
	set fResult to fResult + 304.0
elseif ( Month == 11 )
	set fResult to fResult + 334.0
endif

;Post days passed since last frame. This can become a real issue with high timescale values
	set kBSFGlbFDaysFrame to ( fResult + Day ) - kBSFGlbFDays

;Post current Day
	set kBSFGlbFDays to fResult + Day
;GameHour and this script are not updated concurrently. Post the value as this script sees it.
	set kBSFGlbFGameHour to GameHour
	set kBSFGlbIGameHour to GameHour

END’
Obviously this kind of contribution should be part of a larger library of function-adding mod(s). If this already exists somewhere please link me up. I have quite a few mods that add things like:
post SQRT stats if they're over 100,
post the pc's crosshair trajectory 1000u in front of the PC's camera,
take care of modding stats through spell effects based on global input,
provide distinct teleport spots for multiple companions,
etc.

Hopefully this sells you on my sincerity given that I'm holding a lot back.
Why haven't I just released all of it?

Standardization
Portability
Maintenance
Naming conventions
Load order
Still working out a productive system for asset-credit-management on my local FS.
Life is also chaotic.

Plugin naming conventions have always been meritless in our community. Here's an example of some possible improvements:

a) Given that changing a plugin's master files is difficult I found the simplest solution is to edit their characters via a hex-editor.
However this fails utterly when the length of the master's name expands, so any standardized library of master files should include additional underscores for filename expansion. I recommend 8 underscores prefixed to the file's name.

b) While it is impossible to specifically subcatagorize every plugin there are some standard categories that all plugins fit and these can be assigned a numerical value to assist in load-order management.
0 - Vanilla - Obvious
2 - Asset - Files that contain asset records but put no active content in-game. Could also include function-scripts. Mods used by other mods but not the player. Could also include assets that are maintained by scripts but have no game-impact of themselves.
4 - Master - Master mods that DO add content for player.
6 - Plugins - Content.
8 - Compatibility - Patches for comparability like making telvanni's mods work together
a - Overrides - Files that override all concerns except user's. omwllf is a good example.
c - User-Only - User's personal changes.
Used even numbers in case an override is necessary and odd numbers here flag the mod as an outlier.

c) There are near-infinite possible ways of ordering mods but within the above categories mods can be related usefully to one another with relatively little time investment. If we use base 36[0-z] and three characters then there are 46K+ codes that mod makers/users/curators can use to order their loading relative to one another.
An easier way to visualize base 36 counting:
----123456
1 - 012345
2 - 6789ab
3 - cdefgh
4 - ijklmn
5 - opqrst
6 - uvwxyz
As an example, a mod that has no inter dependencies would use the code "iii" since it's near the center of the options and leaves room for other mods to adjust themselves to it. ooo or taq would load after while azz or hbk would load before.
Or two mods that are interdependent might have codes cii and iii to make sure cii loads first while still leaving room between.
Obviously far more than two mods can relate to each other.

d) Author names help us keep records separate but they can become a burden. I recommend a self-restriction of 5 characters with underscores used for any white space. I use the handle kae so my abbreviation would be kae__ or __kae.

e) Since we now have MANY separate engines for very similar content it makes sense to distinguish between them in the filename. My proposal is that this be a 3-character label and precede the plugin extension as so:
________2.iii.kae__.iiiTrackTimeEasier.omw.esm
Obviously the omwaddon extension does not require this.

Next: MetaData management is a concept I'm beginning to engage with thanks to Linux and there are good uses for our plugins too.
The prime example is the readme.txt which SHOULD be included with every mod but faces quite a few drawbacks.
a) similar names can create conflicts without sub folders
b) subsections of readme files can be difficult to find/batch-process.
c) warnings in readmes can be easily overlooked/ignored. (watch out for counterproductive attention-grabbing-escalation here)
d) versions should NEVER be in plugin filenames because of frequent changes but must be readily-comparable
e) this method would be simpler to use with modern text-filter popups

My proposal is that instead we include separate files with a useful extension like mta:
________2.iii.kae__.iiiTrackTimeEasier.omw.esm.blurb.mta
________2.iii.kae__.iiiTrackTimeEasier.omw.esm.version.mta
________2.iii.kae__.iiiTrackTimeEasier.omw.esm.dump.mta
________2.iii.kae__.iiiTrackTimeEasier.omw.esm.credit.mta
________2.iii.kae__.iiiTrackTimeEasier.omw.esm.knownAlternates.mta

These are easily ordered and simple to batch process into/out-of folders.
It would also be useful to have a flag character for the tags to indicate automation-compatibility like 0 as such:
________2.iii.kae__.iiiTrackTimeEasier.omw.esm.0dump.mta
which I can use via the bash shell to update
________2.iii.kae__.iiiTrackTimeEasier.omw.esm.dump.mta
Probably be useful to make the first line of the file signify the kind of processing the file is geared for like #!/bin/bash or #JSON or <HTML> or whatever.

______________________________
So why am I telling you all of this? Networking, critique, debugging, please tear these up/improve the ideas. Obviously it's not going to be correct the first time but imagine how much time we could all save (both on and off the grid) by wielding some better curation methods.

I'm also aware that the devs here are very good and could point me to some productive literature on the subject or at least post the correct search terms.

Hopefully I can start ironing out my content for practical release within the next few months (and spend as little time maintaining it as possible. something all of us want).

Addendum 1:

Code: Select all

Using the plugin's file name to create an alpha-numeric load order might save everyone a lot of time. (Then I found out about LOOT, thanks bmw)
I always prefer using the file manager to curate things over external utilities that require maintenance and are less portable, so filenames are a natural choice to keep things ordered.

Last edited by kaekaze on 08 Aug 2022, 01:33, edited 1 time in total.
User avatar
bmw
Posts: 81
Joined: 04 Jan 2019, 19:42
Contact:

Re: [PRE-REL] For ALOT of scripts.

Post by bmw »

Given that changing a plugin's master files is difficult I found the simplest solution is to edit their characters via a hex-editor.
However this fails utterly when the length of the master's name expands, so any standardized library of master files should include additional underscores for filename expansion. I recommend 8 underscores prefixed to the file's name.
Are there not tools already which can modify master file names? Underscores for filename expansion sounds like a very ugly solution which also limits filename length.

tes3cmd should be able to do so, though I found a bug in it when trying. The following works, but only if you fix the (?i) that it adds to make the match case insensitive (line 8658) (maybe that's an old feature which perl has since dropped, you should be adding the i modifier at the end of the regex).

Code: Select all

tes3cmd modify --type TES3 --sub-match MAST --replace "/^<old name>$/<new name>/" plugin.esp
Next: MetaData management is a concept I'm beginning to engage with thanks to Linux and there are good uses for our plugins too.
The prime example is the readme.txt which SHOULD be included with every mod but faces quite a few drawbacks.
I don't think they are formally standardised anywhere, but I would recommend looking at open source software practices as in this case they also work well for mods, and there's no need to create a new standard. E.g. files like README, LICENSE, CHANGES/CHANGELOG NEWS AUTHORS, etc (often with extensions). OpenMW itself is a good example of this.
They don't need to include the name of the plugin since openmw can handle having a separate directory for each mod.
Wikipedia has some info on the practice.

I'm guessing you're thinking of the vanilla engine when wanting to keep filenames distinct, but even there it would be better to have a mod manager install those files in to a separate directory (e.g. "doc/<mod name>") than to dump them all into the root directory, which would get cluttered very quickly if you install a lot of mods.
b) While it is impossible to specifically subcatagorize every plugin there are some standard categories that all plugins fit and these can be assigned a numerical value to assist in load-order management.
I agree that those categorizations are helpful.
LOOT for example has plugin groups. Portmod has tiers which have the same purpose. I'm not sure that it needs to be in the filename though. In the documentation is probably sufficient, or if it's going to be standardised for the purpose of automated tools having it in the file is good, but it should be at least more explicit. E.g. a "category=1" line at the top of the description in the plugin header.
d) Author names help us keep records separate but they can become a burden. I recommend a self-restriction of 5 characters with underscores used for any white space. I use the handle kae so my abbreviation would be kae__ or __kae.

e) Since we now have MANY separate engines for very similar content it makes sense to distinguish between them in the filename. My proposal is that this be a 3-character label and precede the plugin extension as so:
Again, not really something that needs to go in the filename. There even is an author field in the header already, so having it in the filename is redundant. Or are you referring to record namespacing via a prefix (which is certainly a good idea) and also including the prefix in the file name (I'm not sure why that's necessary)?

For engines it shouldn't in theory matter since OpenMW hasn't extended the plugin format and plugins meant for the bethesda engine generally should work, but in cases that doesn't match reality an indication in the description would probably suffice.
kaekaze
Posts: 6
Joined: 07 Aug 2022, 18:05

Re: [PRE-REL] For ALOT of scripts.

Post by kaekaze »

bmw wrote: 07 Aug 2022, 20:49
Are there not tools already which can modify master file names? Underscores for filename expansion sounds like a very ugly solution which also limits filename length.

tes3cmd should be able to do so, though I found a bug in it when trying. The following works, but only if you fix the (?i) that it adds to make the match case insensitive (line 8658) (maybe that's an old feature which perl has since dropped, you should be adding the i modifier at the end of the regex).

Code: Select all

tes3cmd modify --type TES3 --sub-match MAST --replace "/^<old name>$/<new name>/" plugin.esp
This is the problem I keep running into with regular expressions outside...well tightly-watched code? I'm sure it works on your system but I just keep getting an "Invalid replacer expression." Tried adjusting quotes, using string literals, even reverted to some of the syntax described in the first perl tutorial but nothing. Also remember that I'm using wine to run tes3cmd in the first place.
I completely agree and the solution here is probably just to seek help but if the tool is that outdated and unmaintained do I have any alternative?
The underscores-as-white-space bother me too btw.
I don't think they are formally standardized anywhere, but I would recommend looking at open source software practices as in this case they also work well for mods, and there's no need to create a new standard. E.g. files like README, LICENSE, CHANGES/CHANGELOG NEWS AUTHORS, etc (often with extensions). OpenMW itself is a good example of this.
They don't need to include the name of the plugin since openmw can handle having a separate directory for each mod.
Wikipedia has some info on the practice.

I'm guessing you're thinking of the vanilla engine when wanting to keep filenames distinct, but even there it would be better to have a mod manager install those files in to a separate directory (e.g. "doc/<mod name>") than to dump them all into the root directory, which would get cluttered very quickly if you install a lot of mods.
I always see the fileManager/shell as my most portable and versatile tool and stay away from mod managers since they need maintenance and don't work on every system. This could also be a product of not fully engaging with the development community though.

I suppose the most elegant solution would simply be a soft-link with a similar name to the plugin that points straight to the mod's directory. That could give the best of both worlds. This process would have to be automated(and therefore maintained) though....hence your suggestion of the mod manager.... Need to think about this.

Placing it all into a doc subdirectory is also a good practice. Probably go with this wether linked/not.
I agree that those categorizations are helpful.
LOOT for example has plugin groups. Portmod has tiers which have the same purpose.
I'll be reading those pages over for a while. Thank. You.
I'm not sure that it needs to be in the filename though. In the documentation is probably sufficient, or if it's going to be standardized for the purpose of automated tools having it in the file is good, but it should be at least more explicit. E.g. a "category=1" line at the top of the description in the plugin header.
The filename puts it where literally everyone and everything can see it immediately without the need to dig even a single step deeper. The overall idea, which I should probably add to the original post, was to create a working alpha-numeric load order without the need for tools like LOOT. Which solution is better? Probably LOOT provided there aren't any exceptions to it's process-of-determination, or a system of adding hard rules to it's graph(still reading. maybe it's already there).
Also the mod description field is an extremely limited resource(255 characters?). Be better to use a standard named script like BEGIN SCNMETAHEADER or something more sensible. <<< This might be a legitimately good idea actually.
Trying to get anyone(let alone the majority) on the same page is a daunting/impossible task but that applies ot most ideas. I think with the right inertia most would see the merit. It's not like any of this should be enforced.
There's also the issue of needing to load up external software like tes3cmd to access the data needed. Maybe I could use a variable but keep it in a text file.
Again, not really something that needs to go in the filename. There even is an author field in the header already, so having it in the filename is redundant. Or are you referring to record namespacing via a prefix (which is certainly a good idea) and also including the prefix in the file name (I'm not sure why that's necessary)?

For engines it shouldn't in theory matter since OpenMW hasn't extended the plugin format and plugins meant for the bethesda engine generally should work, but in cases that doesn't match reality an indication in the description would probably suffice.
Mostly speed, portability, and ease-of-use. It takes an external program to access that field whether it be the CS, tes3cmd, your mod manager/whatever.
Actually, you may have made good argument for me there too! If the prefix is valid and identical in both places then there's no question of what to look for in the CS/whatever. Heh, omwcs could even insert this value (or the author field) into it's default filter string.
So long-term, maybe use the prefix instead of the author name. There's also the issue of networking&advertising but I'm definitely being too aggressive here.

As for the engine I can't actually justify this, it just seemed like a good idea in general but if the mods(and their processing) function identically across the programs? Sure.
One of the reasons I switched to OMW was the stability and consistency of scripting results though. Even if the file format is the same, the environment the script/content runs in can be vastly different. This could also include tags like mge/mwse, it's not soley for omw. Yes, this information could be put in the publishing documentation but you really want to dig through that after the game has been sitting for 6+months because something else was more immediate?

I suppose the best sell I can give in favor of filenames is they are just there. All. The. Time.

I hope that was more productive than combative? Sincere apologies if not.

So the real question. How does this affect my releases? Going to wait and think for a couple weeks. Will stay in contact though. I have internet for another 10 days or so.
Last edited by kaekaze on 08 Aug 2022, 13:13, edited 3 times in total.
User avatar
AnyOldName3
Posts: 2666
Joined: 26 Nov 2015, 03:25

Re: [PRE-REL] For ALOT of scripts.

Post by AnyOldName3 »

Please bear in mind that until the forum software's absolutely certain that you're human, editing your posts hides them and puts them back in the moderation queue until someone re-approves them. Just make new posts for now or no one will be able to see what you've written.
User avatar
bmw
Posts: 81
Joined: 04 Jan 2019, 19:42
Contact:

Re: [PRE-REL] For ALOT of scripts.

Post by bmw »

kaekaze wrote: 08 Aug 2022, 01:49 I completely agree and the solution here is probably just to seek help but if the tool is that outdated and unmaintained do I have any alternative?
The underscores-as-white-space bother me too btw.
I've created a pull request with the fix, though since the author has been inactive for years I don't anticipate it being merged. Hopefully it will at least be more visible that way for anyone else with the issue. See here if you just want the updated script, and with that fix both --replace and --replacefirst should work.
kaekaze wrote: 08 Aug 2022, 01:49 I always see the fileManager/shell as my most portable and versatile tool and stay away from mod managers since they need maintenance and don't work on every system. This could also be a product of not fully engaging with the development community though.
Most mod managers are unfortunately kind of unwieldy (part of why I wrote my own). Like what you said, I prefer working with multiple unix-style simple tools which solve a very specific problem, instead of large complicated monolithic GUI programs.
And while I'll admit that my mod manager (Portmod) is also not perfect in this respect, I have been separating out openmw-specific parts into standalone tools (though some of them are still a little too tightly coupled to Portmod's metadata to have much use by themselves).
kaekaze wrote: 08 Aug 2022, 01:49 The filename puts it where literally everyone and everything can see it immediately without the need to dig even a single step deeper. The overall idea, which I should probably add to the original post, was to create a working alpha-numeric load order without the need for tools like LOOT. Which solution is better? Probably LOOT provided there aren't any exceptions to it's process-of-determination, or a system of adding hard rules to it's graph(still reading. maybe it's already there).
So you essentially want to create a total ordering of all plugins, as opposed to a rules-based system? It should work, but I feel like you'd probably run into issues.

For one, users may have custom rules for things like mods with partially overlapping features where the ordering determines which conflicting feature gets used. Rule-based orderings should have fewer issues resolving such custom rules since they distinguish between orderings which are required and those which are coincidental in the output list, while in a total ordering even plugins in the list where relative orderings don't matter need to have a specific order, so you can't tell when moving a plugin from one position in the list to another if the plugins in-between need to come before/after the plugin you're moving.

A total ordering would also make it harder to document why plugins are ordered in a specific way, since the number doesn't describe very much at all, and any given plugin could have relationships with any number of other plugins so leaving comments in the middle of the list aren't very useful. Rules-based orderings provide more detail by default, and provide a clear way to document each ordering rule since each rule has to be listed individually. This is also why I dislike mlox's rulelist since many of its orderings are just lists of plugins without much explanation rather than relationships between specific plugins. LOOT is better, but still doesn't provide any context for its rules in the masterlists. In portmod's package files I usually leave a comment explaining the order rules.
If the only mandatory information is the number, you'll end up with some mod authors providing the number with no explanation, which is useless if any issues are found with the order, or you have to modify it to satisfy a custom ordering.
kaekaze wrote: 08 Aug 2022, 01:49 Also the mod description field is an extremely limited resource(255 characters?). Be better to use a standard named script like BEGIN SCNMETAHEADER or something more sensible. <<< This might be a legitimately good idea actually.
...
There's also the issue of needing to load up external software like tes3cmd to access the data needed. Maybe I could use a variable but keep it in a text file.

Mostly speed, portability, and ease-of-use. It takes an external program to access that field whether it be the CS, tes3cmd, your mod manager/whatever.
...
I suppose the best sell I can give in favor of filenames is they are just there. All. The. Time.
I think I overlooked the length limit on the description when I glanced at the header info before; I was thinking it was variable-length. A special record isn't a bad idea, but I think a separate text file is probably the best way, though it would probably be good to use a format like json/yaml/toml so that it's easy for tools to read while still being relatively easy to write manually. A file format would also be easier to extend, since it could be more easily versioned, and fields can be named rather than positional.

That's actually sort of what I've done recently with configtool (the OpenMW plugin/data files sorter for Portmod), though I had written that more as a way to separate the config rules from portmod's package files than as a standalone metadata format, but it could be used as such. It's in its early stages still, and I don't have any documentation other than the code structure defining it (which is serialised to/from json).
kaekaze wrote: 08 Aug 2022, 01:49 I hope that was more productive than combative? Sincere apologies if not.
It didn't come across as combative at all. I had actually been wondering if my first reply had been a little abrupt and negative, but to try and make my general view clear, I think that standardisation of this sort of thing is a great idea and I'm glad to see someone else thinking about it, even if I don't agree with everything you've said.

Edit:
kaekaze wrote: 08 Aug 2022, 01:49 One of the reasons I switched to OMW was the stability and consistency of scripting results though. Even if the file format is the same, the environment the script/content runs in can be vastly different. This could also include tags like mge/mwse, it's not soley for omw.
I completely forgot about the non-lua version of mwse (it's been a long time since I used the original Morrowind engine). I was thinking that's all in separate files, so it's unrelated to the plugins, but yes, being able to differentiate between plugins which expect script extensions would be useful.
kaekaze
Posts: 6
Joined: 07 Aug 2022, 18:05

Re: [PRE-REL] For ALOT of scripts.

Post by kaekaze »

Please bear in mind that until the forum software's absolutely certain that you're human, editing your posts hides them and puts them back in the moderation queue until someone re-approves them. Just make new posts for now or no one will be able to see what you've written.
Got it, didn't think that applied to editing as well but it makes sense to keep the hordes at bay. AND HOW DOES ONE DEFINE HUMANITY IN THIS FORUM'S STANDARDS FELLOW ADVERTISING VICT- I MEAN MEAT-BAG? Ideally with specific standards that I can subvert of course.
I've created a pull request with the fix, though since the author has been inactive for years I don't anticipate it being merged. Hopefully it will at least be more visible that way for anyone else with the issue. See here if you just want the updated script, and with that fix both --replace and --replacefirst should work.
Works flawlessly. You've saved me SO MUCH HEADACHE! I was using wine to run the exe! I realize now that I never updated the program when I converted to linux years ago. It's SO much smoother with python.
So you essentially want to create a total ordering of all plugins, as opposed to a rules-based system? It should work, but I feel like you'd probably run into issues.

For one, users may have custom rules for things like mods with partially overlapping features where the ordering determines which conflicting feature gets used. Rule-based orderings should have fewer issues resolving such custom rules since they distinguish between orderings which are required and those which are coincidental in the output list, while in a total ordering even plugins in the list where relative orderings don't matter need to have a specific order, so you can't tell when moving a plugin from one position in the list to another if the plugins in-between need to come before/after the plugin you're moving.
A total ordering would also make it harder to document why plugins are ordered in a specific way, since the number doesn't describe very much at all, and any given plugin could have relationships with any number of other plugins so leaving comments in the middle of the list aren't very useful. Rules-based orderings provide more detail by default, and provide a clear way to document each ordering rule since each rule has to be listed individually. This is also why I dislike mlox's rulelist since many of its orderings are just lists of plugins without much explanation rather than relationships between specific plugins. LOOT is better, but still doesn't provide any context for its rules in the masterlists. In portmod's package files I usually leave a comment explaining the order rules.
If the only mandatory information is the number, you'll end up with some mod authors providing the number with no explanation, which is useless if any issues are found with the order, or you have to modify it to satisfy a custom ordering.
This is very sensible and precisely the kind of critique I needed. TYVM.
After downloading and looking into portmod repos I'm comparing them to udev and modprobe configurations. Haven't got anywhere productive with it yet but there's some definite compraisons.
I should also point out that systems aren't mutually-exclusive but more overhead should be avoided unless it can be automated. This definitely favors rules over a linear-load-order.
portmod's pybuild files are pretty good, to the point where they could be translated to other formats or generated from them. Maybe a separation of the py build's metadata into a variables-only header could separate it from the python-specific instructions.
BTW, the concept used for portmod management is eluding me. Are repos for maintainers or to append to openmw's assets? How does portmod's merge function link to openmmw's data? Does it just install to the filesystem? Merged a few mods but it seems no alterations were made to the MORROWIND_PATH(which is a custom directory). No alterations were made to the config file either which is in the default location.

I took one of the pybuild files and gave an example of something I might do for my mods.
Also some questions below(## used to label):

Code: Select all


Sets employed:
	AllVanilla: Morrowind, Tribunal & Bloodmoon

{
	#The primary archive with all assets and plugins
	ArchiveLabel = "ULPrimary",
	KnownSourceURLs = [
		[ "Direct", "https://www.dropbox.com/s/sqxbej0nef7bg9e/Uviriths_Legacy_3.53.7z?dl=1" ],
		[ "Browser", "https://www.dropbox.com/s/sqxbej0nef7bg9e/Uviriths_Legacy_3.53.7z?dl=1" ]
	],
	DestinationFilename="{P}.7z",
	ArchiveRoot = "./ULPrimary"
},
{
	#Patch for teleporting ring I guess?
	ArchiveLabel = "Teleport",
	KnownSourceURLs = [
		[ "Direct", "https://drive.google.com/uc?id=15epmtn1NSOS_veshHW3IgTnoYOV6qzPa&e=download" ],
		[ "Browser", "https://drive.google.com/uc?id=15epmtn1NSOS_veshHW3IgTnoYOV6qzPa&e=download" ]
	],
	DestinationFilename="Uvi_Teleport_fix.esp",
	ArchiveRoot = "."
},
{
	#Compatability patch
	ArchiveLabel = "OMWPatch",
	KnownSourceURLs = [
		[ "Any", "https://modding-openmw.com/files/UL_3.5_OpenMW_1.3_Add-on.7z" ]
	],
	DestinationFilename="UL_3.5_OpenMW_1.3_Add-on.7z",
	ArchiveRoot = "./OMWPatch"
},
{
	#No point in having the mod without this.
	Plugin = [ "Uvirith's Legacy_3.53.esp", "Uvi_Teleport_fix.esp" ],
	AddRequirement = [
		Type = "set",
		Label = AllVanilla
	]
},
{
	#What it says
	Plugin = [ "UL_Chess_Add-on.esp" ],
	AddRequirement = [
		Type = "Plugin",
		Label = "Uvirith's Legacy_3.53.esp"
	],
	AddRequirement = [
		Type = "UserSelects",
		Prompt = "Would you like to install chess for Uvirith's Legacy?"
		Result = "[y,Y]"
	]
},
{
	#Compatibility patch for Rise of House Telvanni
	Plugin = [ "UL_3.5_RoHT_1.52_Add-on.esp" ],
	AddRequirement = [
		Type = "Plugin",
		Label = "Uvirith's Legacy_3.53.esp"
	],
	AddRequirement = [
		Type = "InstalledPlugin",
		Label = "Rise of House Telvanni.esm"
	]
},
{
	#No clue, MWSE and openmw don't mix
	Plugin = [ "Addons/UL_3.5_MWSE_Add-on.esp", "Addons/UL_MWSE_Companions_v2.esp", "Addons/UL_Upgrader_MWSE.esp" ],
	AddRequirement = [
		Type = "Plugin",
		Label = "Uvirith's Legacy_3.53.esp"
	],
	AddRequirement = [
		Type = "InstalledFile",
		Path = "Data Files/../mwse.exe"
		FailureOutput = [
			"Prompt",
			"MWSE.exe not found. Proceed without installing MWSE plugins [y/n]?: "
		]
	]
},
{
	#No clue actually and I probably should know.
	Plugin = [ "UL_3.5_OpenMW_1.3_Add-on.omwaddon" ],
	AddRequirement = [
		Type = "Plugin",
		Label = "Uvirith's Legacy_3.53.esp"
	],
	AddRequirement = [
		Type = "EngineEnvironment",
		Regex = "openmw*"
	]
},

#Load-Order Impositions. REQUIRED means that if PluginB is selected PluginA has to load first, it's not optional.
#bmw did mention that optional precedence is a thing. Could I get an example of this? <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
{
	AddLORule = [ "REQUIRED", "Morrowind.esm", "Uvirith's Legacy_3.53.esp" ],
	AddLORule = [ "REQUIRED", "Tribunal.esm", "Uvirith's Legacy_3.53.esp" ],
	AddLORule = [ "REQUIRED", "Bloodmoon.esm", "Uvirith's Legacy_3.53.esp" ],
	AddLORule = [ "REQUIRED", "Uvirith's Legacy_3.53.esp", "Uvi_Teleport_fix.esp" ],
	AddLORule = [ "REQUIRED", "Uvirith's Legacy_3.53.esp", "UL_Chess_Add-on.esp" ],
	AddLORule = [ "REQUIRED", "Uvirith's Legacy_3.53.esp", "UL_3.5_RoHT_1.52_Add-on.esp" ],
	AddLORule = [ "REQUIRED", "Rise of House Telvanni.esm", "UL_3.5_RoHT_1.52_Add-on.esp" ],
	AddLORule = [ "REQUIRED", "Uvirith's Legacy_3.53.esp", "Addons/UL_3.5_MWSE_Add-on.esp" ],
	AddLORule = [ "REQUIRED", "Uvirith's Legacy_3.53.esp", "Addons/UL_MWSE_Companions_v2.esp" ],
	AddLORule = [ "REQUIRED", "Uvirith's Legacy_3.53.esp", "Addons/UL_Upgrader_MWSE.esp" ],
	AddLORule = [ "REQUIRED", "Uvirith's Legacy_3.53.esp", "UL_3.5_OpenMW_1.3_Add-on.omwaddon" ]
	AddLORule = [ "CONFLICT", "Uvirith's Legacy_3.53.esp", "Uvirith's Legacy_3.51.esp" ]
)


# Copyright 2019-2020 Portmod Authors
# Distributed under the terms of the GNU General Public License v3

import os
import shutil

from pybuild import DOWNLOAD_DIR, File, InstallDir, Pybuild1
from pybuild.info import P


class Package(Pybuild1):
    NAME = "Uvirith's Legacy"
    DESC = "Turns Tel Uvirith into a fitting home for a mage lord and symbol power"
    HOMEPAGE = "https://stuporstar.sarahdimento.com/"
    KEYWORDS = "openmw"
    LICENSE = "all-rights-reserved"
    RDEPEND = """
        base/morrowind[bloodmoon,tribunal]
        tr? ( landmasses/tamriel-rebuilt )
        roht? ( quests-factions/rise-of-house-telvanni )
    """
	##Triple-quotes? tr? why the question mark for a dependency? to flag that it might be dependency and just print a warning?

    TEXTURE_SIZES = "512"
	##I'm guessing this is to attempt the installation of a mod/package variation that fits the prefix variable? It's included in the use flag docs so I assume it performs a similar function?

    SRC_URI = f"""
        https://www.dropbox.com/s/sqxbej0nef7bg9e/Uviriths_Legacy_3.53.7z?dl=1 -> {P}.7z
        https://drive.google.com/uc?id=15epmtn1NSOS_veshHW3IgTnoYOV6qzPa&e=download
        -> Uvi_Teleport_fix.esp
        https://modding-openmw.com/files/UL_3.5_OpenMW_1.3_Add-on.7z -> UL_3.5_OpenMW_1.3_Add-on.7z
    """
	##Pretty obvious that these are instructions for wget or something similar.

    IUSE = "roht +chess tr"
    INSTALL_DIRS = [
##This section is pretty straight foreward but for newbees it might be helpful to say that these are subdirectories in the downloaded archive to be installed and not destination paths.
        InstallDir(
            "Data Files",
            PLUGINS=[File("Uvirith's Legacy_3.53.esp"), File("Uvi_Teleport_fix.esp")],
            S=P,
##S=P?
        ),
        InstallDir(
            "Data Files/Addons",
            BLACKLIST=[
                "UL_3.5_MWSE_Add-on.esp",
                "UL_MWSE_Companions_v2.esp",
                "UL_Upgrader_MWSE.esp",
            ],
            PLUGINS=[
                File("UL_Chess_Add-on.esp", REQUIRED_USE="chess"),
                File("UL_3.5_RoHT_1.52_Add-on.esp", REQUIRED_USE="roht"),
                File("UL_3.5_TR_16.12_Add-on.esp", REQUIRED_USE="tr"),
                # File("UL_AshArmor_Plangkye.esp"),
                # File("UL_AshArmor_DivineDomina.esp"),
                # File("UL_WeaponRotate_Addon.esp"),
                # The Lighting Mod
                # File("UL_TLM_Ambiance.esp"),
                # File("UL_BookJackets_Add-on.esp"),
                # Children of Morrowind
                # File("UL_CoM_Add-on.esp"),
                # File("Stinkers 2.0 to 3.0 Upgrader.esp"),
            ],
            S=P,
        ),
        InstallDir(
            ".",
            PLUGINS=[File("UL_3.5_OpenMW_1.3_Add-on.omwaddon")],
            S="UL_3.5_OpenMW_1.3_Add-on",
        ),
    ]

#I'm guessing this means .... nope. Why does the teleport fix need an exemption?
    def src_unpack(self):
        for source in self.A:
            if source.name == "Uvi_Teleport_fix.esp":
                shutil.copy(source.path, f"{P}/Data Files")
            else:
                self.unpack([source])

    def pkg_nofetch(self):
        super().pkg_nofetch()
        print("Please download the following file from the url at the bottom")
        print(
            "before continuing and move them to one of the following download directories:"
        )
        print(f"  {DOWNLOAD_DIR}")
        print("  " + os.path.expanduser("~/Downloads"))
        print()
        print("  UL_3.5_OpenMW_1.3_Add-on.7z")
        print()
        print(
            "   https://mega.nz/#!FEwSzSoS!q4EPrN_0aaxWE05TTCdNYhHd1fzTewi1pzcCwjR3glg"
        )

User avatar
bmw
Posts: 81
Joined: 04 Jan 2019, 19:42
Contact:

Re: [PRE-REL] For ALOT of scripts.

Post by bmw »

kaekaze wrote: 09 Aug 2022, 23:53 BTW, the concept used for portmod management is eluding me. Are repos for maintainers or to append to openmw's assets? How does portmod's merge function link to openmmw's data? Does it just install to the filesystem? Merged a few mods but it seems no alterations were made to the MORROWIND_PATH(which is a custom directory). No alterations were made to the config file either which is in the default location.
You may have missed a step in the setup. You should have installed configtool as part of a world update ("merge -uD @world"; configtool is a required "system" package), and that will run after merges (it can be invoked manually with the "portmod <prefix> cfg-update" command) and will update your openmw.cfg, including adding data entries for each directory where portmod installed the files. It installs mods into separate directories within the prefix, which by default on Linux would be "~/.local/share/portmod/<prefix_name>" (with the VFS directories in the pkg subdirectory).
kaekaze wrote: 09 Aug 2022, 23:53 I took one of the pybuild files and gave an example of something I might do for my mods.
Generally speaking, I could see something like that being quite helpful for both tools like portmod and more manual mod managers like Wrye Mash or Mod Organiser.

I think the UserSelects rule could be more descriptive, and doesn't really need a prompt message. If a mod manager wants to provide a prompt (portmod for example has its own system for optional features), I think it would make more sense for the prompt to be a prompt created by the mod manager itself, and for it to just reference the name and description of the option, since the text would be more or less the same each time (E.g. "Would you like to install the {name} feature? '{desc}'").

This sort of requirement information could also be used for directories of assets in addition to plugins (sorting rules too). It might make more sense to have the structures with the "AddRequirement" fields be substructures of the archive data and reference files within the archives (or at least contain a reference to their parent archive and path within the archive).

It's worth noting that a generic "Does file exist" interface will only work between the vanilla engine and OpenMW if it only can refer to files within the VFS/"Data Files". It's not meaningful to refer to the parent directory of data files (or even Data Files itself) for OpenMW, which only concerns itself with the contents. Though I suppose anything implementing this for OpenMW could just assume "Data Files" refers to the VFS and report that all other files don't exist, but that will cause problems if this is extended to other games which have a different structure (given there are plans for OpenMW to support the later Bethesda games).
It would probably be better to allow access to the vanilla-specific files only through a vanilla-specific interface which will be considered an unsatisfied requirement if using it with OpenMW (see my example below).

One interesting problem that I'm not sure has a good answer is how to deal with dependencies on specific versions, e.g. for mods like Tamriel Rebuilt which don't change the name of the plugin when new versions are released.
Hashes of the files? Loot supports this, and it would also work for asset dependencies, but it's not really very intuitive and even with multiple hashes doesn't allow for versions unknown at the time of writing like patch versions which don't make substantial changes (not like many mods follow semantic versioning though).
Some sort of decentralized versioning system could also work, but would be less compatible with mods which aren't installed that way and makes the system a lot more complicated. E.g. each mod using this system could also install a file with relevant installed metadata including the name, version and the files it installed, which other mods can refer to. This is actually what I'd suggested on a different thread.

I've put together my own prototype to illustrate some of the things mentioned above: https://gitlab.com/-/snippets/2386502 (on gitlab for the syntax highlighting). Obviously there are differences in style, and I'm not trying to condemn the use of UpperCamelCase or anything.
Also note that the format is for one file per archive, the idea being that you store it in the root of the archive and it has a particular name, so it can be found easily by tools. Separate metadata files would be needed for the patches, so it's not covering the same information as your example.

Replies to the stuff in the code-block below:

Code: Select all

#Load-Order Impositions. REQUIRED means that if PluginB is selected PluginA has to load first, it's not optional.
#bmw did mention that optional precedence is a thing. Could I get an example of this?
I think mlox has optional orderings, but I'm not entirely convinced they're useful. I guess it's sort of 
like saying that a particular ordering is recommended, but otherwise the user can choose with a 
custom rule (and of course that the rule can be broken if it would cause a cycle).

##Triple-quotes? tr? why the question mark for a dependency? to flag that it might be dependency and just print a warning?
Triple quotes are just python multi-line strings. The question mark is for a use conditional expression. 
Essentially if the use flag on the left hand side is enabled, then the part inside the brackets must be 
enforced.
In this case, the tr flag is used later to enable/disable the tamriel rebuilt patch, and this dependency 
string indicates that if the tr flag (and thus the tr patch) is enabled, then the tamriel-rebuilt package 
must be installed (which may be backwards from the way you'd think it would work, but there's also 
a feature which defaults the flag to enabled if tamriel-rebuilt is installed, but it's not a strict relationship, 
so there has to also be a hard dependency to ensure that the patch gets disabled if you later uninstall
tamriel-rebuilt).

    TEXTURE_SIZES = "512"
    ##I'm guessing this is to attempt the installation of a mod/package variation that fits the prefix variable? It's included in the use flag docs so I assume it performs a similar function?
    Well, it would be except that this package doesn't provide multiple texture options. It's mostly an 
    artifact of importmod, a tool which generates some of the metadata, and left to indicate the size of 
    the included textures in case it's useful. But for mods which list multiple texture options, portmod 
    can automatically choose between them based on the user's configuration.

##This section is pretty straight foreward but for newbees it might be helpful to say that these are subdirectories in the downloaded archive to be installed and not destination paths.
Noted, particularly since there are some changes planned to the install interface and some of the 
docs are going to be moved and rewritten.
It is in the APIdoc: https://portmod.gitlab.io/portmod/dev/pybuild/pybuild.html#pybuild.InstallDir.PATH
Also mentioned in the package guide: https://gitlab.com/portmod/portmod/-/wikis/OpenMW/Package-Guide
though maybe not as clearly as it could be.

#I'm guessing this means .... nope. Why does the teleport fix need an exemption?
It's not an archive. the unpack function will fail if the file can't be unpacked, 
and the default src_unpack runs the unpack function on everything listed in SRC_URI.
kaekaze
Posts: 6
Joined: 07 Aug 2022, 18:05

Re: [PRE-REL] For ALOT of scripts.

Post by kaekaze »

You may have missed a step in the setup.
Exactly. I missed all steps in the -setup-, not the installation. Installed it and started creating prefixes when I saw the interactive responses.
BTW, can't thank you enough for that. Interactive documentation that shows what options are available is amazing and I wish it was used more.
I'm probably unique to this mistake but if not a hook/prompt after profile-symlinking would help and fit with the interactive CLI responses.
-Note that when I say interactive, I don't mean that the CLI takes over stdin/out, just that it feeds the user dynamic choices based on each command.

configtool prompts for each change to the config file but inserts the notification about MERGE_TOOL between the changes and the choices. Is that expected behavior? The choices seemed to regard the MERGE_TOOL instead of the file changes. Also changes that are removed and re-added are listed each time. I assume this is just debugging practice but with large files it tends to overload my terminal's line limit(which may be relatively low).
If you don't see this behavior I wouldn't worry, bash is heavily customized over here.

Ah. Tried it with tee to get the output and all the prompts vanished /wo stdin tied to the terminal; so everything confusing vanished.
I think the UserSelects rule could be more descriptive, and doesn't really need a prompt message. If a mod manager wants to provide a prompt (portmod for example has its own system for optional features), I think it would make more sense for the prompt to be a prompt created by the mod manager itself, and for it to just reference the name and description of the option, since the text would be more or less the same each time (E.g. "Would you like to install the {name} feature? '{desc}'").
So the manager provides a generic question while the metafile provides a description of the content offered. This would work better across multiple languages but you've got me thinking... are these proposals easy enough to translate with automated tools? With a tool like translate-shell sure.
Due to the simplicity a csv file could be used to translate the keys used in the files between versions and languages. I have no experience with symbolic writing using computers unfortunately. Maybe this isn't possible. If it is then fields could also be put through automated translation. It won't be perfect but it's a start.

Code: Select all

Language	American-US	French	American-US(caseless)
Release	1	1	1
	Plugin	D-extension	plugin
	AddRequirement	AjouterBesoin	add_requirement
	Type	Determiner	type
	Label	Marquer	label
	Description	Description description

*I DO NOT know french, in case you didn’t notice.
UpperCamelCase
Got a giggle out of that.
This sort of requirement information could also be used for directories of assets in addition to plugins (sorting rules too). It might make more sense to have the structures with the "AddRequirement" fields be substructures of the archive data and reference files within the archives (or at least contain a reference to their parent archive and path within the archive).
Yes to substructures and yes, similar to USE flags.
It's worth noting that a generic "Does file exist" interface will only work between the vanilla engine and OpenMW if it only can refer to files within the VFS/"Data Files". It's not meaningful to refer to the parent directory of data files (or even Data Files itself) for OpenMW, which only concerns itself with the contents. Though I suppose anything implementing this for OpenMW could just assume "Data Files" refers to the VFS and report that all other files don't exist, but that will cause problems if this is extended to other games which have a different structure (given there are plans for OpenMW to support the later Bethesda games).
It would probably be better to allow access to the vanilla-specific files only through a vanilla-specific interface which will be considered an unsatisfied requirement if using it with OpenMW (see my example below).
So we have types File and VFSFile?
One interesting problem that I'm not sure has a good answer is how to deal with dependencies on specific versions, e.g. for mods like Tamriel Rebuilt which don't change the name of the plugin when new versions are released.

-There are changelogs included. I didn't download it but they are there as of the latest version so tying a plugin to it's version should be possible if arbitrarily difficult. Creating portable options to avoid this kind of difficulty is why I started the topic. Don't see how a project like that could succeed without version control.

-Wait, which is changing? Travels or TR? Both? Based on the pybuild I'd say that your difficulty is having a post-loading-dependency instead of a pre-loaded one.

Hashes of the files? Loot supports this, and it would also work for asset dependencies, but it's not really very intuitive and even with multiple hashes doesn't allow for versions unknown at the time of writing like patch versions which don't make substantial changes (not like many mods follow semantic versioning though).
Some sort of decentralized versioning system could also work, but would be less compatible with mods which aren't installed that way and makes the system a lot more complicated. E.g. each mod using this system could also install a file with relevant installed metadata including the name, version and the files it installed, which other mods can refer to. This is actually what I'd suggested on a different thread.

-This is portmod's repos correct?
Not sure where you're going with this.

Code: Select all

#Does the format need an ID&/version? Probably not but it might be better to cover the possibility and remove it later?
	#actually I'm arguing for it.
		#1) it enables people to find more information and make contributions (maybe even framework release files)
		#2) it acknowledges the possibility for outside contribution/improvement for detractors/alternatives
			#this probably won't be an issue but just for the sake of good will it's worthwhile

meta_standard: nopaque_bovine/whatever
meta_version: 0.5/whatever
	#Prepilot version

name: ScriptAssetsAlpha
version: 0.000000001
urls:
  homepage: null
  docs: N/A
  bugs-to: "TBD"
authors:
  - kaekaze

#acknowledgements:
#not strictly necessary but a common feature of readmes. Maybe just included as a comment-footer instead of metadata.
#  - GhanBuriGhan
#    - Morrowind Scripting for Dummies

license: https://creativecommons.org/licenses/by-nc-sa/4.0/
	#how detailed does this need to be? we could also include
  - year: 2022
  - summary: modOk, distributeOk, commercialProhibited, attributionRequired, licenseChangeProhibited

install_flags: 
  - HDTextures, HighResMeshes
configuration_flags: 
load_order_flags: 
  - someWierdCombo

source_urls:
  - direct: https://www.tbd.xxx/AssetScriptAlpha.tar.gz
#i suppose most applications can decide what type of download it is without us specifying direct/browser? that was just a suggestion but you're on board with it? Nexus is an issue and I like the way portmod handles it. Maybe the info contained in the url is enough.

archive_name: AssetScriptAlpha.tar.gz
#Do we address sub-archives? No opinions.
#Can archives include opaque-compression ftp-like transfers?

docs:
  - Data Files/docs/AssetScriptAlpha/*
#This just means that my docs are archived in the same place the files end up on disc right? No, that depends on the management system. Human/portmod/whatever. This just points to their location. Yeah, this would improve portability.

# No known conflicts
# format could be the same as requirements
conflicts: []
#Can the archive conflict? Obviously certain contents (maybe all of them) can?
#I'm assuming that's potentially what this means because of it's implied scope.
#If portions of the archive are being selected for installation/configuration/asset use, I don't see a reason to invalidate the entire archive. Though at that point it's just a resource and the metadata has no value anyway.
#Different versions of the same archive....this should be handled via the version&id check though.

install:
  - path: Data Files
    exclude:
      - TRUE HDTextures
      - Data Files/Textures
	#USE HDTextures is set, exclude the regular ones
    requirements: []
  - path:
#A tad wonky but if HDTextures is active copy DF/HDTextres to DF/Textures
     - TRUE HDTextures
     - TO Data Files/Textures
     - Data Files/HDTextures

#Overwrite control is an issue. Humans have our (inefficient) ways and managers have theirs I guess.

plugins:
  - path: Data Files/AssetScriptAlpha.esm
    # Not mandatory since it's a master anyway, but here as an example
    load_after:
      - Morrowind.esm
      - Tribunal.esm
      - Bloodmoon.esm
	#Nice, I like this notation.
    load_after:
      - TRUE someWierdCombo
	TheModThatHasTheScriptItHasToOverwriteOrWhatever.esm
#    load_before:
#      - UL_Chess_Add-on.esp
#      - UL_RoHT_1.52_Add-on.esp
#      - UL_3.5_TR_16.12_Add-on.esp
#    conflicts:
#      - Uvirith's Legacy_3.51.esp
#  - path: Data Files/Addons/UL_Chess_Add-on.esp
#    requirements:
#      - type: user
#I see this and immediately think of my login. Good though, makes perfect sense in-context.
#       name: Chess
#This is what gets inserted into a possible prompt then.
#       desc: Adds a functional chess board to the Tel Uvirith upper tower.
#Better with included desc, good call
#  - path: Data Files/Addons/UL_3.5_TR_16.12_Add-on.esp
#    requirements:
#      - type: File
        # Note: This should work with TR >= 16.12
        #       There's no easy way to formalize it
#Hmm. See post.
        path: Tamriel_Data.esm
#  - path: Data Files/Addons/UL_3.5_MWSE_Add-on.esp
#    requirements:
      # Always unsatisfied for OpenMW
#      - type: VanillaFile
#        path: MWSE.exe
#Maybe Mundane/LocalFile since Vanilla implies bethesda-origin. Might also be worth specifying a VFSFile type.

#Yes each archive should be handled separately.

# No known conflicts
# format could be the same as requirements
conflicts: []

A minimal implementation for this installer format won't guarantee consistency, since you really need to continuously track all requirements for installed mods. Installing something later could break the requirements (but a well-designed mod manager could track them all simultaneously and figure out how to fix problems).
I was thinking that a user shouldn't install a mod until all requirements are present. Installing something later could break- isn't this the purpose of tracking conflicts in a portable way? The user is altered when installing/activating a conflict; or at least it's documented. Auto-fixing would be awesome though.
This is designed to work only with the file system, but could be extended to also reference metadata from other installed mods, at the cost of worse compatibility with mods which don't use this system.
How does this compromise compatibility? There's nothing stopping a pybuild/whatever from being generated based on the contents. Even the USE cases are possible via the requirements. Probably need a flags array near the header though.
Not saying this is a done deal, but I'm not seeing an issue here.
Also note that the format is for one file per archive, the idea being that you store it in the root of the archive and it has a particular name, so it can be found easily by tools.
Separate metadata files would be needed for the patches, so it's not covering the same information as your example.
Aren't patches separate plugins or included plugins and therefore subject to the rules or "differently-compliant"? No insult intended, the phrase just hit me.
I think mlox has optional orderings, but I'm not entirely convinced they're useful. I guess it's sort of
like saying that a particular ordering is recommended, but otherwise the user can choose with a
custom rule (and of course that the rule can be broken if it would cause a cycle).
I think finding a friendly way to present rule-generation and selection options to the user is the answer here. If a mod has a lot of USE/Requirement cases like this then the metadata gets much more complicated but still comes from a finite selection of options. If I'm correct then it's a configuration preference, not an installation/publishing/workingLoadOrder problem.
Noted, particularly since there are some changes planned to the install interface and some of the
docs are going to be moved and rewritten.
It is in the APIdoc: https://portmod.gitlab.io/portmod/dev/p ... llDir.PATH
Also mentioned in the package guide: https://gitlab.com/portmod/portmod/-/wi ... kage-Guide
though maybe not as clearly as it could be.
I'll describe that wording as baroqitarian, for no reason.
But yes, my face is red. Sorry.

A few assumptions:
I'm assuming that if mod redistribution is prohibited then modification is valueless for publishing purposes.
Modification must include things like adding metadata or bundling with metadata otherwise mod-unfriendly-mods could be more easily curated.
Mods enter the public domain at some point.

Anyway, I'm tired and that was a lot to take in(tyvm). Be ready for more soon enough though.
User avatar
bmw
Posts: 81
Joined: 04 Jan 2019, 19:42
Contact:

Re: [PRE-REL] For ALOT of scripts.

Post by bmw »

configtool prompts for each change to the config file but inserts the notification about MERGE_TOOL between the changes and the choices. Is that expected behavior? The choices seemed to regard the MERGE_TOOL instead of the file changes.
That is expected, though I'm not positive it's the right choice. It's supposed to be a note at the top of the prompt to try and make the feature more discoverable.

I'll try to reword it so it's less ambiguous.
are these proposals easy enough to translate with automated tools?
I would suggest leaving the format the same for all languages (i.e. don't change the keys) but to make it possible for user-facing values like descriptions to be localised, which we could probably do inline, with each description having a mapping of locale ids to the localised text.
Creating portable options to avoid this kind of difficulty is why I started the topic. Don't see how a project like that could succeed without version control.
...
This is portmod's repos correct?
...
How does this compromise compatibility? There's nothing stopping a pybuild/whatever from being generated based on the contents. Even the USE cases are possible via the requirements. Probably need a flags array near the header though.
Not saying this is a done deal, but I'm not seeing an issue here.
Extra information referencing other mods would certainly be helpful for a tool like portmod which tracks all of that, but other tools which don't provide extra metadata themselves wouldn't be able to make use of it since they wouldn't know what version of a mod was installed unless it was using this system.

But I guess I was assuming that anything that supports this format does so in its entirety, as opposed to mod managers like wrye mash adding support for the more basic installation requirements and ignoring things like version requirements since it doesn't track that information.

That said, I don't think the version information will be very reliable anyway. Mods may have different version schemes which are ambiguous, or which compare differently, so any requirement which uses version comparison like minimum or maximum versions will not to be reliable without enforcing a version scheme, something that that would work for mods using this metadata format, but wouldn't allow them to reference mods which don't. So it's probably better to provide a mechanism for stating version requirements, but with a note that it's for reference, and not something tools are expected to use directly.

I was also thinking from the perspective of creating a simple command line tool which could create template files, validate the metadata file and handle installation. I think that might help with adoption outside of portmod and help avoid situations like other tools improperly implementing the spec and some mods only working on those implementations. It would provide a reference implementation, and portmod could just run that when installing mods which use this format. Having that tool handle version requirements would be complicated, but I guess it doesn't really need to, as long as there's still some easy way of passing that information to it. It also wouldn't really be feasible for such a tool to also handle config sorting at the same time, so there would be a significant part of the spec it wouldn't use (but it could still validate).

E.g. it would be nice to be able to run something like the following to have it install the mod.

Code: Select all

mwpack install uviriths-legacy/mod.yaml --enable-chess --enable-tr -o <outputdir>
But if you were to run it manually and don't pass any --enable or --disable flags, it should do its best to auto-detect whether or not to install the TR patch, and it won't be able to check version requirements without being a much more complicated tool.
#Does the format need an ID&/version? Probably not but it might be better to cover the possibility and remove it later?
...it enables people to find more information and make contributions (maybe even framework release files)
Version, sure. I usually assume that the first version doesn't need to be included in the file, since you can later assume that any unversioned file is for the first version.
An ID I'm not sure about. The only situation I think it would be necessary is if the spec gets forked in such a way that the different specs may behave differently even on the exact same file, so you'd need to know which spec was being targetted. Discoverability is a good point though.
The two could also be combined into one field. E.g. "mwpack_version = 1".
#not strictly necessary but a common feature of readmes. Maybe just included as a comment-footer instead of metadata.
I think acknowledgements are probably still best left in the README. I would think that the metadata file is more for tools and mod developers to use, while the README is meant primarily for users.
- summary: modOk, distributeOk, commercialProhibited, attributionRequired, licenseChangeProhibited
I'm hesitant to include such license tags, if only because (as far as I understand) they aren't legally meaningful if they're just supposed to summarize the license. You'd still need to read the precise terms of the license to know what you're allowed to do with it. And I certainly wouldn't want to have them be used by themselves in the way of nexusmods' permissions, as they aren't meaningful without a clear definition, and are barely better than the short "You can use this as long as..." informal licenses which lots of mods include.
But if we encourage the use of standardized licenses like Creative Commons ones, then it should make these things more clear anyway.
#i suppose most applications can decide what type of download it is without us specifying direct/browser? that was just a suggestion but you're on board with it? Nexus is an issue and I like the way portmod handles it. Maybe the info contained in the url is enough.
I thought that was a good idea for handling mods which don't have direct download links, as you can't always tell from the URL itself since not all direct download links include the name of the file to be downloaded (e.g. anything on mw.modhistory.com).
archive_name: AssetScriptAlpha.tar.gz
#Do we address sub-archives? No opinions.
Do we need to store the archive name in a metadata file which will end up going inside the archive? It could be used for creating the archive, except that I think this sort of thing could just be standardised using the other metadata. E.g. "{name}-{version}", followed by the extension (and whether or not this spec also determines which archive formats can be used is an interesting question. Requiring formats like zip and 7z which allow access to files without extracting the entire archive would allow the metadata file to be read before the archive is extracted).
Edit: Just realised that I had included an archive_name field in my earlier post, so presumably you were just including what I put.

Nested archives seem unnecessary and ugly.
#If portions of the archive are being selected for installation/configuration/asset use, I don't see a reason to invalidate the entire archive. Though at that point it's just a resource and the metadata has no value anyway.
Strictly speaking, yes, this is redundant and could equally be expressed by requirements on the individual components, however it's simpler to list that the mod as a whole conflicts with something, than to list it for everything.
Plus, there is a big difference between being unable to install a mod due to a conflict, and the mod installing successfully, but not actually installing everything since a conflict meant that everything was skipped. They may be functionally the same, but I think the first is much less confusing for the user.
#Maybe Mundane/LocalFile since Vanilla implies bethesda-origin. Might also be worth specifying a VFSFile type.
I'm not attached to the name. By VFSFile do you mean something which would only check the OpenMW VFS, or something which would also check "Data Files" when being used for Morrowind? It would be helpful if most mods could use the same interface and have engine-specific requirements only be necessary for things which only work on that engine anyway.
I'm assuming that if mod redistribution is prohibited then modification is valueless for publishing purposes.
I'm not sure that's strictly true. There are many mods which prohibit verbatim distribution, but allow creating derivative works (e.g. glow in the dahrk).
That said, I'm not really sure how it works legally. You can always make a trivial derivative, or even a derivative that just adds a metadata file, and suddenly it's not verbatim redistribution, however even if that satisfies the "license" (there are no formal licenses I've seen which say this), it's probably not really in the spirit of what was said and you'll probably just end up getting the author annoyed at you. Presumably they mean derivatives are allowed for the purpose of creating mods which use the assets, or making substantial changes to the mod, but if the idea is they don't want verbatim distribution, I don't think re-packaging and adding metadata meets their expectations.
Mods enter the public domain at some point.
Probably not in our lifetimes, unless the author explicitly makes it public domain.

Another thing I think is worth mentioning, given you said you live off the grid most of the time and mentioned omwllf which is a little old and buggy, is that plugin merging tools for Morrowind have improved significantly recently, which significantly reduces the number of plugin issues which would have previously required load order changes.

DeltaPlugin, a tool I wrote, provides merging of all record types (which can be meaningfully merged), though it supports OpenMW only.

TES3Merge is for Morrowind, and it sounds like the latest release even provides basic support for OpenMW. While I think it does now support all records (it was lagging for a while) it still doesn't properly merge records when something is removed from a list (See #21; omwllf also has this issue, but not DeltaPlugin).
kaekaze
Posts: 6
Joined: 07 Aug 2022, 18:05

Re: [PRE-REL] For ALOT of scripts.

Post by kaekaze »

Okay, the metadata is basically meaningless without some way of determining the latest (available-to-user) release, release dependencies/versions, and the dependency versions available to the user.

WARNING: Rant ahead!

LOL! So, it would be a victory if mod makers put this file in the archive root:

Code: Select all

release.txt:
release: 5
	#I incriment this by 1 with each release of the mod.
year: 2022
month: 5
day: 24
count: 3
	#I released 3 new versions on 2022/05/24, aren't I productive!
<EndOfFile>
This definitely needs to be tackled first, didn't consider it. Hashes are definitely the right way to ID files /wo standardized metadata. I can see now why you're leaning away from human management too.
-a) The meta data would have to be stored externally and few would ever bother getting it. Most would just debug their installation instead.
-b) An automated tool does all of it, using the metadata in a repo like portmod.
-c) Hashing is something almost no one does without an automated program. However, once the file is known you still can't really chose an updated version over a previous one without consulting an outside resource like portmod's repos or asking the user to manually check the mod's files so metadata is rendered almost useless.

There (was) no meaningful place to store metadata!

Just now getting the big picture; what assumptions.
"Of course any mod author has an operable version for their mod."
"Of course we can help author's maintain their metadata." Eugh.

Worst of all, because of the inertia built up by so many ignorantly-released (including mine) mods, conforming to a standard now would be virtually meaningless without another 10+years of content generation(if we're lucky). So as well meaning as I am, this is fighting an entrenched culture and working up metadata in this fashion wouldn't help anyone without sharing in the external platform that holds the relevant metadata.

</RANT>
<FOR MODDERS NEW TO THIS TOPIC>
What I didn't understand is that while the well-meaning among us want to (and can) publish responsibly it will have virtually no effect in the current system.

So you've created a perfect metadata format! Cool, where do you put it?
-In your mod archive's root, obviously! Yes, it works, we're crawling in the right direction. In all seriousness: well done, but keep reading.

Then what? Thousands of mods we have no ability to modify lack this data. Even making them downloadable from someone's website like mw.modhistory is almost legally impossible. Check the permissions section at the top (as of 2022).
-So, after a year we have a hundred mods(haha) that use this system. How do we make the rest compliant? The mod authors have no reason to help and in many cases, use assets they can't fully speak for! They can't help.

I hear you say, "so we need a middle man to host the metadata. Surely we could publish the metadata to a mod of it's own and host wherever."
-Now you've created a mod which holds all the metadata for mods. You sir, are correct. No sarcasm. This solves the problem, but keep reading.

Why would a human (who can read a readme) download this metadata? It serves no purpse. Sure, some irresponsibly released mods could benefit from it and if you want to write those readmes and lump them together somewhere: check your legals and go nuts! The community could seriously use that help.

But, what if this metadata did more than just record the mod, it's needs, it's possibilities, it's dependencies, it legalities, etc. If you're going to host metadata, why not make it work for you?
-Enter portmod, which provides a platform for the metadata, while coordinating distribution and installation.
--You want a mod library? Open the .pybuild files there. All the links are present.
--Don't like portmod manager? Download them directly and manage them yourself.
--Better yet, get a program like grep and dump ALL THE links into a single file. Now you're entertained exploring the content listed without even playing the game! Sorry, got carried away.

"But mod managers aren't portable, they lose maintainers, they die out, hosting goes down!"
-True of EVERY SERVICE.
-ANY library of metadata you create could go down. That's the problem? Help portmod/someone by providing mirrors for the metadata they already host!

"But I want the data more accessible!"
-Then build a batch/bash/python/c/assembly program to convert the data into a more accessible format! We could all use more human readable documentation of computer programs! Do you have ANY idea what software developers would PAY for a program that document's their code for them?!
Anyway. Hopefully that helps others buy into portmod's repos.

I actually owe any readers an apology: Due to my lack of understanding I was off topic on my own topic for almost every post above this one.

All that said:
bmw, what do you need/want from me (if anything) in terms of metadata. Yes, I want to maintain my own mod.
Post Reply