[PRE-REL] For ALOT of scripts.

Post about your mods, learn about OpenMW mod compatibility, check for problematic mods, discuss issues, and give us feedback about your experience with modded OpenMW.
User avatar
bmw
Posts: 64
Joined: 04 Jan 2019, 19:42
Contact:

Re: [PRE-REL] For ALOT of scripts.

Post by bmw »

kaekaze wrote: 15 Aug 2022, 00:53 Okay, the metadata is basically meaningless without some way of determining the latest (available-to-user) release, release dependencies/versions, and the dependency versions available to the user.
I'm not sure if I entirely agree. There's a lot more information on the proposals in this topic than most mods usually provide, and having a more standard way of providing that information would be helpful, even if it's mainly portmod which makes use of it.

One way of providing reliable mod identifiers and versions would be to set up an append-only metadata repository (sort of like some software repos like pypi.org or crates.io) which, for the sake of making it easy to add to, only requires the identifiers and versions. It it stores the versions in chronological order that should deal with version comparison issues (which occurred to me when you mentioned dated versions). I feel like there may be exceptions where there are backported patches released for old versions of mods, but this practice is at least far less common than for software, and I can't think of any examples.
But setting up a metadata repo which can be easily contributed to by mod authors would be complex, and I'm not convinced it's significantly better than just listing the mod name and compatible versions in the metadata file.

I've never actually looked at the fomod spec, but it bears comparing, as I don't think this is very useful unless it provides significant benefits over that (the actual spec is a pain to read, but the tutorial gives a nice overview). My conclusions:
1. The fomod spec provides only a formal installer spec (ModuleConfig.xml). They have a suggested spec for info.xml, but only include name, author, description, version (they recommend semantic versioning though!), website and id (unclear).
2. Fomod is an install wizard format and probably isn't entirely compatible with a system like portmod which has its own way of handling optional features (it defines install "steps" rather than features/patches).
3. As far as I can tell, while there is a standalone validation tool, there is no standalone installer. While I don't think most people would care too much right now, it means that it's only useful while mod managers directly support it. But if there was a standalone installer you could still use it as long as that works.
4. It's an xml-based format, so it's neither easy for humans to read, nor easy for humans to write, and the only WYSIWYG editor, the fomod designer, is abandoned and incomplete. On the other hand, a yaml or toml-based format would be much easier to write manually.

I think it's also worth noting that even getting newly released mods to provide metadata would be a huge benefit. The nice thing about old abandoned mods is that they don't change. Portmod, for example, only has to package them once, and users don't need to update them, since there aren't any updates (at least, unless they get revived, either directly, or by a different author). Plus, it seems to me that people are mostly interested in newer and more actively maintained mods anyway (though this is mostly anecdotal I'll admit).
kaekaze wrote: 15 Aug 2022, 00:53 Anyway. Hopefully that helps others buy into portmod's repos.
Do you think that support for the original Morrowind engine would also help? I think we should be able to use a BSA-based VFS much like how we're handling Oblivion and Fallout NV/4, though there are more files which can't go in Morrowind BSAs. I think we're more or less at the point where it would be feasible (there are some changes to the repo code in openmw-mods which will be necessary), but I have no idea what sort of interest there would be.
kaekaze wrote: 15 Aug 2022, 00:53 bmw, what do you need/want from me (if anything) in terms of metadata. Yes, I want to maintain my own mod.
If you want to create .pybuild files for your mods that would be helpful.
See the package development guide, and openmw-mods's CONTRIBUTING.md for details about the format and the packaging process.
kaekaze
Posts: 6
Joined: 07 Aug 2022, 18:05

Re: [PRE-REL] For ALOT of scripts.

Post by kaekaze »

The easy stuff:
Do you think that support for the original Morrowind engine would also help? I think we should be able to use a BSA-based VFS much like how we're handling Oblivion and Fallout NV/4, though there are more files which can't go in Morrowind BSAs. I think we're more or less at the point where it would be feasible (there are some changes to the repo code in openmw-mods which will be necessary), but I have no idea what sort of interest there would be.
Not sure where you're coming from with the BSAs. My computer is running openmw flawlessly but can just BARELY fail to handle FO3:Anchorage due to using a software sound mixer.
As for supporting the original engine? Can't say much except that I'm not doing anything on it and haven't for years.
Maybe the utilities like MGE and MWSE are a decent argument for it but I'm not the one to ask. If you're referring to the above posts, I was tossing ideas and going way out there. I suppose I could change the plugin extension to .omwaddon before release to remove ambiguity.
If you want to create .pybuild files for your mods that would be helpful.
Working on it already.

The hard stuff:

Okay, so all this has been seeping in slowly.
Every time I try replying it just doesn't go anywhere worthwhile. Maybe lack of context/experience.
I have actually written a TON of replies. Even trying to convert the fomod XML to yaml with bash. Maybe one day.

There is one identifiable problem though and it keeps throwing me: versions and ids. So here's a wacky proposal.

Say the ArchiveMetadata and PackageMetadata were split up. They kind of already are via the Manifest files. An extended version of the Manifest could be stored in a table/database.
Here's a proposal for a format similar to the manifest file:

file: _header

Code: Select all

#[SERIES_ID]	[RELEASE]	[ARCHIVE]	[URL_TYPE]	[URL]	[UNKNOWN]	[ALGORITHM]	[HASH]	[CHECKSUM]	[CHECKSUM_VALUE]

Code: Select all

file: arch-misc_glow-in-the-dark.manifest
arch-misc_glow_in_the_dark	99	Glow_in_the_Dahrk-45886-2-11-1-1615063427.7z	BROWSER	"https://www.nexusmods.com/morrowind/mods/45886"	7761079	BLAKE3	b5b15557514dbf5d1900b1e3ec0fc021ed78df468ad20ed379304339dae44467	MD5	b0a845b3786296a804cf279fdb67a1b1
arch-misc_glow_in_the_dark	100	Glow_in_the_Dahrk-45886-2-11-2-1617457025.7z	DIRECT	"https://www.nexusmods.com/LETS_PUT_THE_DIRECT_DOWNLOAD_FIRST!"	7761097	BLAKE3	2c2edaf0dd00de742f62ce77af104836d65c1e51d33d44d22b97701a4ebe2751	MD5	eeabb5f8d9758d66bc962241365a7602
arch-misc_glow_in_the_dark	100	Glow_in_the_Dahrk-45886-2-11-2-1617457025.7z	BROWSER	"https://www.nexusmods.com/morrowind/mods/45886"	7761097	BLAKE3	2c2edaf0dd00de742f62ce77af104836d65c1e51d33d44d22b97701a4ebe2751	MD5	eeabb5f8d9758d66bc962241365a7602
arch-misc_glow_in_the_dark	100	Glow_in_the_Dahrk-45886-2-11-2-1617457025.7z	BROWSER	"https://AN_ALTERNATE_URL.com"	7761097	BLAKE3	2c2edaf0dd00de742f62ce77af104836d65c1e51d33d44d22b97701a4ebe2751	MD5	eeabb5f8d9758d66bc962241365a7602
#READERS: compare the download links for glow_in_the_dark:100

Code: Select all

file: FAKEglow_in_the_dark_supplement.manifest
arch-misc_FAKEglow_in_the_dark_supplement	100	FAKE_Glow_in_the_Dahrk-SUPPLEMENT.7z	BROWSER	"https://www.nexusmods.com/morrowind/mods/SPPLEMENT"	[UNKNOWN]	BLAKE3	[UNKNOWN]	MD5	[UNKNOWN]

Code: Select all

file: FAKEglow_in_the_dark_series_2.manifest
arch-misc_FAKEglow_in_the_dark_series_2	100	FAKE_Glow_in_the_Dahrk-45886-2-10-3-1609634335.7z	BROWSER	"https://www.nexusmods.com/morrowind/mods/FAKE"	[UNKNOWN]	BLAKE3	[UNKNOWN]	MD5	[UNKNOWN]
[SERIES_ID] is just a more/less arbitrary id agreed on by contributors to the repo.
[RELEASE] starts at 100 just to leave some breathing room but should always ascend.

Any repository could require the fields to be present before including the archive.
This would give all archives a unique-relatable-ID and an ascendancy-order independent of the archives, packages, or metadata.
This also means fewer packages need to be updated in the event of a change or new release as they could specify:
-[glow_in_the_dark, LATEST]
or
-[glow_in_the_dark, 99]
--If this is NOT the latest available a manager can prompt the user for a choice between package-specification or latest-version or abort.
or
-[glow_in_the_dark, ONLY99]
--to signify that this is the only working release for the package

Should an archive release fork significantly from previous versions it can just be classified into a different series_id.

You could store this database in the separate packages as with the Manifests but a package trying to pull in an archive from another package would need to maintain duplicate data or have a referencing system. Cumbersome.

Instead consider a manifest catalog directory on your repository. In this system each archive-series would have it's own file with the data formatted as above.
dir: archive_catalog_or_whatever
-file: -DATABASE
-file: _header
-file: arch-misc_glow-in-the-dark.manifest
-file: FAKEglow_in_the_dark_supplement.manifest
-file: FAKEglow_in_the_dark_series_2.manifest
From here a simple cat * > -DATABASE would pump out a unified database of all the available archive data.
Plus, if you did need to alter package directories on your repo, it wouldn't invalidate the series_id which would need to be updated if they were stored in packages and being used to generate paths or reference other packages.

Another option is to keep the manifests in the packages and just use a:

Code: Select all

find -type f | grep Manifest | while read ln; do
	IFS=
	cat "$ln"
done > "$PATH/-DATABASE"
As for human-maintainability; as long as there is a link to the -DATABASE from the package, the user can download and browse at will.
Contributors would probably have an easier time too but I don't actually know that.

*****
As for my mod: Here's the library in it's ~0.25 state. Still compiling features from my other mods into it. Commenting, improving, debugging, etc.
Attachments
lib_extant.esm
(870.69 KiB) Downloaded 13 times
User avatar
bmw
Posts: 64
Joined: 04 Jan 2019, 19:42
Contact:

Re: [PRE-REL] For ALOT of scripts.

Post by bmw »

kaekaze wrote: 02 Sep 2022, 20:58 Not sure where you're coming from with the BSAs.
Mostly just wondering if Portmod would be be more widely used, and if more mod authors would be interested in it, if it also supported the original engine. The aside about BSAs is just some technical stuff that would be necessary to support it (BSAs can act like the separate mod directories which OpenMW supports but Morrowind doesn't).
kaekaze wrote: 02 Sep 2022, 20:58 Say the ArchiveMetadata and PackageMetadata were split up. They kind of already are via the Manifest files. An extended version of the Manifest could be stored in a table/database.
Here's a proposal for a format similar to the manifest file:
I'm not sure I entirely understand what you're trying to do with the more complex manifest files. Are these supposed to be included with a mod, or as part of metadata in a portmod repository?

However, I think you're trying to solve the wrong problem. Mods which use some sort of metadata specification can be required to also follow a specific version format. Similarly, mod metadata in a package repository like portmod's can be adapted to follow a specific version format. So in either case, version ordering isn't an issue.

I think the only time it is an issue is for a distributed metadata file format, which can't reference the versions of other mods in a consistent manner without some sort of centralized (or even decentralized) registry. Portmod provides such a registry, but it doesn't specify a metadata format which can be included with mods.
Having a registry mod authors could submit their metadata to isn't a bad idea, but would require some more complex infrastructure, and need relatively wide adoption for it to be worthwhile.
Post Reply