Page 1 of 3

Addin database

Posted: 05 Jan 2019, 09:40
by Jwcolby54
I have been a database developer, SQL Server (that you would care about) and C# programmer. I am wondering whether an actual database has been developed for storing objects. It looks like stuff is stored in text files. I have read (over, not memorized) the format for the TES3Mod file formats found here

Coming from databases, it occurs to me that if objects were imported into an actual database, then using SQL, comparisons could be done between objects to find and flag collisions between objects. I am just starting to think about this but things like physical position in the "real" world could be analyzed to see where addins might overlap, find objects which collide with walls or are half buried in the ground. Long ago I remember adding in addins only to discover that rooms in caves defined in one addin collided with rooms in other addins.

I don't know the capabilities of the construction set but it feels like it would make a poor tool for this purpose if it did so at all.

Do tables (definitions) exist for the various objects in such files? Does code exist to parse an object file and get the pieces into tables in something like PostgreSQL or MySQL. Lacking that, does code exist that parses such files and passes back the pieces as data objects that I could then write code to place into tables?

It seems like with the hundreds (thousands?) of addins available, it would be pretty useful to place them all in a directory, then import all of them into a database, then use a database engine to analyze each one against every other one. Even against the various buildings, crypts, caves etc in the Morrowind world.

I have an overly powerful Unraid server on which I use Docker for Plex. I know that there are docker containers for MySQL and MariaDB to name a couple. That might be a good tool for implementing something like this.

I am volunteering to take on this project if it is deemed useful. However I would rather not reinvent the wheel if any or all of this has been done already.

Thoughts anyone?

Re: Addin database

Posted: 06 Jan 2019, 03:10
by bosmer
Would this become another app separate from the openmw-cs used for the purpose of checking compatibility of datafiles and addons?

Re: Addin database

Posted: 06 Jan 2019, 06:38
by Jwcolby54
I am looking for guidance for what makes sense. I am also looking for information on what already exists. I am studying the CS docs now.

When I read the CS guide they talk as if items are records in tables, they discuss parent / child, database kinda stuff. But nowhere does it actually say that a real database engine is used. In fact Google shows absolutely nothing in that regard, which makes me think that there is none.

My concept is to build a real database, using a database engine such as MySQL. Define tables that match the objects discussed in the CS. Then read in the various objects, both in the MW world and in add-ins. A database engine allows the use of SQL to perform analytics. I did see a number (11K ish) of objects in the Morrowind universe.

The system apparently uses object definitions (ring / shirt / sword etc) then distributes instances of those objects out in the world, with properties filled in to make them specific types of rings / shirts / swords etc. So there might be dozens or hundreds of rings scattered around in chests, on tables etc. but they are all just instances of a parent object - ring.

I haven't yet gotten into how spaces are defined - caves, rooms and so forth. I am seeing "mesh" thrown around and I assume that a mesh defines boundaries in space, but I haven't yet located a discussion of how that might work.

In any event the idea is to catalog every object type, then every instance of every object type. Once that information is in a database, then how it is used is up in the air. Location of every ring in Morrowind? Count of rings? Location of every potion? Every object with a given enchantment? It could probably map Morrowind, objects, quests, NPs, buildings, caves, everything.

How it ties back in to OpenMW is up for discussion. User interface would tend to be determined by the intended user. Databases are easy enough to use if the UI is clearly defined. I have to say that all the discussion of "filters" to work with items feels clumsy, and it feels clumsy I think because it is not based on an actual database engine.

Re: Addin database

Posted: 06 Jan 2019, 21:08
by psi29a
I don't, immediately, see the benefit of having of having a RDBMS for OpenMW-CS or more the point, what problem it would be leveraged to fix?

The content of files being read in are binary data files, not text files.

If there is to be a SQL like solution, it would probably be SQLite ( and not a full blown RDBMS. However I don't believe this is even necessary considering that the content of the files are read in and mapped in memory anyway just like a key-value store (NoSQL).

Again, what problem are you trying to solve?

At this point, where more interested in moving the OpenMW-CS forward, adding features. :)

If you would to develop something for yourself, please do. Be sure to work through the developer checklist and make a gitlab account to track and create issues along with cloning our repo. :) ... ion_Wanted

Re: Addin database

Posted: 07 Jan 2019, 02:02
by Jwcolby54
I don't necessarily see it as a data store for the game engine, though it would solve problems if that were done. If a binary file works for you it works for me. But a binary file is not a database, it is a blob. It cannot be analyzed in any easy way. It cannot be viewed to see what is in there. It cannot be directly edited, added to etc.

I am trying to solve the problem of what add-in collides with what other add-in, or objects in the game. Add-in authors have no clue what their add-in may damage in the game. I have actually seen the following... An author makes a new house on a hill. Another author likes that hill and makes his house on that hill. Collision if I try to add in both add-ins. But how am I supposed to know that? Maybe it sets next to the other addin? Maybe one of my rooms exists in the same space as one of his? The person installing addins is flying blind as to what can and can't be used alongside what he is installing next. Does the engine throw a meaningful (to the user) error when two addins collide? At run time AFTER he has already done all the install stuff necessary? Or even at all?

How about a database (real database) where the entire world is loaded, then every addin is loaded. Everything is checked against everything else. AddinX collides with Addin K. The database allows a user to select addins and throw up errors in real time as they select things they'd like to install. "You can't add that next thing because AddinK in the list you already selected collides with this next one". An addin author can design houses and caves and immediately process it against houses and caves already in the game, as well as against all know existing addins.

I can do that with a database. Can you do that with a blob? Do you already provide such a tool?

Re: Addin database

Posted: 07 Jan 2019, 11:45
by psi29a
That's an mod issue regardless of a blob or a database.

Regardless of binary or database, the files are read into memory. Once in memory we do with it as we please, having an extra layer (several with a RDBMS) of abstraction to go through would actually mean: more infrastructure, maintenance and latency.

I mean, we do this already with OpenMW. (Otherwise you wouldn't be able to play Morrowind on OpenMW) ;)

From the OpenMW-CS point of view, you'll see the problem directly if you have the original mod with the house on the hill then you can perhaps move your house to a clear spot yourself visually (3d) and fix your mod. Or you can remove the original mod and solve it that way too. I don't believe it is the engine's job to handle static collision, I don't think any game engine does that either. That is the job of the mod author. It's common when you ground-cover mod along with another mod that changes the heightmap/land in the area. Then you have floating grass.

It would be better to have this all managed for us in OpenMW-CS where the mod author can see the effect of their mod with all other mods loaded, which it does now.

So again, what problem are you trying to solve here? (Apologies sound like an broken record here, but I'm still not seeing how a broken mod's mesh can be resolved with a RDBMS.)

Re: Addin database

Posted: 07 Jan 2019, 14:23
by Jwcolby54

I appreciate the dialog. Nobody else is responding so there we go.

So look, I am not saying that it is the game engine's job to handle this. It is not the CS app's job either.

"It would be better to have this all managed for us in OpenMW-CS where the mod author can see the effect of their mod with all other mods loaded, which it does now."

"where the mod author can see the effect of all other mods loaded" is precisely the issue. CS doesn't know anything about mods NOT LOADED. And mod authors aren't loading 4,769 other existing mods to check what their mod collides with.

THAT is the problem I was looking at solving.

There are an unknown number of mods out there. There is no way to know what collides with what until the poor user starts loading mods. Then all hell breaks loose. When you add more than one mod (and something goes wrong), the common wisdom is to read for hours and google and fiddle and fart around checking and unchecking and changing load order, and clicking the heels of your red shoes together and muttering "I'm not in Kansas any more" and hoping that you figure this out.

THAT is the problem I was looking at solving.

Imagine a tool that loads the game data. It provides a directory where 1 or 1 thousand mods are stored. Mod authors could drop their mods into a directory. Mods accumulate in the directory.

The app loads each mod. Doesn't care whether they collide, just loads the data in. THEN once all the data is in, it moves through the data, checking each mod one by one to check for collisions with the game data. And collisions with the game data are noted in an error table.

Then it moves through the data checking each mod against every other mod for collisions. Any collisions are noted in an error table.

When that process is done a report is generated saying

Mod X1 collides with ...
Mod a42 collides with...
Mod k101 collides with...

Mod authors could use the tool to discover collisions between not only their mod and the game, but their mod and the other known and tested mods.

Users wanting to use mods could check the results to see if there are any collisions between the mods they want to install.

It would no longer be urban legend... "We know that this mod collides with XYZ because someone out there found and actually noted it somewhere and google found it"

It would be "The tool says this mod doesn't collide with anything else tested." or alternatively "this mod collides with these things, so don't use these things together."

>>>"That's an mod issue regardless of a blob or a database."

That is precisely correct, it IS a mod issue. That said, A game gets a reputation. The original Morrowind was buggy. You guys are straightening out the engine to make that side of things go away. But if we want to fully support MODS, then providing a tool to test mods against each other would go a long way towards lowering the USER'S frustration as they try to use these mods.

You guys are doing a great job of cataloging bugs in the game engine, working through those bugs fixing them, adding features etc. GOOD JOB.

The CS authors are doing a great job of getting that tool up and running, working through bugs and fixing them, adding features etc. GOOD JOB!

Then you provide a solid working OpenMW engine, a solid working CS engine, and the mod author is allowed to build any mod he wants, placing objects anywhere he wants, completely blind to other mod authors around him. The mod author is NOT provided any feedback whatsoever that what he is trying to do is going to cause problems. A Mod author can and will decide "I like this hill and other mods be damned, I'm putting my mod there." But at least the mod author would KNOW that the hill is already taken by another mod. Now the USER (who has no control over ANYTHING other that finding and installing the mods they want to try) is stuck figuring out and cleaning up the resulting mess. With no tool at their disposal to KNOW "don't use these mods together".

I mentioned that ages ago I played the game for hundreds of hours and loaded dozens of mods. I actually ran into mods (Balmora specifically) where mod authors created huge castles, houses with caverns etc. on a hill on the west (?) or left side of the balmora map. That hill seemed popular with modders. IIRC (and it was long ago so I might not) the game engine just merrily loaded in the conflicting mods without complaint. One way or the other, it was on ME, the user to to figure it out, with nothing at my disposal except "download and try".

So maybe what I am proposing isn't doable for some technical reason that I am not aware of. I was hoping to get feedback from folks working on the game to comment in that regard. The thread has generated a ton of reads which indicates an interest in the concept.

Re: Addin database

Posted: 07 Jan 2019, 14:32
by psi29a
You may address me as psi29a, mindwerks is just my website/business. ;)

Now I have a better understanding of the issue and sounds like an interesting idea.

Would it be interesting to have something like this running as a online service with an pollable API?

Front-end using Twisted (async python) which offers HTTP/REST to the public that then does queries to a RDBMS or NoSQL database filled with mod entries?

One option would be to use MongoDB which uses document storage, treating every mod as a document (common and un-common/non-structured fields)?

The project could be fed mods, by version, and parsed into the application to later be queried? LIke for example; position data of all meshes? So no IP would be store on the server, just the metadata which is legal allowing you to sail around any legal issues. :)

OpenMW-CS could then have a call to this API to check a modder's mod with existing mods for any collisions? A 'check collisions with existing mods' button?

Re: Addin database

Posted: 07 Jan 2019, 23:27
by Jwcolby54

Yes, that is the general idea I had in mind.

A database of mods.
A means for Mod authors to check their mods against other mods, and store their own finished mods in the database.
A means for Users to check mods they were interested in, perhaps read descriptions of what they do.
A means for users to select and then check collisions between the mods they are interested in.

Assuming 5000 existing mods, a central location to store, test and even discover mods by authors. and allow users to find and select mods desired would be valuable I think.

Re: Addin database

Posted: 08 Jan 2019, 06:42
by bmw
So, if I may be permitted to jump in, like a mod repository? Something I suggested here, admittedly very recently.

Though some of what you've suggested (categorized data about every object in every mod) is above and beyond what I was thinking of, though not actually necessary for users, just modders and for building the repository (such information could be used to generate lists of conflicts for a mod etc. as you've mentioned).
For the mod repository I was thinking of structuring it like a Linux package repository (or CKAN, if you want an example of this done with mods), implemented most simply by a git repo containing text files (maybe not ideal for storing object information, but we could have both the mod repo and a database of mod objects). For each mod a file (e.g. this) would contain the information relevant for users installing the mod (or more accurately, for an installation tool) such as dependencies, conflicts with other mods, download links and commands necessary to get the mod working with OpenMW.
Users would then have a local copy of the repository which would be referenced when installing mods.

The idea of including details on everything within each mod to automatically detect conflicts is a great one, but an issue I feel should be pointed out is that, depending on the amount of information involved, there may be issues integrating old mods. A very large number of mods are not distributed with licenses that allow redistribution, or even derivatives, and we would not want to limit such a project to mods that have been newly submitted. I'm really not sure that a database containing detailed information about every single item in a non-libre mod (or for that matter, the original Morrowind data files) would be something that we should be distributing (Note: I'm not a legal expert).

Edit: Also just saw this thread. However the conclusion there of using modding-openmw as a compatibility resource is wholly inadequate for anything automated. It could become a web frontend for the aforementioned repository though.
At the end there is mention of a backend using a database, but I feel I should point out that a database is not a good idea (for the mod repo that is, I don't object to using one for your idea of storing objects for detecting conflicts) if we want automated installation of mods to be a possibility, as we should be treating it more as a source repository (i.e. Gentoo, or the Arch User Repository) due to the amount of modifications that may need to be made to get a mod working than a binary repository (like Ubuntu or CKAN). A binary repository could use a database, but using one for the source repository would overcomplicate things, as each file in the repository is essentially a script.