Addin database

Join the team. This area is for people who want to participate in OpenMW's development in one way or another.
silentthief
Posts: 375
Joined: 18 Apr 2013, 01:20
Location: Currently traversing the Ascadian Isles

Re: Addin database

Post by silentthief » 08 Jan 2019, 14:52

The topic of mod compatibility and being able to better determine mod errors has been a common topic lately. Perhaps there is a way that we can offer a way to make mod conflicts easier to find? (That was my motivation behind my post here -> viewtopic.php?f=3&t=5671) Or perhaps a database would help with conversation topics? I don't know.

The idea of a morrowind db (MWSQL if you like) may be useful, likewise a repository and mod conflict list would be helpful as well. I am not a dev, but I do know that the devs that are working on the project are hesitant to put in feature creep adding new features especially before 1.0

ST
"Hurry, hurry! Last boat to Solstheim! Until the next one. Hah-ha-hah."

User avatar
psi29a
Posts: 4619
Joined: 29 Sep 2011, 10:13
Location: Belgium
Gitlab profile: https://gitlab.com/psi29a/
Contact:

Re: Addin database

Post by psi29a » 08 Jan 2019, 20:05

I think that this out of scope of OpenMW and more a general Bethesda modding issue. If someone would make something we could call against (via API) or as a library/plugin...

Don't forget that python is still on the table for openmw-cs. :)

Jwcolby54
Posts: 21
Joined: 01 Jan 2019, 08:48
Location: NC
Contact:

Re: Addin database

Post by Jwcolby54 » 09 Jan 2019, 17:37

BMW, I just read the thread you pointed me to. My thoughts in no particular order :

1) Someone pointed me to SQLite which is an obvious choice for this task. While it has certain issues for a "big data guy" (me) in particular the dearth of defined data types, that doesn't seem to stop folks from using it. And in most regards it seems to be a stellar choice.

2) While I want to learn python, OMW and MWC are written in C. With an eye toward possible integration it would probably be good to use C.

3) By loading the mod data into a database, it becomes possible to "discover what's in there".

4) I think a mod "repository" is a great idea, but a mod database is not that. In fact afaict a database and app around it could deconstruct all the pieces of a mod and store the pieces right in the database. Thus the app could open the cfg file, open the openmw directory, build out the directory structure, drop the pieces from the database into the directories, and update the cfg file. That is NOT what I was wanting to do but it could be done.

5) Immediately what I want to do is to break into the CS and where the code loads stuff from the mod into the CS, ALSO load it into my database. IOW I REALLY don't want to write a new parser to do that!

6) immediately what I want to do is where the CS saves back out to the file, ALSO store it back out to my database. IOW I REALLY don't want to write a new writer to do that.

7) once I have a database full of tables that can store any object from the game, and I am tied in to the CS in this manner, simply by loading a mod into the CS I have it in my database.

Sounds simple! 😉🤔😕

What specifically is done with it from there is up for discussion.

Listening to the creator of SQLITE, he talks about apps using (his words) "piles of files" as the data store for applications. IOW directories of directories of files. Sound familiar?

He discusses replacing all that with a single file which is a SQLite database. Which makes a lot of sense to me. But I have to come up to speed on it, get the table infrastructure designed, and a method to load the database up with mods, and (in the absense of openmw using the dB directly) a way to spit it back out into a "piles of files" format.

Which has little to do with mod analysis. 😁

Jwcolby54
Posts: 21
Joined: 01 Jan 2019, 08:48
Location: NC
Contact:

Re: Addin database

Post by Jwcolby54 » 10 Jan 2019, 02:07

psi29a wrote:
08 Jan 2019, 20:05
I think that this out of scope of OpenMW and more a general Bethesda modding issue. If someone would make something we could call against (via API) or as a library/plugin...

Don't forget that python is still on the table for openmw-cs. :)
Agreed outside the scope of the game engine. My concept of a database to hold and analyze game mods dances tightly around the cs however. I am studying SQLite atm.

My background is programming clear back to the early 80s and databases since the early 90s. I spent about 2 years learning C# and using it to heavily automate SQL server. I haven't programmed in C for several years though so I have to get a dev environment set up and then back up to speed. I actually have a vm on a server "ready to go" but it uses VS 2015 and C#, and I think I'd rather look at something like vs code and C++ for portability.

Anyway lots to do, just getting started.

User avatar
bmw
Posts: 23
Joined: 04 Jan 2019, 19:42
Contact:

Re: Addin database

Post by bmw » 10 Jan 2019, 02:57

Jwcolby54 wrote:
09 Jan 2019, 17:37
4) I think a mod "repository" is a great idea, but a mod database is not that. In fact afaict a database and app around it could deconstruct all the pieces of a mod and store the pieces right in the database. Thus the app could open the cfg file, open the openmw directory, build out the directory structure, drop the pieces from the database into the directories, and update the cfg file. That is NOT what I was wanting to do but it could be done.
On reflection I realise that the meaning of my earlier post may not have been entirely clear. While the mod database is certainly distinct from a mod repository, largely I wanted to point out that some of the benefits you had said you were hoping for would also be satisfied by a mod repository, particularly the below:
Jwcolby54 wrote:
07 Jan 2019, 23:27
A means for Users to check mods they were interested in, perhaps read descriptions of what they do.
A means for users to select and then check collisions between the mods they are interested in.
I also wanted to point out how useful this would be for the mod repository idea, as it could massively help with generating lists of conflicting mods. The database could also probably be set up to auto populate from the mods in the repository, so the two projects could be mutually beneficial.

It occurs to me that the solution to the legal issues I mentioned before could be to only expose a conflict checking API for the database and keep most of the details of the mods in the database hidden, then you're not redistributing the data.
silentthief wrote:
08 Jan 2019, 14:52
The idea of a morrowind db (MWSQL if you like) may be useful, likewise a repository and mod conflict list would be helpful as well. I am not a dev, but I do know that the devs that are working on the project are hesitant to put in feature creep adding new features especially before 1.0
The mod repository is certainly outside the scope of the game engine and isn't really feature creep in that respect. I'm also perfectly happy to do it myself, or at least most of the infrastructure, and already have a mostly functional prototype version. Adding mods to the repository (which I plan to automate as much as possible) and making sure their build files are correct will certainly benefit from the help of others though.

Jwcolby54
Posts: 21
Joined: 01 Jan 2019, 08:48
Location: NC
Contact:

Re: Addin database

Post by Jwcolby54 » 10 Jan 2019, 16:46

BMW,

IF I can get a database set up and functioning, and particularly if I can get it hooked in to the CS such that it can be written to as mods load, then the database and some C++ wrapper around it can serve a lot of purposes. SQLite is written in C++ but is language agnostic, there apparently are already written APIs for many major languages.

Personally, C++ is not a language I have used but I did a stint on a microcontroller long ago that used C and I spent almost 2 years in that project (building a vending machine). I do not anticipate a huge problem learning enough C++ to get started on this.

A database is not an application of course. It is simply a place to store data that is used in an environment. So no it is not a repository in the sense that you think of it. That said, a database is in fact nothing more than a repository, things "repose" or are stored there.

When you look at how mods are currently stored, we have a Data Files directory, inside of which are directories for each mod. Inside of that are directories which hold the pieces - most mods keep it simple with perhaps Meshes and textures etc.

So I could relatively quickly simply build out tables to point top down to the directories

tables:

Mod
>> Meshes
>> Textures
>> Book Art
>> Icons
>> Music
>> Shaders
>> Sound
>> Video

Etc.

Then I could do nothing more that walk the directories pulling each file in each directory in and storing it as a blob in the database. Instant repository. Given the directory names and contents, I can write them back out to a location of your choice and you have that mod on your disk.

However stored as a blob I cannot "see" what is inside of the blob, so I cannot do any collision analysis.

However if I can hook the CS and call a method in my code to create / update / delete records in my database as stuff is loaded, then I know the same things that the CS knows. Or at least my DB stores the same things that the CS is working on. That allows analysis.

If we wanted to get fancy, we could hash the objects and store the hashes with the objects which would allow us to instantly see objects which have changed. If an object has changed, then store a new copy, leave the old. Instant undo, regression etc. All from data right at our fingertips.

Databases allow a lot of functionality that has to be otherwise hard coded. It is certainly possible to write code to scan two mods and compare the pieces. YUK. A database doesn't totally eliminate that but adds a powerful tool for finding similar objects where pieces match, or showing the data about objects. Locations of rings, spells on armor, NPC classes and specific abilities and so forth.

Assuming it is possible to do so, loading the game files into the database would allow us to know every single piece and part of the game. A discussion for another day / month / year!

Jwcolby54
Posts: 21
Joined: 01 Jan 2019, 08:48
Location: NC
Contact:

Re: Addin database

Post by Jwcolby54 » 13 Jan 2019, 04:19

I spent all day trying and failing to get a gpu to pass through to my vm. The things that get in the way of work.

luthis
Posts: 4
Joined: 21 Feb 2018, 19:45

Re: Addin database

Post by luthis » 15 Jan 2019, 21:01

This idea sounds great! I have spent far too long messing around with mods and not actually playing the dam game.

Storing mods as a database also makes migrating installations / reinstalling etc super easy.

Would this eliminate the whole load order : trial and error fiasco?

I have experience with C and C++ and from my perspective, there is a big jump in understanding to go from procedural to object-oriented. Once you understand the concepts though, OO is super delicious.

I think moving forward, taking modding to the next level (like openmw is doing with the engine) is a necessary step. Modding at the moment (and getting mods to work together without breaking things) feels magical and unpredictable. I'm sure it would benefit the community hugely to have a proper mod integration tool, because all Bethesda provided originally was a mod creation tool.

User avatar
bmw
Posts: 23
Joined: 04 Jan 2019, 19:42
Contact:

Re: Addin database

Post by bmw » 19 Jan 2019, 03:02

@luthis,

What Jwcolby54 is proposing, if I understand correctly, is using a database to do more complex analysis of mods en masse, among other things making it easier to detect conflicts between them. Someone who just wants to install mods would not actually need to directly use this database, though the data gained from this analysis could benefit them indirectly through the incorporation of that data into other tools.

(Aside to Jwcolby54: if this is not what you are proposing, I apologize. You’ve said an awful lot of things and I’m somewhat unsure as to the scope of what you’re proposing the database be used for).

If you’re just looking for an easy way of installing mods without conflicts, I suggest you look at OpenMMM v2 (in development, details here), a project that I've been working on which will be able to do exactly that. Among other things, it will be able to do migrations by copying some config files and running a single command and will eventually automatically determine both data and esp load order.
Note that while I say that it will install mods without conflicts, it doesn’t detect them automatically. Instead it provides a way to ensure that conflicts, once reported, are handled automatically for everyone using it.

Jwcolby54
Posts: 21
Joined: 01 Jan 2019, 08:48
Location: NC
Contact:

Re: Addin database

Post by Jwcolby54 » 19 Jan 2019, 15:34

I have been hard at work working hard. Sorry for the lack of response.

I am leaning heavily towards Python 3x, TKInter and SQLLite as a killer combination for implementing whatever it is I implement. I worked in C and C# and while there are aspects of it that I love (over Python) such as the well defined and varied datatypes, I am discovering that Python has a lot to love as well.

As for what I intend...

I have been slogging through the docs for the objects in Morrowind, in the wiki. Sadly it seems that it was kind of "hacked" together. Probably a result of the tools at the time, it is after all close to 20 years old.

The problem that leaves me with is that while the things in the game appear to be objects, which seem to have "primary keys" or unique identifiers, what I am not finding is the "instance" identifier information. IOW a ring has a well defined definition, and I can tell that an object actually out in the world is a ring, there are many many rings in the world. Each instance of a ring should be identified in addition to it's type. A ring can be picked up and moved for instance. Can I tell that a specific ring picked up off a table in Balmora is the same ring now sitting on the bed in the fighter's guild? That matters because a mod may try to modify that specific ring in a game already underway. You get the picture.

At any rate, it appears that I can build out tables to hold all of the objects, with child tables holding attributes of that object and so forth - OO stuff at the database level. But the game has an odd (to me) mixture of or disregard for object classes (the definition of something) and instances of that something.

The thing to know is that a database allows very powerful tools for analysis of "stuff". If the stuff is (are) mods, once I load all of the pieces and parts into tables, I believe I can more easily compare objects (classes) and their properties.

What does that do for us? It allows the Mod artist to compare their mods to already existing mods. It also allows mod users to do the same thing of course. These two use cases would use different interfaces simply because the Mod artist needs access to the actual DATA of the pieces and parts, IOW to see the internals, modify the internals etc. I will likely write an interface for each use case using tkinter or something like it.

The Mod user simply needs to know that two (or more) mods "collide". The truth is that collisions may be gross or very detailed. The user isn't going to understand what the details mean in terms of problems created in the game. Most likely it will truly be a simple flag that tells the user that "something collides" between ModA and ModB, IOW don't use both of the Mods, choose one or the other.

It appears to me, looking in from the outside, to be a case of "the last one loaded wins" but the result could be truly wretched so it is better to punt and just select one. But at least the user KNOWS that there is a collision and can make an informed choice as to what to do. That appears to be totally missing in the game today.

It would also allow the main game to store and load from a db as opposed to from blobs. As to whether that ever happens is pretty much irrelevant to me. It would matter to me if I were on the OpenMW dev team but I am not. As long as I can get the algorithm for parsing the blob (and maybe writing it back out again) I am happy.

So yes, I am proposing an "en masse" analysis. Which starts with an "en masse" load of every mod to be found. Once mods are loaded, then a set of tools needs to be defined and implemented to assist the Mod artist in avoiding collisions, or fixing his mod if it collides with something (and (s)he cares).

I come back to the fact that mods are often things and locations that simply do not collide with other mods. However it would be good to know (for example) physical locations of existing caves if I want to design an underground house mod.

I am going to assume (perhaps incorrectly) that the CS "knows about" the stuff in the main game and assists the Mod artist in avoiding such collisions between his new cave mod and any caves in the game. If that is NOT the case, then the CS could use my database to discover the locations of objects found in the base game model. Assuming that my db contains all the stuff in the base game model. There is no particular reason that it could not.

In the end, trying to analyze stuff in blobs is just ick. Databases exist for a reason. Whether the final game file remains a blob or moves to using a database for storage, being able to get everything into a database should allow real, live, easier analysis of stuff. What the database is used for is up to the folks that want to do stuff with the game and the mods at the dev level.

My intention is to write the database. Get the mods into the database. Generate reports of collisions. If the CS devs want to use the database then I will assist in getting them into the database. Likewise with the OpenMW devs. Once that part is done, I will likely look at joining the CS team if they will have me.

I am sensing a ton of resistance from some folks about using a database, and I totally understand that. I have lived databases for 25 years and I have programmed for 30 years. Both are familiar to me and I understand how databases solve problems but many, perhaps most devs don't. Databases aren't magic, nor particularly difficult to use so I will assist however I can in making them available in OpenMW, and assist the devs in understanding how they can make life easier. But I am not pushing them on anyone.

Post Reply