Addin database

Join the team. This area is for people who want to participate in OpenMW's development in one way or another.
silentthief
Posts: 322
Joined: 18 Apr 2013, 01:20
Location: Currently traversing the Ascadian Isles

Re: Addin database

Post by silentthief » 08 Jan 2019, 14:52

The topic of mod compatibility and being able to better determine mod errors has been a common topic lately. Perhaps there is a way that we can offer a way to make mod conflicts easier to find? (That was my motivation behind my post here -> viewtopic.php?f=3&t=5671) Or perhaps a database would help with conversation topics? I don't know.

The idea of a morrowind db (MWSQL if you like) may be useful, likewise a repository and mod conflict list would be helpful as well. I am not a dev, but I do know that the devs that are working on the project are hesitant to put in feature creep adding new features especially before 1.0

ST
"Hurry, hurry! Last boat to Solstheim! Until the next one. Hah-ha-hah."

User avatar
psi29a
Posts: 4420
Joined: 29 Sep 2011, 10:13
Github profile: https://github.com/psi29a/
Contact:

Re: Addin database

Post by psi29a » 08 Jan 2019, 20:05

I think that this out of scope of OpenMW and more a general Bethesda modding issue. If someone would make something we could call against (via API) or as a library/plugin...

Don't forget that python is still on the table for openmw-cs. :)

Jwcolby54
Posts: 16
Joined: 01 Jan 2019, 08:48
Location: NC
Contact:

Re: Addin database

Post by Jwcolby54 » 09 Jan 2019, 17:37

BMW, I just read the thread you pointed me to. My thoughts in no particular order :

1) Someone pointed me to SQLite which is an obvious choice for this task. While it has certain issues for a "big data guy" (me) in particular the dearth of defined data types, that doesn't seem to stop folks from using it. And in most regards it seems to be a stellar choice.

2) While I want to learn python, OMW and MWC are written in C. With an eye toward possible integration it would probably be good to use C.

3) By loading the mod data into a database, it becomes possible to "discover what's in there".

4) I think a mod "repository" is a great idea, but a mod database is not that. In fact afaict a database and app around it could deconstruct all the pieces of a mod and store the pieces right in the database. Thus the app could open the cfg file, open the openmw directory, build out the directory structure, drop the pieces from the database into the directories, and update the cfg file. That is NOT what I was wanting to do but it could be done.

5) Immediately what I want to do is to break into the CS and where the code loads stuff from the mod into the CS, ALSO load it into my database. IOW I REALLY don't want to write a new parser to do that!

6) immediately what I want to do is where the CS saves back out to the file, ALSO store it back out to my database. IOW I REALLY don't want to write a new writer to do that.

7) once I have a database full of tables that can store any object from the game, and I am tied in to the CS in this manner, simply by loading a mod into the CS I have it in my database.

Sounds simple! 😉🤔😕

What specifically is done with it from there is up for discussion.

Listening to the creator of SQLITE, he talks about apps using (his words) "piles of files" as the data store for applications. IOW directories of directories of files. Sound familiar?

He discusses replacing all that with a single file which is a SQLite database. Which makes a lot of sense to me. But I have to come up to speed on it, get the table infrastructure designed, and a method to load the database up with mods, and (in the absense of openmw using the dB directly) a way to spit it back out into a "piles of files" format.

Which has little to do with mod analysis. 😁

Jwcolby54
Posts: 16
Joined: 01 Jan 2019, 08:48
Location: NC
Contact:

Re: Addin database

Post by Jwcolby54 » 10 Jan 2019, 02:07

psi29a wrote:
08 Jan 2019, 20:05
I think that this out of scope of OpenMW and more a general Bethesda modding issue. If someone would make something we could call against (via API) or as a library/plugin...

Don't forget that python is still on the table for openmw-cs. :)
Agreed outside the scope of the game engine. My concept of a database to hold and analyze game mods dances tightly around the cs however. I am studying SQLite atm.

My background is programming clear back to the early 80s and databases since the early 90s. I spent about 2 years learning C# and using it to heavily automate SQL server. I haven't programmed in C for several years though so I have to get a dev environment set up and then back up to speed. I actually have a vm on a server "ready to go" but it uses VS 2015 and C#, and I think I'd rather look at something like vs code and C++ for portability.

Anyway lots to do, just getting started.

User avatar
bmw
Posts: 7
Joined: 04 Jan 2019, 19:42
Github profile: https://gitlab.com/bmwinger
Contact:

Re: Addin database

Post by bmw » 10 Jan 2019, 02:57

Jwcolby54 wrote:
09 Jan 2019, 17:37
4) I think a mod "repository" is a great idea, but a mod database is not that. In fact afaict a database and app around it could deconstruct all the pieces of a mod and store the pieces right in the database. Thus the app could open the cfg file, open the openmw directory, build out the directory structure, drop the pieces from the database into the directories, and update the cfg file. That is NOT what I was wanting to do but it could be done.
On reflection I realise that the meaning of my earlier post may not have been entirely clear. While the mod database is certainly distinct from a mod repository, largely I wanted to point out that some of the benefits you had said you were hoping for would also be satisfied by a mod repository, particularly the below:
Jwcolby54 wrote:
07 Jan 2019, 23:27
A means for Users to check mods they were interested in, perhaps read descriptions of what they do.
A means for users to select and then check collisions between the mods they are interested in.
I also wanted to point out how useful this would be for the mod repository idea, as it could massively help with generating lists of conflicting mods. The database could also probably be set up to auto populate from the mods in the repository, so the two projects could be mutually beneficial.

It occurs to me that the solution to the legal issues I mentioned before could be to only expose a conflict checking API for the database and keep most of the details of the mods in the database hidden, then you're not redistributing the data.
silentthief wrote:
08 Jan 2019, 14:52
The idea of a morrowind db (MWSQL if you like) may be useful, likewise a repository and mod conflict list would be helpful as well. I am not a dev, but I do know that the devs that are working on the project are hesitant to put in feature creep adding new features especially before 1.0
The mod repository is certainly outside the scope of the game engine and isn't really feature creep in that respect. I'm also perfectly happy to do it myself, or at least most of the infrastructure, and already have a mostly functional prototype version. Adding mods to the repository (which I plan to automate as much as possible) and making sure their build files are correct will certainly benefit from the help of others though.

Jwcolby54
Posts: 16
Joined: 01 Jan 2019, 08:48
Location: NC
Contact:

Re: Addin database

Post by Jwcolby54 » 10 Jan 2019, 16:46

BMW,

IF I can get a database set up and functioning, and particularly if I can get it hooked in to the CS such that it can be written to as mods load, then the database and some C++ wrapper around it can serve a lot of purposes. SQLite is written in C++ but is language agnostic, there apparently are already written APIs for many major languages.

Personally, C++ is not a language I have used but I did a stint on a microcontroller long ago that used C and I spent almost 2 years in that project (building a vending machine). I do not anticipate a huge problem learning enough C++ to get started on this.

A database is not an application of course. It is simply a place to store data that is used in an environment. So no it is not a repository in the sense that you think of it. That said, a database is in fact nothing more than a repository, things "repose" or are stored there.

When you look at how mods are currently stored, we have a Data Files directory, inside of which are directories for each mod. Inside of that are directories which hold the pieces - most mods keep it simple with perhaps Meshes and textures etc.

So I could relatively quickly simply build out tables to point top down to the directories

tables:

Mod
>> Meshes
>> Textures
>> Book Art
>> Icons
>> Music
>> Shaders
>> Sound
>> Video

Etc.

Then I could do nothing more that walk the directories pulling each file in each directory in and storing it as a blob in the database. Instant repository. Given the directory names and contents, I can write them back out to a location of your choice and you have that mod on your disk.

However stored as a blob I cannot "see" what is inside of the blob, so I cannot do any collision analysis.

However if I can hook the CS and call a method in my code to create / update / delete records in my database as stuff is loaded, then I know the same things that the CS knows. Or at least my DB stores the same things that the CS is working on. That allows analysis.

If we wanted to get fancy, we could hash the objects and store the hashes with the objects which would allow us to instantly see objects which have changed. If an object has changed, then store a new copy, leave the old. Instant undo, regression etc. All from data right at our fingertips.

Databases allow a lot of functionality that has to be otherwise hard coded. It is certainly possible to write code to scan two mods and compare the pieces. YUK. A database doesn't totally eliminate that but adds a powerful tool for finding similar objects where pieces match, or showing the data about objects. Locations of rings, spells on armor, NPC classes and specific abilities and so forth.

Assuming it is possible to do so, loading the game files into the database would allow us to know every single piece and part of the game. A discussion for another day / month / year!

Jwcolby54
Posts: 16
Joined: 01 Jan 2019, 08:48
Location: NC
Contact:

Re: Addin database

Post by Jwcolby54 » 13 Jan 2019, 04:19

I spent all day trying and failing to get a gpu to pass through to my vm. The things that get in the way of work.

luthis
Posts: 4
Joined: 21 Feb 2018, 19:45

Re: Addin database

Post by luthis » 15 Jan 2019, 21:01

This idea sounds great! I have spent far too long messing around with mods and not actually playing the dam game.

Storing mods as a database also makes migrating installations / reinstalling etc super easy.

Would this eliminate the whole load order : trial and error fiasco?

I have experience with C and C++ and from my perspective, there is a big jump in understanding to go from procedural to object-oriented. Once you understand the concepts though, OO is super delicious.

I think moving forward, taking modding to the next level (like openmw is doing with the engine) is a necessary step. Modding at the moment (and getting mods to work together without breaking things) feels magical and unpredictable. I'm sure it would benefit the community hugely to have a proper mod integration tool, because all Bethesda provided originally was a mod creation tool.

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest