SQLite based approach to storing game world state

Everything about development and the OpenMW source code.
User avatar
urm
Posts: 83
Joined: 02 Jun 2017, 16:05
Gitlab profile: https://gitlab.com/uramer

Re: SQLite based approach to storing game world state

Post by urm »

akortunov wrote: 10 Nov 2020, 12:34 It depends on what you claim to be a "complete data". If you mean low-level data (including ESM records for visited cells), it is already stored in RAM (because it is valid only for current game session, and it is just no other places to store it).
If you mean all high-level objects from apps/openmw, they are present only for loaded cells by design (because they consume too many resources to have them for everything), and they nothing common to low-level storages, such as SQL database.
I really don't see what you mean by this.
Are you saying it's a good thing we don't unload visited cells, and consume more and more memory as the play session continues?
Are you saying we should never allow access to data about non-visited cells at all? MWSE, and even custom scripts for tes3mp do allow it, and not having that option would severly limit potential mod functionality.
User avatar
akortunov
Posts: 899
Joined: 13 Mar 2017, 13:49
Location: Samara, Russian Federation

Re: SQLite based approach to storing game world state

Post by akortunov »

urm wrote: 10 Nov 2020, 12:43 I really don't see what you mean by this.
I just ask what you mean by "complete data for unloaded cells", which can not be loaded to RAM, and which have too slow access for them. Once you clarify, we can continue this discussion.
User avatar
urm
Posts: 83
Joined: 02 Jun 2017, 16:05
Gitlab profile: https://gitlab.com/uramer

Re: SQLite based approach to storing game world state

Post by urm »

akortunov wrote: 10 Nov 2020, 12:34 It is nice if you can guarantee that your ESM data is really immutable (or at least server and only server can modify it in the single thread). For example, there are many mods which adjust base records during game start (you referred some of them) and during some game events (e.g. when you create a new item with arbitrary properties on the fly). I suppose that all such scripts are supposed to be server-side-only ones, right?
ESM data would be immutable. Any changes to records would be saved into the save game database (analogous to the way it happens currently). They don't necessarily need to be server-side only, all the record changes will be synchronized anyway.
Tes3mp already has this logic covered, and I don't see any obstructions to implementing it with SQL storage.
I just ask what you mean by "complete data for unloaded cells", which can not be loaded to RAM. Once you clarify, we can continue this discussion.
AFAIK we only load the actual list of cell objects and actors when the cell is first visited. Correct me if I'm wrong there.
User avatar
akortunov
Posts: 899
Joined: 13 Mar 2017, 13:49
Location: Samara, Russian Federation

Re: SQLite based approach to storing game world state

Post by akortunov »

urm wrote: 10 Nov 2020, 13:18 AFAIK we only load the actual list of cell objects and actors when the cell is first visited. Correct me if I'm wrong there.
OpenMW uses at least these data when cell becomes active:
1. Base ESM records, which are common for all objects with the same ID
2. CellRef data, which represents an initial state of object in cell (transformations, owner, charges count, etc).
3. LiveCellRef data, which represents current object state (CreatureStats, ContainerStore with resolved items, etc) in visited cell. If you drop such data, it is a respawn.
Most of cell data in saves is actors skills, attributes and inventories.
4. Additional state, which is used only for active cells (e.g. OSG scene nodes, character controllers, Bullet shapes) - occupy most of resources, so we do not create them for every visited cell.

As I understood, you want to store first two types of data in the one database (on disk) and the third type in the another database (in RAM). Backups of second database are treated as savegame files.

In this case you do not really need to store first database on disk - this data generally does not occupy too many space - a significant part of ESM/ESP files are landscape heights/colors, dialogues, etc. I doubt that you really need to copy all this stuff to the SQL database and then validate this database on every startup.

Also there are cases when user changed his mods setup, so his save files become obsoleted, and there is an additional logic to make them usable again (clean up removed cells, delete instances of non-exisent now objects, assign default values instead of missing classes, etc). With such cases a simple "drop existing database and take an another one from backup" approach more likely will not work properly, and with complex validation it will be simpler just to use existing save format with existing and working validation, and then copy loaded data to the in-memory database, if it present.

And most of singleplayer and coop setups will not need data from unloaded cells anyway, even with Lua mods:
1. Base ESM records can be handled en masse once during startup and when the game creates a new dynamic ESM record on the fly.
In theory, client needs to get a large amount of changes once when he is logged in.
2. Modmaker can handle objects in cell once it becomes active instead of altering objects in whole game world. For example, if you want to add torches to all actors, you can add torches for all actors in given cell once it become active. For non-active cells players will not see the difference anyway, so modmaker does not need to query all actor instances in game world.
Data from non-active loaded cells can be useful in singleplayer to create some kind of "offline" life simulation (when different events happen even in unloaded cells), but it is not a performance-critical feature.

IMO, it would be better to use SQL storage as an optional indexing mechanism (mostly on servers with a lot of players), but on such servers an amount of active cells actually may be larger issue than data format (after all, OpenMW is not an MMO engine). In singleplayer it is just a redundant entity and additinal source of bugs and slowdowns (you will need spend additional resources to keep databases and sync them with the game itself). Probably it is worth even to add a compile flag to do not build SQL-related code at all.
User avatar
AnyOldName3
Posts: 2666
Joined: 26 Nov 2015, 03:25

Re: SQLite based approach to storing game world state

Post by AnyOldName3 »

While we've got distant statics, there are at least a few things a modder might want to meddle with outside the active cells, even if it's just basic stuff like adding and removing large objects.
User avatar
urm
Posts: 83
Joined: 02 Jun 2017, 16:05
Gitlab profile: https://gitlab.com/uramer

Re: SQLite based approach to storing game world state

Post by urm »

akortunov wrote: 12 Nov 2020, 11:53 OpenMW uses at least these data when cell becomes active:
1. Base ESM records, which are common for all objects with the same ID
2. CellRef data, which represents an initial state of object in cell (transformations, owner, charges count, etc).
3. LiveCellRef data, which represents current object state (CreatureStats, ContainerStore with resolved items, etc) in visited cell. If you drop such data, it is a respawn.
Most of cell data in saves is actors skills, attributes and inventories.
4. Additional state, which is used only for active cells (e.g. OSG scene nodes, character controllers, Bullet shapes) - occupy most of resources, so we do not create them for every visited cell.

As I understood, you want to store first two types of data in the one database (on disk) and the third type in the another database (in RAM). Backups of second database are treated as savegame files.

In this case you do not really need to store first database on disk - this data generally does not occupy too many space - a significant part of ESM/ESP files are landscape heights/colors, dialogues, etc. I doubt that you really need to copy all this stuff to the SQL database and then validate this database on every startup.
I agree that for now we could store that in RAM (as we already do), although I wonder if doing a complete pass through that storage is faster than querying it even from an on-disk database. Also, reading it entirely from RAM is more sensitive to thread-locks, and so far it's looking like the Lua script support will be heavily threaded.

In my opinion, long term we should not store all of the records in RAM. It might limit some mod/game designs, such as procedural generation of records.
Also there are cases when user changed his mods setup, so his save files become obsoleted, and there is an additional logic to make them usable again (clean up removed cells, delete instances of non-exisent now objects, assign default values instead of missing classes, etc). With such cases a simple "drop existing database and take an another one from backup" approach more likely will not work properly, and with complex validation it will be simpler just to use existing save format with existing and working validation, and then copy loaded data to the in-memory database, if it present.
It was implied in the post, and I've written it in Discord, but in case it isn't clear: all of the suggestions of replacing existing features with this new approach are not necessary to perform immediately. I've mentioned them to show that we can broadly reuse these features to reduce code duplication, but we would only go for it when we are certain this works well.
And most of singleplayer and coop setups will not need data from unloaded cells anyway, even with Lua mods:
1. Base ESM records can be handled en masse once during startup and when the game creates a new dynamic ESM record on the fly.
In theory, client needs to get a large amount of changes once when he is logged in.
2. Modmaker can handle objects in cell once it becomes active instead of altering objects in whole game world. For example, if you want to add torches to all actors, you can add torches for all actors in given cell once it become active. For non-active cells players will not see the difference anyway, so modmaker does not need to query all actor instances in game world.
Data from non-active loaded cells can be useful in singleplayer to create some kind of "offline" life simulation (when different events happen even in unloaded cells), but it is not a performance-critical feature.
This ignores some of the examples of mods I gave.
IMO, it would be better to use SQL storage as an optional indexing mechanism (mostly on servers with a lot of players), but on such servers an amount of active cells actually may be larger issue than data format (after all, OpenMW is not an MMO engine). In singleplayer it is just a redundant entity and additinal source of bugs and slowdowns (you will need spend additional resources to keep databases and sync them with the game itself). Probably it is worth even to add a compile flag to do not build SQL-related code at all.
The main goal of this whole discussion is to come up with a solution which works well both for singleplayer and multiplayer, so that we can merge multiplayer in without increasing the amount of code to maintain dramatically. Having two different ways to store world state will just be a source of bugs and extra work on its own.

In my opinion, going through a transition period and eventually switching to this approach, when it is stable, is a better way to approach this long term.

A minimal build of SQLite is under a megabyte, I don't see how it's worth bothering with not building it, even if we would make it an optional feature.
User avatar
urm
Posts: 83
Joined: 02 Jun 2017, 16:05
Gitlab profile: https://gitlab.com/uramer

Re: SQLite based approach to storing game world state

Post by urm »

Seems like nobody has anything else to add to the topic. If that's the case, I will start prototyping this soon.
User avatar
cc9cii
Posts: 523
Joined: 28 Mar 2013, 04:01

Re: SQLite based approach to storing game world state

Post by cc9cii »

I have only skim-read the previous posts so I probably missed it but I don't quite understand the proposed approach. In particular I don't understand why it is an issue that only the loaded cell data are kept in RAM (* not quite true, see below) and discarded from RAM when the cell is unloaded. An example of what could be done with this data might be useful. Or conversely, what we might be missing out by not having *all* the data.

(*) some data are kept for much longer, if not forever - for example the physics handles or generated materials (at least in the old Ogre/Bullet implementation)
User avatar
urm
Posts: 83
Joined: 02 Jun 2017, 16:05
Gitlab profile: https://gitlab.com/uramer

Re: SQLite based approach to storing game world state

Post by urm »

cc9cii wrote: 16 Nov 2020, 22:03 I have only skim-read the previous posts so I probably missed it but I don't quite understand the proposed approach. In particular I don't understand why it is an issue that only the loaded cell data are kept in RAM (* not quite true, see below) and discarded from RAM when the cell is unloaded. An example of what could be done with this data might be useful. Or conversely, what we might be missing out by not having *all* the data.

(*) some data are kept for much longer, if not forever - for example the physics handles or generated materials (at least in the old Ogre/Bullet implementation)
I have a few mod examples which would use unloaded cell data in the justification section of the OP. Point 4 also applies to your question about global scripts in the Lua thread btw.

Keeping unloaded cells in RAM is already an issue, formally speaking, it's just people don't play for long enough on weak enough hardware to run into it. At the very least back in 0.45 my PC (16Gb RAM) could only handle visits to about 1/3 of Morrowind.esm interiors, after that the game would just crash, and slow down long before that. So even in singleplayer, a very long play session (let's say a 24h marathon focused on visiting all the big expansion mods) on a weak hardware (say, an android phone, which OpenMW now supports) would be very likely to run into issues.

This whole issue gets far worse in multiplayer, if we switch to reusing as much of the client code as possible (which is a good idea - it reduces the amount of code we need to maintain, especially since many OpenMW devs are not particularly interested in maintaining multiplayer). There have already been servers, which had 20+ players pretty much at all times for months, and were expected to have 24/7 uptime. If we don't fully unload cells, this will just not work period.
User avatar
cc9cii
Posts: 523
Joined: 28 Mar 2013, 04:01

Re: SQLite based approach to storing game world state

Post by cc9cii »

Ok, I skipped those parts :)

I still suspect you'll have some challenge to get this accepted until you can demonstrate a *killer* use-case. But I think you should still go ahead and implement it regardless. You'll have fun doing it and will learn more about OpenMW and game engines in general. I doubt I can be of much help other than some knowledge of ESM/ESP data, especially with later games (but they're already available elsewhere e.g. UESP).

EDIT: just to explain my interested in Lua scripting (by the way, I am not keen on Lua at all, having had some poor introduction to it on angband) - you may be aware that I'm trying to get later games (Oblivion, FO3, etc) to work. But I have little knowledge / experience in many subject areas, one of which is MODs and scripts. So I'm trying to understand more about them so that I make less fundamental errors that will lead to major re-writes.
Post Reply