After discussion with ptmikheev and David C I came up with the following.
Issues with current OpenMW approach
- The only way to access data about cells which haven't been loaded is to read esp files
- Every cell touched during a session will stay loaded - potential for running out of RAM
- Can't search through the world state quickly
- No straightforward way to serialize data into network packets for multiplayer or external communication (relevant for Lua scripts)
- A lot of code can't be reused between the server and the client
- A significant amount of boilerplate code related to Lua bindings and RakNet packet conversions
- Conversion from and to JSON is not efficient
- Client and server data formats are completely different
- Universal API for reading the current game world state (esp and save data combined)
- Minimal extra dependencies
- Same or better performance
- Identical storage system for single- and multiplayer
- Support large multiplayer servers (at least 50 concurrent players)
Overview
- Cache .esp data into an SQLite database
- Convert save data into an in-memory SQLite database
- Query through them with an ORM query builder
- Provide Lua bindings for the query builder
Everything that's read from .esp files on startup, and is not changed during play
- Stored as an on-disk SQLite database
- Generated on the initial pass through the .esp files (game launch). Can be cached for future launches as long as load order doesn't change
- At runtime only allows reads
- SQLite allows simultaneous reads, so easily scales to multiple threads if necessary
- Can scale even further by creating multiple copies on different physical drives, so won't be a bottleneck in multiplayer regardless of player count
- If proves to be faster than parsing the esp format, can be used for loading cells as well
Changes compared to .esp game world state
- Stored as an in-memory SQLite database
- All changes to the game world are periodically written here
- A save is just a backup of this database
- If the filesize is acceptable (potentially after zip-ing), can replace the current OpenMW save file format
- When used in multiplayer, easy to make regular backups
Quickly changing data
Information that's unreasonable to save into in-memory database whenever it changes, such as actors' locations and some MWScript variables
- Stored in fully loaded models (current OpenMW world state storage)
- If the in-memory DB turns out to be fast enough, this will only have to include data for currently active cells
- Any changes to the game world are done on completely loaded models (the way OpenMW functions now), and are saved to the in-memory DB automatically (at regular intervals, or at cell unloads, ...)
On any query, the data is read in order, first from fully loaded models loaded, then from the in-memory database, then from the on-disk esp database, and merged (loaded models have priority over in-memory db, in-memory has priority over on-disk).
We should probably have a way to specify only querying data about active cells, which means we can skip steps 2 and 3 for performance.
This way we can rely on SQLite to query most of the data efficiently, and only need to implement filtering on loaded models through a complete pass over them.
Dependencies
- sqlite: can likely just use the included API, but optionally there are many libraries (such as SQLite ORM)
- a query builder which works both with C++ data structures of our choice, and with SQL generation
It appears a C++ query builder which perfectly fits our use case doesn't exist. There are many LINQ-like libraries, but they use lambdas, which won't allow string generation in C++. There are also multiple SQL generating libraries, but we would have to wrap them in our own query builder, or maintain a fork to add support for C++ data structures. So we will probably have to write our own query builder with string/enum based syntax. On the bright side, we might not need to implement all the intricacies of a general use library.
Most of the small query builder libraries I've found are under 1000 lines of code, so it isn't actually that much code to write and maintain. Although in our case I expect it will be roughly double that - we want to support both SQL and C++ data structures.
For every model (actor, cell, ingredient, cell object, ...) we will need a way to map its fields into SQLite types, and define relations between some of them (cell objects and cells, for instance).
This serialization into a collection primitive types might also be used for network packets used by TES3MP, which will make the synchronization code simpler and cleaner.
Justification
Lua Mod API
It should be straightforward to provide a Lua binding for the query builder. This will automatically give us many desireable features.
For a basic example, the scripting API will certainly need a way to get all the currently active players. Instead of implementing a separate function just for that, we should be able to allow something like this:
Code: Select all
players=world.Actors.Where("type", Actor.TYPE.PLAYER).Where("active", true).All()
More involved examples, which use most of the features of a query builder:
- A custom inventory UI, which groups/sorts/flters/searches for items in specific ways
- A multiplayer world building script (a commonly requested feature), which provides a catalogue of all items from the current load order, with search/filtering/etc.
- An automated armor and weapon balancer (already exists for MWSE) which searches through them on every load order change, and overrides them only in the current save
- Mods which operate on unloaded cells, such as enemies following players through doors or NPC scheduling
Dehardcoding
One of the post 1.0 goals is to transfer as many Morrowind-specific game mechanics into Lua scripts as possible. Having a powerful data query API will definitely make it easier. Some examples:
- alchemy/enchanting will want to find relevant items in player's inventory
- finding all autocalc spells for a given NPC
- loading dialogue topics
Queries through .esp data immediately give us most of the functionality of tes3cmd (a commonly used tool for automated .esp generation). It would be easy to add this to OpenMW-CS after the changes above are introduced.
We could also re-implement the existing search/filters in the editor to reduce the amount of code to maintain.
Custom records
One of the major features of TES3MP are custom records - a way to create new records (NPCs, weapons, cells...) at runtime, on the server side. Storing save games and world files in the same way will allow to override or add extra records only for the current save game, rather than load order, which makes it much easier to implement custom records for OpenMW (to be used both for multiplayer and singleplayer mods).
Some runtime-created records (particularly user-brewed potions) are too numerous to store them indefinitely. Currently tes3mp stores many-to-many relations between them and containers (players and cells) in a map on each container and record. With the database storage and advanced querying it will be trivial to find all unused records.
OpenMW as a general engine, and supporting other games
This approach would allows us to rapidly implement new game mechanics, and to provide flexible Lua access to any new object types necessary.