SQLite based approach to storing game world state

Everything about development and the OpenMW source code.
User avatar
urm
Posts: 83
Joined: 02 Jun 2017, 16:05
Gitlab profile: https://gitlab.com/uramer

Re: SQLite based approach to storing game world state

Post by urm »

darkbasic wrote: 02 Apr 2021, 08:29 Partially true, it depends on the complexity of the query. An ORM makes simple queries easy, but whenever you need anything remotely more complex it makes your life a hell. Doing complex queries with an ORM can be either extremely difficult or plain impossible.
We probably don't want modders to be able to run arbitrary queries on the database anyway. In fact, it's probably best they don't even have to know there is a database involved.
darkbasic wrote: 02 Apr 2021, 08:29 Don't count on it. We've already tried this path and failed many times. When you fail to scale then switching to a different database alone won't save your day: you will have to rewrite your queries to take advantage of its features and an ORM will only make your life more difficult.

Don't get me wrong, I like working with ORMs because they allow for fast prototyping and easy to read code while also lowering the barrier for junior developers, but overall they won't make your life easier. On the contrary they make it harder in the long term because you will waste much more time optimizing your queries/writing complex ones. I'm talking about complex web applications with lots of db needs, probably that won't be the case for openmw.
I don't know what you are working on, but this all depends on how much do you need to scale, and from where do you start. There is plenty of performance gains to be had by replacing sqlite with a "proper" server-based database. And even the latter can scale significantly further with relatively plug-and-play solutions in specific cases.

In any case, I don't see how this discussion applies to OpenMW. The query API is mostly meant for modders, and making modders write highly advanced queries specific to a particular database is just not an option in any scenario. If we need to run raw SQL queries in the engine itself, it will obviously be an option.
I like working with ORMs because they allow for fast prototyping and easy to read code while also lowering the barrier for junior developers
This is our exact use case 95+% of the time.
User avatar
urm
Posts: 83
Joined: 02 Jun 2017, 16:05
Gitlab profile: https://gitlab.com/uramer

Re: SQLite based approach to storing game world state

Post by urm »

Status update:
I've pushed the new version to the MR branch. Main changes are:
1. new query builder implementation (included in the Lua MR as well)
2. fewer limitations for the models (e. g. multiple relations to the same table, multi-column foreign/unique keys)
3. cleaner and less code, cleaner and fewer commits
4. a working example (see test_esmdb.cpp) of writing ESM records (for now just a simple one, ESM::Door) to SQLite and then querying from it.
5. to implement 4 in a reasonable way I've included my test files MR (one small commit) https://gitlab.com/OpenMW/openmw/-/merge_requests/597. It's just a CMake setup to copy a folder recursively to the build directory
Post Reply