4g. Data storage

Involved development of the OpenMW construction set.
Locked
User avatar
lgromanowski
Site Admin
Posts: 1193
Joined: 05 Aug 2011, 22:21
Location: Wroclaw, Poland
Contact:

4g. Data storage

Post by lgromanowski »

Zini wrote: We can reuse components/esm for recording loading. Unfortunately there is no equivalent code for saving. Maybe we should integrate it into components/esm.
Maybe we can also reuse the ESM store class in components/esm_store. But we won't be able to re-use the cell store. The current cell store class in components/esm_store will receive some more modifications, that will make it unsuitable for the editor.

Here is what we must do. We create two record storages. Storage A contains all esms and all esps except for the one that is edited. We can merge them all into a single data structure (we still may want to keep track of by which esx file the record was changed last; may come in handy for another filter option).
The records from the edited plugin go into storage B and storage B is the only data structure the editor is allowed to edit.

Special attention must be paid to record deletion. Deleting a record that exists in storage A actually means adding a record to storage B, which is flagged as deleted. On the other hand a record, that exists only in storage B, must be deleted conventionally from storage B.
Zini wrote:
pvdk wrote:First of all I would like to know how we are going to store the data for editing. I suppose we have to read the esm and esp data and put it in a separate storage. Should the data go in a XML file (size?!) or in a SQLite database or something else?

Second, I would like to suggest the usage of project files. That way we can combine user settings like custom component views and application settings with esm and esp files to edit. Think of MS Visual Studio solutions or Photoshop .psd's. Saving the plugins would be called Export. Obviously we would have to decide upon the project format and extension but I guess a .zip file containing .xmls would work. Added benefit of using projects is that overwriting existing plugins resulting in compatibility issues is much harder if you have to manually export your results.

An usage example of this would be:
Total conversions using their own project file which holds a severely customized layout with custom item view components for different data files.
Zini wrote:
First of all I would like to know how we are going to store the data for editing. I suppose we have to read the esm and esp data and put it in a separate storage. Should the data go in a XML file (size?!) or in a SQLite database or something else?
We already have the record structs in components/esm. We should reuse these. The data model suggests a flat data structure. A basic STL container should do it.
Second, I would like to suggest the usage of project files. That way we can combine user settings like custom component views and application settings with esm and esp files to edit. Think of MS Visual Studio solutions or Photoshop .psd's. Saving the plugins would be called Export. Obviously we would have to decide upon the project format and extension but I guess a .zip file containing .xmls would work. Added benefit of using projects is that overwriting existing plugins resulting in compatibility issues is much harder if you have to manually export your results.
Not in favour of this solution. Having a separate export stage would be burden for the content developer, since the usual workflow is: make a small change, run plugin for testing, make another small change, run plugin for testing. Also, having a second file format for storage would be a complete waste of developer time.

Let's use the regular ESP files instead. But the editor can use project files internally for additional project-related settings that are matched via ESP filename/profile. We can even do that transparently. The user should not be required to bother with project files (or even know about them), but if he wishes he should be able to make use of them explicitly.
pvdk wrote: We do have the esm reader, but we have to put the data somewhere it can be read and edited with Qt. A STL container will not work. Here's some documentation on what containers Qt provides (they are mostly platform-independent STL equivalents):
http://doc.qt.nokia.com/4.7/containers.html

I'm not entirely sure how we can provide a model for our data using containers. Most likely we will have to subclass QAbstractListModel/QAbstractItemModel.
Zini wrote: We can't use the stored data as a model directly. As I wrote in the first post we need two data storages that must be combined at runtime. I assumed you could write some model-adapter, that can integrate a qt-view with a custom model class.
pvdk wrote:
Zini wrote:We can't use the stored data as a model directly. As I wrote in the first post we need two data storages that must be combined at runtime. I assumed you could write some model-adapter, that can integrate a qt-view with a custom model class.
We also can't use the stored data as a model because that's not how Qt works. You can't assign storage containers to views.
I assumed that our QAbstractItemModel subclass could be used for both the reference as the edit model. One model gets filled with the data read from the .esm and .esp files as the reference model (and set to read-only) and the other one gets filled with the modifications to this. Right?
Zini wrote:
One model gets filled with the data read from the .esm and .esp files as the reference model (and set to read-only) and the other one gets filled with the modifications to this. Right?
Mostly. Storage A is not really a reference. When accessing a record, it should work like this:

- Look it up in storage B. If it is there, use it.
- Else, find it in storage A

The actual record list would be composed of the union of records in A and in B, minus those records from A that are also in B (only the copy in B is kept).

Modifications only go to storage B. By removing a record from storage B you are either deleting it (if is is not in storage A) or reset it to the state of storage A.
Locked