4. Framework

Involved development of the OpenMW construction set.
Locked
User avatar
lgromanowski
Site Admin
Posts: 1193
Joined: 05 Aug 2011, 22:21
Location: Wroclaw, Poland
Contact:

4. Framework

Post by lgromanowski »

Zini wrote: 4a. General framwwork considerations

There are several topics to consider when designing the framework. We need to address application-wide functions (like loading and saving) and we need to address how components are integrated into the framework.

Before we go into designing components, we need to get the framework right. It might be a good idea to make a dummy-component for testing component-handling in the framework.

An important point is responsiveness. The only way to achieve this goal is to offload non-instant tasks into separate threads. But we should try to minimize locking, because else we might end up with the opposite effect.

We should also investigate, if it is possible to put all of OGRE into a separate thread. On the OGRE side, that should not be a problem. But the OGRE integration into a Qt-widget might pose some obstacles.

Only once all of the framework is working properly, we should start to develop the individual components (starting with the easier ones).


4b. Components

Definition: A component is a view (including controls) that handles a certain aspect of the ESM/ESP data.

We should create a component base class or even a flat hierarchy of component base classes.

On the framework side we need at least the ability to:
- move components between container windows
- open top-level views (like ID-list, Cell-list and such)

Each component should be closable and clone-able (new view). Also we might consider the ability to lock components programatically for editing.
Here is an example how this last feature could be used: If we offload saving into a separate thread, we can do this lock-free, if we disable editing during the save. The user could still view the content of the plugin while the save is progressing and we would avoid the GUI locking up completely.
We may move the lock-flag to the model instead of the view, because if one view for a record is locked, all views for this record will be locked.

There are different types of components, that we might want to represent with a separate base class each.

I. Top-Level Record Lists

A few examples would be the ID list and the cell list.

Please note, that these components still might be used to change records other than their own, e.g. the Region component will change region records and cell records.

II. Non-List Top-Level

A component, that does not directly corresponds to a single record type. An example would be the 3D editing component.

III. Single Record, identified by a string ID.

Examples: A class, a weapon or a script.

In the TES-CS these are handled via dialogue windows. We should avoid making the same mistake and turn them into full-blown components instead.

IV. Single Record, identified by a numeric ID.

Basically the same as above. Example: Magic effect.


4c. Filtering

We already discussed filtering for ID lists. But we will want this feature for other record types too. Probably for all record types (and especially for NPC dialogues). Maybe we can create some re-usable classes.
Its important here, that these filters must exist persistently across multiple editing sessions. But we can't store them in an esp. Maybe OpenCS can store them internally and match them to the esp file name. But we also should be able to use them across multiple esp files. This needs more thought and discussion.


4d. Undo-Buffer

This should be fairly obvious. But it still is a feature we need to get right, before we can get to implementing actual editing functions.


4e. Search & Replace

This is actually a pretty complex topic. The TES-CS has a search & replace function. We should see, if we can make it even more powerful. But that is worth a separate discussion thread.


4f. Load and Save

That is a more complex topic. Well, "save" is easy. "Save as" too. The esp file must be saved in the root of the data directory structure. We don't even need to offer a regular "save as" dialogue. This function degenerates to a simple rename.

Later (post 1.0) we can think about some more advanced resources file management (included resources file usage tracking and an option to separate resources trees for TCs). But for now the implementation described above is sufficient.

The problem is the load part. Obviously what the TES-CS does it totally wrong. Its one of the most confusing dialogues the whole CS has to offer. We must do better. I think we should reuse the profile system from the launcher. But that needs more thought and maybe a separate discussion thread too.


4g. Data storage

We can reuse components/esm for recording loading. Unfortunately there is no equivalent code for saving. Maybe we should integrate it into components/esm.
Maybe we can also reuse the ESM store class in components/esm_store. But we won't be able to re-use the cell store. The current cell store class in components/esm_store will receive some more modifications, that will make it unsuitable for the editor.

Here is what we must do. We create two record storages. Storage A contains all esms and all esps except for the one that is edited. We can merge them all into a single data structure (we still may want to keep track of by which esx file the record was changed last; may come in handy for another filter option).
The records from the edited plugin go into storage B and storage B is the only data structure the editor is allowed to edit.

Special attention must be paid to record deletion. Deleting a record that exists in storage A actually means adding a record to storage B, which is flagged as deleted. On the other hand a record, that exists only in storage B, must be deleted conventionally from storage B.


4h. Drag & Drop

Drag & Drop of records. All records should be draggable. Each component should specify, which record type it can receive and if it can receive multiple records at once.
The result of a drop depends on the component. The 3D editor will create a new reference of the dragged ID. A container or inventory will add the dragged record to itslf. The script editor will insert the name of the record into the script.
We shuold also see, if we can unify this with copy & paste. No idea what Qt offers here.


4i. Other functions

The original CS allows making new esms by merging an esp into an esm. I think we can implement this function later (after the components are done), since it is an advanced functionality. We need it for OpenCS 1.0 though.

4j. Compatibility

The plugins created by our editor will not be compatible with vanilla MW. MW requires scripts to be pre-compiled. But we compile to a completely different virtual machine.

This however should work:
- Save plugin with OpenCS
- Load plugin with TES-CS
- Hit "Compile all"
- Save plugin
- Play plugin in vanilla MW

It would be great if someone could run this testing procedure repeatedly during the entire course of the OpenCS development.

Edit: fixed 4f, wrote source instead of resources in one place.
Hircine wrote: 4d Undo Buffer

Should be something like what Inkscape Offers. a complete history of what you have done and selecting an object at a particular point will show everything that was on the drawing board at that point and not after.

so if you do 20 steps and don't like it, you can compare what you have done with what was 20 steps ago and revert back to anypoint during those 20 steps.
<!-- ia0 -->historylist.png<!-- ia0 -->
hth.
Zini wrote: Makes sense. Especially with the complexity of the CS.
Hircine wrote: the thing i like most about it, is that you will be able to tell if you have accidentally changed something. :)

should we have a thread for each component and discuss how each should work and look like?
Zini wrote: Please don't call them components. The terminology for OpenMW is already confusing. We should try to keep it orderly at least for the CS. Let's call them framework elements. And yes, we should have a discussion thread for each of them.

For the actual components, we should have discussion threads too. Probably one per group of closely related components. But we will address these later, once the framework is ready.
pvdk wrote: First of all I would like to know how we are going to store the data for editing. I suppose we have to read the esm and esp data and put it in a separate storage. Should the data go in a XML file (size?!) or in a SQLite database or something else?

Second, I would like to suggest the usage of project files. That way we can combine user settings like custom component views and application settings with esm and esp files to edit. Think of MS Visual Studio solutions or Photoshop .psd's. Saving the plugins would be called Export. Obviously we would have to decide upon the project format and extension but I guess a .zip file containing .xmls would work. Added benefit of using projects is that overwriting existing plugins resulting in compatibility issues is much harder if you have to manually export your results.

An usage example of this would be:
Total conversions using their own project file which holds a severely customized layout with custom item view components for different data files.

About the undo feature: Qt offers a framework for that, this is how it looks:

Image

To be continued, lots of things to comment on.
Ruarl wrote: Just my two cents: If the ID lists are displayed in a spread sheet format please add a Sort function similar to MS Excel. It should be simple to implement and makes life a bit easier. I hated it that the TES-CS only allowed you to sort the lists according to a single column, not helpful at all.

Rock on

Ruarl
pvdk wrote:
Ruarl wrote:Just my two cents: If the ID lists are displayed in a spread sheet format please add a Sort function similar to MS Excel.
This is something that's high on our list, see this topic's first post, 4c. Filtering.
Edit: I was too quick with my reply. I looked up Excel's sorting function and it looks nice, although it could be implemented in a more user-friendly way.
Zini wrote:
First of all I would like to know how we are going to store the data for editing. I suppose we have to read the esm and esp data and put it in a separate storage. Should the data go in a XML file (size?!) or in a SQLite database or something else?
We already have the record structs in components/esm. We should reuse these. The data model suggests a flat data structure. A basic STL container should do it.
Second, I would like to suggest the usage of project files. That way we can combine user settings like custom component views and application settings with esm and esp files to edit. Think of MS Visual Studio solutions or Photoshop .psd's. Saving the plugins would be called Export. Obviously we would have to decide upon the project format and extension but I guess a .zip file containing .xmls would work. Added benefit of using projects is that overwriting existing plugins resulting in compatibility issues is much harder if you have to manually export your results.
Not in favour of this solution. Having a separate export stage would be burden for the content developer, since the usual workflow is: make a small change, run plugin for testing, make another small change, run plugin for testing. Also, having a second file format for storage would be a complete waste of developer time.

Let's use the regular ESP files instead. But the editor can use project files internally for additional project-related settings that are matched via ESP filename/profile. We can even do that transparently. The user should not be required to bother with project files (or even know about them), but if he wishes he should be able to make use of them explicitly.
About the undo feature: Qt offers a framework for that, this is how it looks:
Well, that is a user-interface I definitely would not want to use. Completely overloaded with stuff. But we still may be able to make use of the Qt undo feature. Requires more discussion.

Okay, this thread is getting out of control. We should delay discussion of components until we now how to do the framework. Also, each framework should be discussed in a thread of its own.
Zini wrote: I moved the framework element discussions to separate threads. Please use these instead where available. From here on this thread should only contain discussions that are not covered by the other threads.

I think I moved all important postings. But if I have missed yours, just repost it.
Zini wrote: A note regarding git usage: Editor related branches should not be called editor or opencs (with the exception of my integration branch). Name the branch according to the feature you wish to implement.
Zini wrote: I added two new categories to the issue tracker and the first batch of framework related tasks.

I think we need to get 4g. Data Storage nad 4b. Components right first.
pogzy wrote: Hi,

It could be nice if the component design enables i18n and built-in help support, like tooltips or any other mean. Even if help content if rather empty for now, it is easier to think about it during design stage than after several implementation
iterations.

A very basic help like a web page per component could work. May be component tooltips can grab their help text from the concerned web page. Any other mean that could be easy to translate for non coder contributor should be OK.

An editor easy to use and easy to learn with a smart help could be very attractive in itself for any game designer that want to build RPG mods.
pvdk wrote: Help and online documentation can be provided using the Qt Help framework

Translation is done using Qt Linguist, we just have to make sure to define strings using tr("string"). This is already done in the launcher.
Ruarl wrote:
pvdk wrote:
Ruarl wrote:Just my two cents: If the ID lists are displayed in a spread sheet format please add a Sort function similar to MS Excel.
This is something that's high on our list, see this topic's first post, 4c. Filtering.
Edit: I was too quick with my reply. I looked up Excel's sorting function and it looks nice, although it could be implemented in a more user-friendly way.
Since the first reaction is that Filtering would get the job done I should explain how I see the sort function as beneficial to the Editor. The Filter function helps a modder filter out a large chunk of objects in order to only display the objects that he deems necessary for the task at hand. Modders would want this function mostly for editing exteriors and interiors.

Although TES-CS allowed you to sort the displayed data, it only allowed you to sort it according to one Column. (One Criteria) MS-Excel allowed you to sort the spreadsheet according to multiple columns. (Multiple Criteria) Multiple column sorting will help the modder better arrange the displayed data so he can use it easily and efficiently. Modders would want this function to manipulate the data (Objects) itself. One example is to balance the weapons against each other or arrange furniture according to chairs and tables.
pvdk wrote: I looked up Excel's sorting function and it looks nice, although it could be implemented in a more user-friendly way.
I agree. MS sucks at implementing stuff. I brainstormed a little and came up with one (possible) solution:

1) When you click on a column for the fist time it arranges the data in ascending order. When you click on a column for the second time it arranges the data in descending order. The third time you click on a column it returns to the original state.

2) A visual symbol on the column will show it is sorted in ascending (??) and descending (?¸) order. When no symbol is displayed on a column then it means that the column is not used as a criteria for sorting.

3) The order in which the user has selected the columns, is the order of priority in which the data is sorted.

4) If the order of priority is: Column A (1), Column B (2), Column C (3). When you remove Column B as a criteria then Column C's priority is increased to (2). Thus the order of priority is now: Column A (1), Column C (2)

5) I did not work in a visual cue to see in which order of priority the data is sorted. This could be changed, but IMO we should keep the GUI as simple as possible so we do not bombard the user with unnecessary information.

Well this is my other two cents. (Im a poor man :D)

Edit: On further reflection it is possible to work in a visual cue for order of priority:
2) In stead of a up/down arrow signifying ascending and descending order you can color code a priority number to signify ascending (1) and descending (1) order. So it will look like this: Column A (1), Column B, Column C (2), ect.

Ruarl
Zini wrote: Actually, that is part of a larger issue. We still have to consider how to display ID lists. I think we will need more configuration options. At least we need a way to configure which columns are displayed in a ID list (on a per-view basis). Since we are merging different ID types otherwise the list would look worse than the original TES-CS implementation.

My first thought was, that this is a ID list specific problem. But actually we could use table views for most record types, which means that we also can use column configuration for most record types.

I think the filtering feature was just upgraded from

- filtering the record list

to

- filtering the record list
- and filter which part of a record is displayed and in what sorting order.

Alongside the 2nd part we may consider batch changes: a function to change a single field in all selected records to a new value.

Might be a good idea to split it into two separate features. I just wish we could find a separate name for the second one. Calling them both filtering will be confusing.
Ruarl wrote:
Zini wrote:Actually, that is part of a larger issue. We still have to consider how to display ID lists... Since we are merging different ID types otherwise the list would look worse than the original TES-CS implementation.

My first thought was, that this is a ID list specific problem. But actually we could use table views for most record types, which means that we also can use column configuration for most record types.
First the decision should be how Id lists should be displayed.
1)Display all Id's then filter it.
2)Select/filter what Id's are wanted then display it.

Next I would assume that the Id lists would be displayed in a Table.
Zini wrote:Alongside the 2nd part we may consider batch changes: a function to change a single field in all selected records to a new value.

Might be a good idea to split it into two separate features. I just wish we could find a separate name for the second one. Calling them both filtering will be confusing.
The 2nd part should be seen as functions to configure the table, which does not really need a name.

Possible Configuration options:
1) A sort function like I explained in the previous post. (Or something similar)
Zini wrote:Batch changes: a function to change a single field in all selected records to a new value.
2) Batch Changes
Zini wrote: At least we need a way to configure which columns are displayed in a ID list (on a per-view basis). Since we are merging different ID types otherwise the list would look worse than the original TES-CS implementation.
3) Manually hide (Not display) columns deemed unnecessary. (If only clothes are displayed then slash damage columns ect. should automatically not be displayed. Unless you want hit someone with a shoe ;) )
OpenMW Wiki wrote:Allow much greater modability: change game rules, create new spell effects, etc through scripting.
4) Be able to add Fields (columns) for more modability. For example: Add chance to hit column to weapon ID's. (Of cause these fields should come with a corresponding script to handle the new data.)

5) Perhaps be able to remove Fields (columns)? Might be necessary to support number 4)

Ruarl
Zini wrote:
First the decision should be how Id lists should be displayed.
1)Display all Id's then filter it.
2)Select/filter what Id's are wanted then display it.
The ID list will be a persistent component. I guess 1) would be a better description.
The 2nd part should be seen as functions to configure the table, which does not really need a name.
It's a feature, that will be applied to several types of components; probably implemented in a separate class. Having a name would be useful.
3) Manually hide (Not display) columns deemed unnecessary. (If only clothes are displayed then slash damage columns ect. should automatically not be displayed. Unless you want hit someone with a shoe ;) )
Nope. Even if you reduce the table automatically, it would still be too big most of the time. Manual configuration is a must.
4) Be able to add Fields (columns) for more modability. For example: Add chance to hit column to weapon ID's. (Of cause these fields should come with a corresponding script to handle the new data.)

5) Perhaps be able to remove Fields (columns)? Might be necessary to support number 4)
Definitely not! When we add a field, we must modify the file format, which means we must modify OpenMW. Then we can as well modify the CS. No point in adding more configurability here.
Ace (SWE) wrote: I think we should have separate file formats for legacy plugins and ones using OpenMW exclusive features, to keep confusion to a minimum. You could also tell the user when he uses a OpenMW added feature that he will be losing backwards-compability.
The Starcraft and Warcraft 3 editors both did something similar to this, forcing you to save your level as an expansion level if you used any of the features that were added by an expansion.

Image example of the Starcraft editor:
Image
Ruarl wrote:
Zini wrote:
4) Be able to add Fields (columns) for more modability. For example: Add chance to hit column to weapon ID's. (Of cause these fields should come with a corresponding script to handle the new data.)

5) Perhaps be able to remove Fields (columns)? Might be necessary to support number 4)
Definitely not! When we add a field, we must modify the file format, which means we must modify OpenMW. Then we can as well modify the CS. No point in adding more configurability here.
The Idea is to attach data (adding a field) to a batch of ID's without changing the ID itself. Like a data range of a script whose values you can configure inside the table. OpenMW won't load these new values until a script made by a modder makes use of the values. But if that is impossible then we should leave it at that for now. No. 5 will have to go, because it will change the ID itself and that might screw up OpenMW.
Ace (SWE) wrote:I think we should have separate file formats for legacy plugins and ones using OpenMW exclusive features, to keep confusion to a minimum. You could also tell the user when he uses a OpenMW added feature that he will be losing backwards-compability.
I think losing backwards compatibility should be avoided at all costs. There should always be an option to reverse the changes that the modder has made.
Zini wrote:
3) Manually hide (Not display) columns deemed unnecessary. (If only clothes are displayed then slash damage columns ect. should automatically not be displayed. Unless you want hit someone with a shoe ;) )
Nope. Even if you reduce the table automatically, it would still be too big most of the time. Manual configuration is a must.
I should have split up number 3 into two:
3a) Manual hide function (Part of multiple configuration functions)
3b) Auto hide function (Options for the user to configure what is hidden when.)

Ruarl
Zini wrote:
I think we should have separate file formats for legacy plugins and ones using OpenMW exclusive features, to keep confusion to a minimum.
The ESP/ESM format is very expansible. Having a 2nd format would be a waste of resources.

The editor should keep track of extended feature though. But we can look into this later (1.0 won't have any changes, that require a format change).
Ace (SWE) wrote:
Zini wrote:The ESP/ESM format is very expansible. Having a 2nd format would be a waste of resources.
You don't need to create a full new file format, just let them have another file extension that uses an extended ESP/ESM format. If all we do is extend the existing format then we could just treat the new files as ESP/ESMs
Zini wrote: I am not so sure about that. I see our work as a new version of Morrowind/the ESM/ESP-formats. Usually when you move a file format to a new version, you don't rename the extension. It is likely that we will end up with format extensions for nearly every major release after 1.0. We can't rename the extension each time.
Oblivion is using the same extension as MW, so its not like we are making it any worse.

Also the ESM/ESP-formats contain a version specifier, that is checked by the original game. MW/Oblivion should handle an erroneous attempt to load one of our files gracefully.

From a practical point of view I don't see much of a problem anyway. Unless we screw up, OpenMW will displace the original Morrowind executable completely.
Star-Demon wrote: the version number included in the file will make things easy for us to deal with.
Zini wrote: @Ace: I re-read the topic. Your arguments still don't convince me. But after reading my last posting, I noticed that one of my counter-arguments is apparently an argument in favour of your position.

Newer TES-games are using a different file format with the same extension (at least Oblivion does). Having new versions of the same format using the same extension is reasonable. Having a different branch/format using the same extension is not.

I suggest the following course of action:

- for now, we do nothing
- after 1.0, we invent a new extension (I will veto against anything espx-like; sounds too much like docx).
- OpenMW will check for both extensions and will treat them equally; if the user is putting an Oblivion plugin in the OpenMW data directory it is his own fault (the version check mechanism would catch this problem anyway)
- The editor will read both extensions and will treat them equally.
- The editor gets a global setting, that decides which extension is used for new plugins (the default will be the new extension).
Star-Demon wrote:
Zini wrote:- after 1.0, we invent a new extension (I will veto against anything espx-like; sounds too much like docx).
lol!

I don't remember what the latest version number for the esm/esp format is (1.30?), but we could increment it for the extended format if we don't want to deal with any filename extension problems.

Although I do like the idea that you could develop it in a way that different extensions have different mechanical features. Hmmm...
Locked