OpenMW CS - Merge

Questions specific to OpenMW-CS can be asked, and content development related topics can be discussed here
User avatar
Zini
Posts: 5538
Joined: 06 Aug 2011, 15:16

Re: Openmwcs-merge

Post by Zini »

Surely the conflicts still exist between C2 and B' and can, therefore, be detected?
No, they can't. Conflict means both C1 and C2 modify the same record. With B' the original state of records in B (that got overwritten by C1) are lost. Therefore the conflict doesn't exist anymore and C2 would simply overwrite any changes made in C1.

With a typical multi-person Morrowind-project workflow that is not the desired outcome. Here is what we did with Redemption (and I am pretty sure TR does something similar):

- The master file is handed out to all content developers
- Content developers create a content file based on that master file.
- The project leader/coordinator makes a "pens down" call. Everybody stops working and sends in their content files.
- A new master file is created by merging in all content files (hopefully without conflicts, but shit happens).
- Start from the top.
I was under the impression that the Fallout 4 and Skyrim SE creation kits have a merge tool built in which is intended for use in version control and has basically the same workflow as Git. It's what was used to build the games in the first place. That's the concept I have in my mind for this kind of tool, and I don't see any reason it shouldn't be feasible if Git itself is feasible. Someone should probably investigate what the official Bethesda tools do as that's a good thing to try and emulate if it's fit-for-purpose.
That would require building an entire version control system (git sucks for binary files). I don't see how we could possibly achieve that in any realistic span of time. Anyway, that is a separate issue from the merge tool.
Atrayonis
Posts: 7
Joined: 17 Jul 2018, 20:10

Re: Openmwcs-merge

Post by Atrayonis »

Hello, I'm from Tamriel Rebuilt and I'm here to help.

Our workflow seems to be a bit different from that of Redemption.

As our assets are kept in the Tamriel_Data.esm and BSAs, modders just make esps adding to or overriding assets. They are first merged in a TR and PT data addon esp and merged into the real Tamriel Data later. Deletion notices are placed in text form and the lead developer tasked with merging the files has to go and do these manually. This is PT's call, as they have been leading efforts on Tamriel_Data.

The claim workflow is similarily compartmentalised: first we cut our heightmap files (TR_RestExterior) into section files, chop off worldspace bits (3x3 cells or larger) and then put them out for developers to claim. There's some border (landscape) matching happening at this stage, but that is between individual modders and claims and the esps themself never exceed their original cell boundaries.

Interiors are similarily posted and developed individually: one interior, one claim. They have temporary names until they are merged back into the section files.

Both exterior and interiors are stitched together into the section file. As these files are either using assets implemented globally in Tamriel_Data or claim-specific ones (using existing assets but a unique ID), there are (theoretically) no ID or cell conflicts.

Once exteriors and interiors are finished enough, we have NPC/Dialogue and quest claims, which is where the merge complexity actually comes in.

What I would really like:
  • Better dialogue conflict resolution. When dialogue has gone wrong, it drops to the bottom of the order, something to help re-stick a bulk of changed replies would be a godsend.
  • Conflicting cell names (for interiors specifically) would be really good. With the size of TR, there have been cases of duplicate cell names which weren't caught until months after merging as the content of the interior cells were spaced too far apart to actually visibly conflict.
  • Automatically discarding empty "edited" wilderness cells and landscape edits without corresponding cell edits (this has actually happened).
  • An option to automatically discard duplicate entries between an esp and an esm.
  • A pipe dream would be a way to merge deletions of existing assets.
User avatar
MinerMan60101
Posts: 24
Joined: 09 Sep 2017, 15:40
Location: California: U.S.A.

Re: Openmwcs-merge

Post by MinerMan60101 »

Further comments from TR devs here:
https://www.tamriel-rebuilt.org/forum/o ... t-comments
User avatar
Zini
Posts: 5538
Joined: 06 Aug 2011, 15:16

Re: Openmwcs-merge

Post by Zini »

Can we keep the discussion in this thread, please? Splitting it across two forums would be kinda bothersome.

I am not entirely clear about the workflow you described. Will have to read through it again, once I am not so much in a hurry anymore. Probably will have to ask for a couple of clarifications.

For now just one important point: Your workflow is obviously defined by the limitations of your tools. Once OpenMW-CS is available, it would make sense to adjust the workflow. But then, we are still in the design stage of the merge tool. It would make even more sense to design the merge tool and the optimised workflow in parallel.

I'll get back to you soon. Watch this space.
User avatar
Zini
Posts: 5538
Joined: 06 Aug 2011, 15:16

Re: Openmwcs-merge

Post by Zini »

It seems we are using different terminology and different abbreviations, which would explain why half of your post didn't make any sense to me on the first read.

First, what about "Asset": Is that your name for a record? Race, Activator, Cell and so on?
As our assets are kept in the Tamriel_Data.esm and BSAs, modders just make esps adding to or overriding assets. They are first merged in a TR and PT data addon esp and merged into the real Tamriel Data later. Deletion notices are placed in text form and the lead developer tasked with merging the files has to go and do these manually. This is PT's call, as they have been leading efforts on Tamriel_Data.
Sorry, I am not really clear what is going on here. Too many esm and esp files for me to follow the process. Can you explain a bit more in detail? Also, what is PT?
The claim workflow is similarily compartmentalised: first we cut our heightmap files (TR_RestExterior) into section files, chop off worldspace bits (3x3 cells or larger) and then put them out for developers to claim. There's some border (landscape) matching happening at this stage, but that is between individual modders and claims and the esps themself never exceed their original cell boundaries.
Do I read that correctly? You take your main content file, delete everything but 3x3 cells (or larger) and hand that out to the content developer?
Better dialogue conflict resolution. When dialogue has gone wrong, it drops to the bottom of the order, something to help re-stick a bulk of changed replies would be a godsend.
Can't say anything about that right now. We seem to have a few problems with dialogues ourself (bugs that may indicate deeper problems, plus a missing feature that we have no implementation path for yet). Will have to look at the whole situation again at a later time.
Conflicting cell names (for interiors specifically) would be really good. With the size of TR, there have been cases of duplicate cell names which weren't caught until months after merging as the content of the interior cells were spaced too far apart to actually visibly conflict.
This will probably become more clear once I understand the initial paragraphs. Right now I don't understand what exactly conflicting cell names would be. Conflicts between which files?
Automatically discarding empty "edited" wilderness cells and landscape edits without corresponding cell edits (this has actually happened).
That shouldn't really happen with OpenMW-CS, because with the right configuration you can not accidentally change a cell. If this feature is still required it is most likely a task for an editor script (post-1.0).
An option to automatically discard duplicate entries between an esp and an esm.
Again, not sure what actually would constitute a duplicate here.
A pipe dream would be a way to merge deletions of existing assets.
That is already implemented in our current merge tool and should also make its way into any new tool.
User avatar
MinerMan60101
Posts: 24
Joined: 09 Sep 2017, 15:40
Location: California: U.S.A.

Re: Openmwcs-merge

Post by MinerMan60101 »

First, all of TR's internal files are right here to look at so you can better understand how they work:
https://www.tamriel-rebuilt.org/releasefiles
First, what about "Asset": Is that your name for a record? Race, Activator, Cell and so on?
Pretty much, Tamriel Data has everything that one would drag over from the objects window into the render window for use, and everything that isn't an esp/m file that TR or PT needs to have to work properly
Sorry, I am not really clear what is going on here. Too many esm and esp files for me to follow the process. Can you explain a bit more in detail? Also, what is PT?
PT is Project Tamriel, which is another province mod group making Skyrim and Cyrodiil, quite similar to us at this point.
So, Tamriel Data.esm has a CS record for every single nif we use, TR_Data.bsa and PT_Data.bsa are Morrowind standard archives containing every nif and dds the mods need (downloading TR or PT won't get you any of them, you have to get Tamriel Data first).
Both project has an internal file that mimics Tamriel Data but only for new assets, which are the Data Addons, where our assets get merged into before finally, both PT's and TR's data addons are merged into Tamriel Data and deletion and such is sorted out.
Do I read that correctly? You take your main content file, delete everything but 3x3 cells (or larger) and hand that out to the content developer?
Yes.
Can't say anything about that right now. We seem to have a few problems with dialogues ourself (bugs that may indicate deeper problems, plus a missing feature that we have no implementation path for yet). Will have to look at the whole situation again at a later time.
Can't comment on this, dialogue is a mystery to me
This will probably become more clear once I understand the initial paragraphs. Right now I don't understand what exactly conflicting cell names would be. Conflicts between which files?
The problem is that when two files have an interior of the same name, then when those files are merged it just combines the two worldspaces with both interiors in the same interior worldspace and same idea. My solution (as previously stated) would be asking to combine them or rename one of them. - Also to be extra sure, interior/exterior names as seen ingame are their IDs unlike later games
That shouldn't really happen with OpenMW-CS, because with the right configuration you can not accidentally change a cell. If this feature is still required it is most likely a task for an editor script (post-1.0).
See my comment here for my question related to this (for TR people reading this rather than Zini): viewtopic.php?f=7&t=4322&p=56773#p56761
Again, not sure what actually would constitute a duplicate here.
The same thing as interiors, which I have previously mentioned with regards to them.
The CS stores things as IDs with additional information in them (take a look at TESAME, link/tutorial in the TR Handbook, gives a list of all of a esp/m's ID's) for everything that it can do. What Atrayonis was referring to was merging two esps that both have something stored under the same ID. I think this should be tackled in different ways for different things:
Most CS objects in the Object Window (except leveled lists): just ask which one to keep (There may be room for more specification, but I can't think of any off of the top of my head)
Different types of things (like an interior and a book with the same ID): Ask which one to keep or to rename one of them
Interiors: Prompt to combine the cells into one worldspace or to rename one of the cells
Leveled Lists: like interiors, prompt to merge or rename one of them
Exteriors: Prompt to select which file's landscape (the 'LAND' record) wins and whether to delete the statics (and whatnot, basically the 'CELL' record) from the one that doesn't.
Dialogue nonsense: [shrug] You'll have to wait for a Quest/Dialogue person's input on this.
That is already implemented in our current merge tool and should also make its way into any new tool.
Woot! This is one of the biggest hangups/annoyances with Tamriel Data related things.

As is hopefully the theme in this post, these are based solely on my experience and knowledge and what I personally would like to see in a merge tool as a result of that. Other TR or PT devs will certainly have different experience and thus different suggestions
Last edited by MinerMan60101 on 18 Jul 2018, 17:01, edited 2 times in total.
User avatar
MinerMan60101
Posts: 24
Joined: 09 Sep 2017, 15:40
Location: California: U.S.A.

Re: Openmwcs-merge

Post by MinerMan60101 »

As Zini was against splitting the discussion across two forums, here is the one comment on the TR forums that didn't make it here:
From Rot:
Repeating another point from chat, as I saw it the main block for merging in TR comes not from the merging process itself but from the ESM instance index being what forces you to cut and replace portions and never directly edit any cells, wrye mash updating aside. I don't know why beth's CS generates a different cell index every time a plugin is saved and if openMW-CS doesn't do that, the issue is already gone. Maybe the CS does it to save time when saving files, an option to not do it would be good enough.

Re: " When dialogue has gone wrong, it drops to the bottom of the order " in Atrayonis' post, that comes from a dialogue entry not being able to find its previous entry reference. The CS gives an error message but modders often don't know to wait for them and skip all warnings from the first one, and when they save the file the dialogue is saved at the bottom of the list and there isn't an error anymore. It's something the loading process for plugins needs to pick up on, but since loading plugins is the first thing a merging tool does, all these checks are necessarily part of it already.
Atrayonis
Posts: 7
Joined: 17 Jul 2018, 20:10

Re: Openmwcs-merge

Post by Atrayonis »

I'm going to try to clarify some points. We really are using different terminology and I'm only going to supplement MinerMan's reply in the parts where I think further elaboration is necessary. Otherwise he's pretty much right.

For terminology example, "content developer" is not a term we could or do use, it's way too unspecific. We divide by asset developer (further: modeler, texturer, sound guy) and exterior/interior/quest developer.
First, what about "Asset": Is that your name for a record? Race, Activator, Cell and so on?
An Asset (the things we keep in Tamriel_Data) are globally(1) defined and used Objects (as the CS uses the term) except for NPCs, plus Sounds, Scripts, Races, Classes, Regions, and Factions.(2)

Assets generally follow a top-to-bottom workflow: a developer posts them on our website, they are evaluated, categorized and included for use or rejected. They are merged into project-specific Data Addons over time and eventually merged into Tamriel_Data whenever a new release of whatever project is due or the backlog has become too big to manage.

(1) "Globally" defines the cooperative Tamriel that is being developed by Tamriel Rebuilt and the Project Tamriel mods (Skyrim: Home of the Nords, Province: Cyrodiil, and to a lesser extent High Rock 427 and Elsweyr Paradise of Sugar).
(2)There are also local assets, which are simply location- or NPC-bound scripts or Objects (etc.) edited for use in a cell or quest. These do use a different CS ID but never have icons, meshes, textures, bookart, or sound which isn't already part of Tamriel_Data.

Do I read that correctly? You take your main content file, delete everything but 3x3 cells (or larger) and hand that out to the content developer?
No. To start with, Claims (unlike Assets) are a top-down workflow.

There is no main content file per se.

RestExterior is a heightmap file, it's a big blob in the shape of Morrowind (3) containing only worldspace cell edits (exteriors) - usually only a heightmap. It only exists to be cut into pieces - no development takes place here, it's literally the "rest" of the landmass.

From RestExterior, after planning has progressed to a sufficient state (we know where the cities, biggest landmarks, and dungeons go and have an overview of the character of the region), a Section Files is cut out of RestExterior so that actual development can happen. For example, here is an image of the Lake Andaram section. Everything south, west, and east is RestExterior, while other section files exist to the north.

The section file is then further cut into Exterior Claims, the 3x3 cell chunks.
This is where the actual TR developers come into play - up until now, it has all been part of the Lead Developer workflow. "Claims" get their name from how this workflow operates - a lead developer posts a claim on our website and an accredited developer can claim it and work on it exclusively until they deem it ready to be merged back into the section file after a review.
For example, see how the Lake Andaram section was divided and one resulting exterior claim.

Interior Claims are created when an exterior has either been merged back into the section file or is judged to be sufficiently far along by the Lead Developers. These are the interiors for individual houses, grottos, temples.
For example, the Lake Andaram 1&5 claim contains the settlement of Idathren. A claim map for that settlement was generated when it was in a fairly rough state and interior claims were made from this, like this one.

When the exteriors/interiors have been merged back into the section file, NPC Claims are made. This is usually one claim per settlement or cave/grotto/temple and consists of a list of NPCs (derived from the planning that went into the interior claims) and their generic/personal dialogue, ownership, etc. For an example, the NPC claim for Menaan (a settlement in the Roryn-Bluff section file) is here.

After the settlement essentially exists, Quest Claims are opened for faction and miscellaneous quests. Since these always edit preexisting characters, dialogue, cells, and objects, and might need to insert their own, quest-specific, NPCs, they are implemented last (planning, of course, has happened before). For an example, see Where has the horse gone? a quest for Old Ebonheart, which includes new Ashlanders, scripts, and

While this process is ongoing, for releases the section file is going to be merged with the other section files and whatever remains of RestExterior into TR_Preview.esp.
When the section is actually finished (exteriors, interiors, npcs, quests, and cowbell are all done and merged), it is merged into TR_Mainland.esm, which is - as far as I understand the term - our main content file alongside the Tamriel_Data asset repository.

(3) Or Cyrodiil, Skyrim, High Rock, depending on the project, the idea is pretty universal.
Can't comment on this, dialogue is a mystery to me
As Rot was quoted, the problem with Morrowind's dialogue is that it isn't strictly a database, it's a chain. Every reply contains not only an ID, but also a Previous ID and a Next ID. When these can no longer be resolved, the dialogue drops to the bottom of the reply list.
This will probably become more clear once I understand the initial paragraphs. Right now I don't understand what exactly conflicting cell names would be. Conflicts between which files?
Morrowind has no cell id, just worldspace coordinates and the <interior> demarcation. The cell name ("Seyda Neen, Arille's Tradehouse") is the cell's unique identifier (also used for dialogue filtering), which means that if two esps add things to the "Seyda Neen, Arille's Tradehouse" interior, both will coexist if they are merged into one file.

This is not desirable for us without a warning prompt. No cell (name) we are editing should exist prior to the NPC claim stage and the projects are so big that it's a very real danger that two ancestral tombs or shipwreck interiors halfway across Morrowind will end up with the same name.
User avatar
Zini
Posts: 5538
Joined: 06 Aug 2011, 15:16

Re: Openmwcs-merge

Post by Zini »

Repeating another point from chat, as I saw it the main block for merging in TR comes not from the merging process itself but from the ESM instance index being what forces you to cut and replace portions and never directly edit any cells, wrye mash updating aside. I don't know why beth's CS generates a different cell index every time a plugin is saved and if openMW-CS doesn't do that, the issue is already gone. Maybe the CS does it to save time when saving files, an option to not do it would be good enough.
Not sure if I understand that correctly. This is the first time I hear of such complications. What exactly is the meaning of the terms instance index and cell index? The FRMR sub-record?
User avatar
Zini
Posts: 5538
Joined: 06 Aug 2011, 15:16

Re: Openmwcs-merge

Post by Zini »

For terminology example, "content developer" is not a term we could or do use, it's way too unspecific. We divide by asset developer (further: modeler, texturer, sound guy) and exterior/interior/quest developer.
Content developer is just a more generalised term for modder (because people may use OpenMW to create whole new games that don't mod anything).

With Redemption back then we had also different names for specific developer types: Modeler we called 3D artist, exterior/interior developers we called worldbuilders and quest developers where split into writers and the poor guy who failed to dodge the implementation task.

Your definition of asset is interesting. Basically stuff that can be used by worldbuilders. Makes sense.
As Rot was quoted, the problem with Morrowind's dialogue is that it isn't strictly a database, it's a chain. Every reply contains not only an ID, but also a Previous ID and a Next ID. When these can no longer be resolved, the dialogue drops to the bottom of the reply list.
Yeah, that is the same issue that gives us trouble, though not necessarily in the same way. As I said, we will have to take another look at that when the other issues are out of the way. Its been a long while since I looked at this problem and I am not entirely sure where we are standing with this right now.
Morrowind has no cell id, just worldspace coordinates and the <interior> demarcation. The cell name ("Seyda Neen, Arille's Tradehouse") is the cell's unique identifier (also used for dialogue filtering), which means that if two esps add things to the "Seyda Neen, Arille's Tradehouse" interior, both will coexist if they are merged into one file.
Yeah, I understand the issue. I wasn't clear on between which files the conflict could arise, but I think I understand now. Unfortunately there is not much we can do about that in the merge tool itself, because from the merge tool's perspective the right thing to do would be to overwrite the old cell with the one newly merged in (that is if one is merged after the other). For many projects that is probably the desired behaviour. But it may be possible to add some editing scripting later on, that warns about such issues.

Thanks for taking the time to explain the procedures to me. You surely have a lot of them ;) Could I have a last clarification?

If I understand it the procedure is as follows (slightly simplified):
- Create section file from RestExterior
- Cut section file into claims and let exterior worldbuilders work on them.
- Collect claims and merge them into section file
- Interiors are likewise merged into the section file
- When everything is done, the section file is merged into TR_Mainland.esm

Is the merging into the section file done all at once or in batches or piece by piece as the finished claims come in?

Edit: Sorry in advance for the confusion we will cause if and when TR moves over to OpenMW-CS. I just noticed that we also use the term assets in our editor, but in a somewhat different way.
Post Reply