OpenMW CS - Merge

Questions specific to OpenMW-CS can be asked, and content development related topics can be discussed here
User avatar
MinerMan60101
Posts: 24
Joined: 09 Sep 2017, 15:40
Location: California: U.S.A.

Re: Openmwcs-merge

Post by MinerMan60101 »

Is the merging into the section file done all at once or in batches or piece by piece as the finished claims come in?
With interiors it can be either, it moreso depends on who is doing it. Basically there is no set way to do it, they just get merged when someone merges them in.
With Exteriors we are in new territory since we haven't yet merged exteriors into a section file since we've restructured (and even i came into the project after), but for the current set of exteriors we're waiting on my exterior claim to be finished before merging anything, so it can be a continuous area, as there is one finished claim to the west of mine, and two finished claims to the right of mine. So, I think once we have a solid continuous base, we'll just merge finished exteriors that are adjacent to the ones we already have.
With Quests, it is basically as they get done, especially as most of them happen leading up to a release.
Unfortunately there is not much we can do about that in the merge tool itself, because from the merge tool's perspective the right thing to do would be to overwrite the old cell with the one newly merged in (that is if one is merged after the other). For many projects that is probably the desired behaviour. But it may be possible to add some editing scripting later on, that warns about such issues.
Please see my comment on my previous post for what that advanced behavior should be in the end for conflicting IDs:
Most CS objects in the Object Window (except leveled lists): just ask which one to keep (There may be room for more specification, but I can't think of any off of the top of my head)
Different types of things (like an interior and a book with the same ID): Ask which one to keep or to rename one of them
Interiors: Prompt to combine the cells into one worldspace or to rename one of the cells
Leveled Lists: like interiors, prompt to merge or rename one of them
Exteriors: Prompt to select which file's landscape (the 'LAND' record) wins and whether to delete the statics (and whatnot, basically the 'CELL' record) from the one that doesn't.
Dialogue nonsense: [shrug] You'll have to wait for a Quest/Dialogue person's input on this.
It can probably get even more advanced than this in certain cases, but this should cover pretty much all of the non-dialogue bases for conflicts in a useful, straightforward, and user-friendly way.
Atrayonis
Posts: 7
Joined: 17 Jul 2018, 20:10

Re: Openmwcs-merge

Post by Atrayonis »

Not sure if I understand that correctly. This is the first time I hear of such complications. What exactly is the meaning of the terms instance index and cell index? The FRMR sub-record?
I don't know what FRMR sub-records are, but Rot is referring to internal references, Morrowind's internal IDs for any object placed into the worldspace or an interior. As soon as the master file has a diverging amount of objects placed into the worldspace, plugin files referring to them will cease to work right - loading them up in the CS will create a lot of "reference no found" errors and in most cases will result in a duplicate created instead.
This makes esp edits and mergers back into changed masters impossible, instead we need to delete and replace.

OpenMW 0.40.0 had a very welcome change - "Fixed segfault in Atrayonis’s “Anthology Solstheim: Tomb of the Snow Prince” mod" - that dealt with a similar problem (two master files editing the same object, thereby creating two divergent internal references, error-crashing vanilla Morrowind and crashing OpenMW).

Also, adding to what MinerMan has already said:
Is the merging into the section file done all at once or in batches or piece by piece as the finished claims come in?
Bit by bit, but with the progression exterior->interior->npc->quest as the basic assumption and the section file as the parenthesis holding everything together. I'm pretty sure that this is how the Project Tamriel mods do it as well. The idea is that the average exterior or interior claim is essentially like a bird bath - a modder flies in, dips their beak into the water, then flies off again.

We only have developed this formal workflow since 2015/2016 but as far as I can tell, the workflow of old followed the same general idea. Ultimately, for a mod with Tamriel Rebuilt's size, anything but a highly compartmentalized workflow is detrimental to its long-term survival and Morrowind's tools pretty much require it anyway.

Aside from the internal reference problem mentioned above, the biggest change a new merging tool could supply us with - beyond what the CS, TESAME and TES3CMD can offer with a lot of manual fumbling - would be the holy grail of enabling merging of byte changes akin to a source code project as you can find them on Git or CVS; I do not believe that is possible with Morrowind's esm/esp format, but I'd gladly be shocked and awed.
Edit: Sorry in advance for the confusion we will cause if and when TR moves over to OpenMW-CS. I just noticed that we also use the term assets in our editor, but in a somewhat different way.
How do you use "asset" in OpenMW-CS? Everything involving meshes?

As this is probably a long-term problem, there's going to be enough time to adjust our vocabulary. ;)
User avatar
Zini
Posts: 5538
Joined: 06 Aug 2011, 15:16

Re: Openmwcs-merge

Post by Zini »

I don't know what FRMR sub-records are, but Rot is referring to internal references, Morrowind's internal IDs for any object placed into the worldspace or an interior. As soon as the master file has a diverging amount of objects placed into the worldspace, plugin files referring to them will cease to work right - loading them up in the CS will create a lot of "reference no found" errors and in most cases will result in a duplicate created instead.
This makes esp edits and mergers back into changed masters impossible, instead we need to delete and replace.
The FRMR sub-record is the only thing that kinda forms an internal ID in OpenMW and is present in files. Not sure how to go about clearing that up. But let's assume for now that we are talking about the same thing.

With worldspace, you actually mean cell, right? I do not understand why the situation you describe would lead to such problems, at least not within OpenMW and OpenMW-CS. I am pretty sure that an error like "reference not found" doesn't exist in either application. But our internal data structures are different from the ones used in the original applications. Maybe this problem simply does not exist with OpenMW. If you happen to find the time to run a test, I would be very interested in hearing about the results.
OpenMW 0.40.0 had a very welcome change - "Fixed segfault in Atrayonis’s “Anthology Solstheim: Tomb of the Snow Prince” mod" - that dealt with a similar problem (two master files editing the same object, thereby creating two divergent internal references, error-crashing vanilla Morrowind and crashing OpenMW).
I looked that up and if our record keeping is correct, this fix was about a plugin moving an instance (what you call reference, I believe) from one cell to another. That sounds like a totally different problem, unless I complete misunderstood your original statement.
How do you use "asset" in OpenMW-CS? Everything involving meshes?
Any file that comes with a mod/content that isn't an esm/esp/omwgame/omwaddon or an ini file: Meshes, textures, icons, music, sound, videos. If you happen to have OpenMW-CS you can see for yourself. Main menu -> assets and then everything below the second separator line.


Okay, from what I understand I can go ahead with the new merge tool (as described higher up in this thread). Together with a porting tool it should provide you with everything you need to proceed, though certainly not with everything on your wishlist. This should be enough for 1.0 and we can build on it in stage1 and beyond.
User avatar
MinerMan60101
Posts: 24
Joined: 09 Sep 2017, 15:40
Location: California: U.S.A.

Re: Openmwcs-merge

Post by MinerMan60101 »

Any file that comes with a mod/content that isn't an esm/esp/omwgame/omwaddon or an ini file: Meshes, textures, icons, music, sound, videos. If you happen to have OpenMW-CS you can see for yourself. Main menu -> assets and then everything below the second separator line.
That looks to be the same definition we use, except we may include books, or they're only on this part of the website because that's a good place for them. Quest designs are not assets, they're just put here until they can be made into claims.
TR's asset browser:
https://www.tamriel-rebuilt.org/asset
Okay, from what I understand I can go ahead with the new merge tool (as described higher up in this thread). Together with a porting tool it should provide you with everything you need to proceed, though certainly not with everything on your wishlist. This should be enough for 1.0 and we can build on it in stage1 and beyond.
That sounds good to me: basic strictly necessary features now, but that laundry list of conflict resolution features I described sometime in the future.
What do you mean by porting tool? I know you can rename .ownaddon's to esp's and they still work, does it not yet go the other way around?

I'm glad that you guys reached out to us for our opinions, and be sure to reach out to TR, PT, and the modding community as a whole in the future to find out just what OpenMW is missing!
User avatar
Zini
Posts: 5538
Joined: 06 Aug 2011, 15:16

Re: Openmwcs-merge

Post by Zini »

What do you mean by porting tool? I know you can rename .ownaddon's to esp's and they still work, does it not yet go the other way around?
That's not what I meant. If you have content files A and B, both depending on C and then merge A into C you get a new content file C2. But since B is depending on C and not C2, you can not merge B into C2, unless you first change the dependency in B from C to C2. That's what we have started to call porting. Technically if you give C2 the name C you could skip the porting step. That is somewhat risky though, because you skip a bunch of useful error checks.

Hm, but apparently you do that anyway. So maybe we can even get away with no porting tool for 1.0. Less work for us. I am not complaining.

In regards to omwaddon/esp: This conversion isn't officially supported. It should work, but we don't test this functionality. After 1.0 (with OpenMW-CS 1.1) this will definitely not work anymore.

Edit: This is explained in the section "Content File Format & Namespaces" in our stage1 design document: https://gitlab.com/OpenMW/openmw/blob/m ... -stage1.md (loads a bit slowly right now, GitLab seems to be under high load).
User avatar
MinerMan60101
Posts: 24
Joined: 09 Sep 2017, 15:40
Location: California: U.S.A.

Re: Openmwcs-merge

Post by MinerMan60101 »

snip
Ah, makes sense.

As you suspect/alluded to, the way we do things naturally avoids dependability in many scenarios. Interiors and exteriors only depend on Tamriel_Data, but not on whatever we're merging it into. Section files are all dependent on TR_Mainland.esp (note: not esm, we use esp's internally because of the whole 'when a mod edits an esm and that esm gets updated it breaks the mod' thing) Whenever we load up a section file to edit, we (ideally) load it with the latest Mainland file, and it gets updated to work with that one automatically somehow. The only time we merge directly into mainland are quests and section files about to be released (i.e. the Spring/Summer Release Section containing Old Ebonheart), and even then when we do the file is working off of the latest Mainland file. So, to sum up, we basically avoid the need for a porting tool. (I'm pretty sure, Atrayonis will probably have more comments tomorrow)

I'll be sure to remember to take a look at that excerpt of the 1.0 doc when it's not super laggy.
Post Reply