openmw.org

Tarius wrote:Actually, I have over 425mb of esp and esm files. Granted I dont use all of them at once, but I think a couple hundred MBs isnt out of the question for heavily modded stuff.

Sure, I could envision some really heavily modded games to get up around 300 or 400MB in esm/esps, though it's also known that having a lot of mods increases memory requirements, too.

That said, I agree that not loading things is not good. The main reason is because it elminates the possiblity of having NPCs move around. Although, I dont think there is any reason to load everything in existance and it would be fine to only do those nearby cells.

Right, for unloaded(+uncached) cells all you'd really have is the reference data in the esx files. Loaded cells get everything (scene nodes, entities, etc), and that's where the vast majority of the memory use comes from. The cell cache is likely used to keep some of this in memory when the cell itself gets unloaded, so it can be brought back very quickly (I wouldn't be surprised if it's essentially just disconnecting the cell's root scene node and stopping AI/script processing; loading a cached cell would then just be reconnecting the scene node and processing stuff again).

That doesn't sound optimal. Can't scripts can change references that aren't in loaded cells?

Yes and that is already implemented. There is a preload stage that roughly determines if an inactive cell might be touched by a script. OpenMW then runs through these candidate cells and loads them until the right reference is found. There are also some other optimisations in place and once we start changing the ESX format there is even more we can do.

In the future, won't we allow AI processing for actors not in loaded cells (e.g. allowing guards to continue to track you after you leave the cell)?

I don't think that is a good idea. If you have all NPCs (and creatures) running all the time that would kill performance. If you only want to run a single NPC (or a small number) then there are other methods. In the worst case you can keep a few additional cells loaded. But you would most certainly want to apply some LOD techniques to far away NPC activity.

Zini wrote:Yes and that is already implemented. There is a preload stage that roughly determines if an inactive cell might be touched by a script. OpenMW then runs through these candidate cells and loads them until the right reference is found. There are also some other optimisations in place and once we start changing the ESX format there is even more we can do.

The ESX format seems to be closely tied to the way the engine works with the data. If OpenMW has trouble handling the data, or is significantly altering the data before use, it's an indication that it's handling it wrong. As such, this seems to be reaching into stages of premature optimization... trying to keep otherwise accessible data inaccessible (unloaded), and applying tricks to figure out if it needs to be accessed, when indications are the ESX data gets loaded at game start and stays loaded.

Given too how little the ESX data is compared to everything else, I'm not sure it's saving much, either. With the vanilla GOTY, keeping the ESM's cell data unloaded can't be saving more than 30 or 40MB... and that's assuming the cell data is near 50% of the ESM data size, and that a bunch of cells don't get forced to load anyway. Such savings don't seem to be worth it for the issues it causes (such as trying to figure out what cells need to be loaded and when, and these with moved references; instead of getting it and fixing it all up at once with everything there on load, you have to piecemeal it together by searching through a bunch of plugins during run-time).

But you would most certainly want to apply some LOD techniques to far away NPC activity.

That's apparently what Oblivion does. NPCs in a loaded cell have full processing, while the farther away an NPC is, the less processing they have done. They still get processed though, just at a lower frequency and in bigger steps (and some AI packages can just be skipped too, if it will have no appreciable effect if the player isn't around).

As a result, the engine needs the reference and cell information: e.g. if an Actor is set to seek out another NPC, a specific item, or to simply hang around in another cell, they have to know first of all where they are, as well as where the target is, which direction to go to reach it, plus whether they can or should use doors along the way or if they can just walk (or both)... and the engine has to know that for every cell they happen to traverse on their way to the target. And this can happen if the player isn't anywhere near the Actor or its target, as the player can come by days later and see that it's been long done.

The ESX format seems to be closely tied to the way the engine works with the data. If OpenMW has trouble handling the data, or is significantly altering the data before use, it's an indication that it's handling it wrong

I disagree with this statement. MW is obviously doing it wrong. MW's flakiness can't be just the result of sloppy coding. There are design issues. OpenMW has always ignored how MW works internally (or what we can guess about it) and chosen a different path.

As such, this seems to be reaching into stages of premature optimization... trying to keep otherwise accessible data inaccessible (unloaded), and applying tricks to figure out if it needs to be accessed, when indications are the ESX data gets loaded at game start and stays loaded.

This a partially agree with. Please note, that the cell loading on demand wasn't my idea. OpenMW did that already when I joined the project. At one point I proposed to load all cells on startup, but there was a lot of opposition, particularly with the possibility in mind of porting OpenMW to small portable devices later.
However during my work on fixing up script access I noticed that the performance hit of loading more/all cells on startup is very significant. That may be caused by suboptimal data structures. Or not. I did not investigate further, because it would have been time intensive. Just adding the missing functionality for script access in the way we have it now took a lot less time.
It is possible that we will drop the concept of on-demand cell loading at some point; after we have optimised everything enough that we don't get very bad performance hits from it and have tested the waters for porting to small portable devices. That would be a post-1.0 change though. I don't want to waste anymore time on something that already works in a satisfying way prior to 1.0.

My position is still that it is a bad idea to have OpenMW resolve object movement on startup (or during play). That may work with Morrowind.esm (it obviously does not work in the original MW engine), but there is serious trouble ahead for more complex content that constantly moves stuff around.

Zini wrote:It is possible that we will drop the concept of on-demand cell loading at some point; after we have optimised everything enough that we don't get very bad performance hits from it and have tested the waters for porting to small portable devices. That would be a post-1.0 change though. I don't want to waste anymore time on something that already works in a satisfying way prior to 1.0.

But it doesn't work, I thought, and that's the problem. It has trouble with the way moved references are defined, since there's no link back from the target to the source, causing an iterative search any time a new cell is loaded that contains a moved reference. That tells me it's acting the wrong way, that we shouldn't need to find the source from the target, but should instead find the target from the source.

As for the performance hit, it may be worth trying again. There was a notable improvement for the loading of NIF files after I removed Mangle's Stream/VFS bits.

My position is still that it is a bad idea to have OpenMW resolve object movement on startup (or during play). That may work with Morrowind.esm (it obviously does not work in the original MW engine), but there is serious trouble ahead for more complex content that constantly moves stuff around.

I'm not sure I follow. Aren't the MVRF and NAM0 markers how Morrowind defines a moved reference? If it doesn't work in the original engine, how do patches move objects, and how does a save game define a moved object?

But it doesn't work, I thought, and that's the problem. It has trouble with the way moved references are defined, since there's no link back from the target to the source, causing an iterative search any time a new cell is loaded that contains a moved reference.

It does work. The trouble with the way moved reference are defined is the way moved references are defined.

As for the performance hit, it may be worth trying again. There was a notable improvement for the loading of NIF files after I removed Mangle's Stream/VFS bits.

No additional resources (NIF) were loaded. This is purely about reference data in cells.

I'm not sure I follow. Aren't the MVRF and NAM0 markers how Morrowind defines a moved reference? If it doesn't work in the original engine, how do patches move objects, and how does a save game define a moved object?

Okay, I'll explain in more detail. It does work, but only under very specific conditions, which happened to be all that was used in Morrowind.esm. When you move outside of this small channel of functionality, MW becomes very unstable.
For example it is extremely easy to cause a crash by moving a NPC from one cell to another in a certain fashion. Or maybe I should say "not in a certain fashion" instead, because the situation were this works is very narrowly defined, with everything outside it failing, either partially or with a crash.

Zini wrote:It does work. The trouble with the way moved reference are defined is the way moved references are defined.

I'm not sure I see the problem with it. Some esx file adds a reference in a given cell. Another esx adds a MVRF tag to the reference, specifying the place it moves to. The cell it moves to than has a NAM0 count incremented, to say "a reference will be placed here".

As I understand it, deleting a reference causes far more problems when something tries to modify/access it since it's physically no longer there, so the reference has to stay put for it to continue working. Actually moving it is just as bad as deleting it since it's no longer where things look for it... imagine a patch that moves a reference from Cell A to Cell B, and another patch that moves the same reference from Cell A to Cell C; the second patch would not see any such reference in Cell A to move. Or worse, a save game that moves a ref from Cell A to Cell C, and an esp is added that moves the initial placement of the ref from Cell A to Cell B. The esp will move the ref to Cell B, and the save game won't find the ref in Cell A, so the ref data is completely reset to be as the esp defines it, instead of as it was in the save game.


The way I imagine it to work is, a cell contains a list of normal references, and a separate list of dynamic references (pointers to references in other cells that have been "moved" here). The MVRF tag on the ref specifies which cell has it in its dynamic reference list, and the NAM0 tag in the cell is essentially a count of the dynamic references list.

When a reference moves to another cell, it's removed from the old cell's dynamic list (if any), a MVRF tag is created or modified to point to the cell it's being moved to, and that cell then gets the reference added to its dynamic references list.

When processing references in a cell (scripts, AI, etc), it skips those that have been moved elsewhere (has a MVRF tag) and processes those in the dynamic references list.

The only real question I'd have with this method is why Beth has the NAM0 tag in the target cell. Given that moving is inherently a dynamic thing, I don't see the reason for needing to know the number of references moved into a given cell ahead of time (given the way vectors or lists grow, preallocating them won't buy much).

No additional resources (NIF) were loaded. This is purely about reference data in cells.

What I mean is the actual reading of data from the disk is faster without Mangle now, particularly lots of small reads (at least with shared streams, Mangle added a lot of overhead for the read() call).

Okay, I'll explain in more detail. It does work, but only under very specific conditions, which happened to be all that was used in Morrowind.esm. When you move outside of this small channel of functionality, MW becomes very unstable.
For example it is extremely easy to cause a crash by moving a NPC from one cell to another in a certain fashion. Or maybe I should say "not in a certain fashion" instead, because the situation were this works is very narrowly defined, with everything outside it failing, either partially or with a crash.

What conditions cause it to fail? Or I guess, what do you have to do to make it fail? A moved reference is given a MVRF tag that specifies what cell it's moved to, the cell it's moved to has a NAM0 tag to specify that it goes there. It doesn't sound like there's much that can cause problems, there.

What conditions cause it to fail? Or I guess, what do you have to do to make it fail? A moved reference is given a MVRF tag that specifies what cell it's moved to, the cell it's moved to has a NAM0 tag to specify that it goes there. It doesn't sound like there's much that can cause problems, there.

To be honest I don't remember all the nasty details, because OpenMW, some organisational problems in Redemption and other RL stuff has kept me from working on Redemption for a while. There is one case I do remember: Use PositionCell to move a NPC from a cell that has not been loaded this session into the cell where the player is. Then talk to that NPC. Result: Boom! I am not entirely sure when moving the NPC with a esp first, if for this to happen the original cell needs to be unloaded or the cell where the esp dropped the NPC. Sorry, too long ago.

The way I imagine it to work is, a cell contains a list of normal references, and a separate list of dynamic references (pointers to references in other cells that have been "moved" here). The MVRF tag on the ref specifies which cell has it in its dynamic reference list, and the NAM0 tag in the cell is essentially a count of the dynamic references list.

When a reference moves to another cell, it's removed from the old cell's dynamic list (if any), a MVRF tag is created or modified to point to the cell it's being moved to, and that cell then gets the reference added to its dynamic references list.

I don't think our positions are that different. But I just don't agree to have an object's data in a cell that the object is not in. We have a very simple world model. We shouldn't complicate it with this cell moving stuff. At least we would have to encapsulate it properly, so that the rest of OpenMW doesn't have to deal with these details. Sounds to me like a waste of time.

We need to track changes across cell borders eventually, but that is better done in a separate data structure, that only records movement of objects and not the actual object data.

Good evening, gentlemen. Hopefully I can provide some reversing insight.

Zini wrote: But it doesn't work, I thought, and that's the problem. It has trouble with the way moved references are defined, since there's no link back from the target to the source, causing an iterative search any time a new cell is loaded that contains a moved reference. That tells me it's acting the wrong way, that we shouldn't need to find the source from the target, but should instead find the target from the source.

My position is still that it is a bad idea to have OpenMW resolve object movement on startup (or during play). That may work with Morrowind.esm (it obviously does not work in the original MW engine), but there is serious trouble ahead for more complex content that constantly moves stuff around.

Everything is loaded on startup with Morrowind, at the phase before the title screen. Every reference, item name, script constants, etc. are resolved to object pointers. On loading a save, all of that plus script variables stored in the save have to be resolved too. This at least allows early warnings on missing references, spells, etc.

Chris wrote: Aren't the MVRF and NAM0 markers how Morrowind defines a moved reference? If it doesn't work in the original engine, how do patches move objects, and how does a save game define a moved object?

NAM0 is a specific marker with two roles. FRMR references that appear in a CELL record before the NAM0 marker are "temp refs", refs to objects that may migrate to another cell. The value coded in NAM0 is the number of references plus one; its purpose is to point to a safe index to add another reference. If it used any free slot, then it would be possible to revive deleted objects from another mod. In the case of a MVRF, the game keeps the reference alive in the target cell and resolves it back to the original cell on saving.

Chris wrote: The only real question I'd have with this method is why Beth has the NAM0 tag in the target cell. Given that moving is inherently a dynamic thing, I don't see the reason for needing to know the number of references moved into a given cell ahead of time (given the way vectors or lists grow, preallocating them won't buy much).

Nearly every dynamic dataset in Morrowind is a singly linked STL list. Yeah. Having the highest id stored in NAM0 also allows for multiple editors adding objects without conflict, which probably came up during development at Bethesda.

Zini wrote: I don't think our positions are that different. But I just don't agree to have an object's data in a cell that the object is not in. We have a very simple world model. We shouldn't complicate it with this cell moving stuff. At least we would have to encapsulate it properly, so that the rest of OpenMW doesn't have to deal with these details. Sounds to me like a waste of time.

The data model is designed to always handle mod conflicts, and operate at a fine granularity such that multiple mods can edit objects without stomping each others' changes. The general case is two or more mods editing a base mod object.

If the reference was just moved, two mods that move the same object would create duplicates (either two copies or two references to the original). Mods that edit part of the reference but don't move it, no longer have a simple way to find the reference to alter. Mods that delete the reference now leave a hanging reference in another cell. You may see there is a requirement for a placeholder where the original object was, at the very least.

No, I'm not dead yet, I haven't given up, and my computer hasn't blown up either. However, my RL job went nuts during the holiday season, and I had to spend over one month worth of hacking time for some extra work for the NASA (including a short trip to DC). Oh, and there seem to be some new complications with the git repository. Take all this together and you will easily get two months without any visible progress.

But let's start at the beginning. First, I have figured out why my earlier merge attempts did fail. I was actually missing a tracking branch for Zini's repository, and trying to merge with my own master branch - which is free of any changes - certainly does not do anything. So I set up the tracking branch, called git merge, got three pages worth of output, looks good. Except for the fact that my own changes were quitely overwritten by the merge, as git somehow came to the conclusion that a fast-forward merge would be possible.

What went wrong? I am not the kind of person that spams a forum because of something that seems to work for everybody else, as this usually hints towards another mistake from my side. So I tried to fix (or at least understand) it by myself. Google (plus my growing collection of links related to git) just told me that I did everything right, so I had to dig deeper. I learned quite a bit about git this way, including stuff about fast-forward/"real" merges, merhing with and without commit, and also how to use git reset/merge with a specific patch/log. That's nice to know and actually useful on some occasions, but it didn't quite help. Finally I found the problem. Looks like for each file touched by my branch, there is a separate patch in the main branch that somehow (erraneously) confuses git into believing that there is no conflict! How likely is that?

Well, whatever, so my branch has transformed into a clone of Zini's master branch. The most economic way to reintroduce my patches would be to convert my earlier commit to the new code, reapply it, and forget about this incident. After all, how difficult can this be? Quite difficult, actually. For the last four weeks, Zini's master banch seems to be missing the contents of an entire folder, or so my build system tells me. CMake says: I looked into the folder, and indeed, it is completely empty of anything relevant. Missing a CMakeLists.txt file also does not look like something done on intent. The call to make also agrees with this: so maybe there's something wrong with my local copy. Tried a fresh clone of Zini's repo, and still nothing. (Well, at least it did complain at 5%, and not at 95% - saves *some* time.) Am I doing something wrong, did I miss something, or is it just an instance of Murphy's Law in action? I am especially wondering why I seem to be the only guy to encounter this problem.

I actually hoped to have more positive things to report. I have seen that the last release added the ability to move references across cell boundaries, and I wanted to do some work on this. I wanted to join the discussion on this (and other) topics and so many other things - but without the ability to build even the unmodified code, most of what I could add would be speculative.

So, in a nutshell - I want to address the "moved references" issue that you have been discussing for the last two pages, at least by looking at the current code so I can contribute to the on-going discussion, but I'll need to fix the issue with the repository first. Anybody able to help me getting the missing header file (and whatever may be missing on top of this)?