Elder-scrolls IV Oblivion

Feedback on past, current, and future development.
User avatar
XJDHDR
Posts: 10
Joined: 12 Apr 2017, 07:39
Location: New Zealand
Gitlab profile: https://gitlab.com/XJDHDR
Contact:

Re: Elder-scrolls IV Oblivion

Post by XJDHDR » 10 May 2020, 21:00

cc9cii wrote:
10 May 2020, 11:37
I believe the pathgrids work the same way (both Morrowind and Oblivion), at least within the cell. How they work over long distance via roads, I don't know yet.
Would these posts help?
https://forums.nexusmods.com/index.php? ... ry71563733
https://forums.nexusmods.com/index.php? ... ry73572273

User avatar
cc9cii
Posts: 511
Joined: 28 Mar 2013, 04:01

Re: Elder-scrolls IV Oblivion

Post by cc9cii » 10 May 2020, 22:03

Hi, thanks for the links. Yes, I guess the posts help in the sense that they confirm what I was suspecting - that is, there has to be a hierarchy of pathfinding (I guess there really isn't any other way?) i.e. at a high level, path finding between cells and at a lower level pathgrid points within a cell (and then priorities amongst pathgrid points)

What I'm lacking is some of the details - e.g. how are the PGRR sub-records meant to be used? how does the game know a particular interior cell is connected to a pathgrid point outside?

I put up a short video to illustrate. You can see that the entry points to other cells do *not* have a blue pathgrid point nor any inter-cell links (orange lines). My guess is that the vanilla engine is using either the doors or teleport markers (FO3/FONV appears to have explicit portal markers which I suspect are used to link cells).

All these can be worked out eventually, or at least a workaround can be found. I guess I've not put in too much effort since the later games have all moved on to using navmesh and I'm not 100% sure whether that this implementation should do the same.

ponyrider0
Posts: 14
Joined: 11 Jun 2019, 23:53

Re: Elder-scrolls IV Oblivion

Post by ponyrider0 » 11 May 2020, 00:12

cc9cii wrote:
10 May 2020, 22:03
What I'm lacking is some of the details - e.g. how are the PGRR sub-records meant to be used? how does the game know a particular interior cell is connected to a pathgrid point outside?

I put up a short video to illustrate. You can see that the entry points to other cells do *not* have a blue pathgrid point nor any inter-cell links (orange lines). My guess is that the vanilla engine is using either the doors or teleport markers (FO3/FONV appears to have explicit portal markers which I suspect are used to link cells).
PGRR subrecords are the links between nodes in the same cell/pathgrid. So maybe you meant PGRI or PGRL? As far as I know, the PGRI (links between cells) are only used to join the border between two contiguous, physically connected exterior cells. The x,y,z position always points to a node of the adjacent exterior cell's pathgrid. I can't find any instance of them being used to link to an interior cell or a non-adjacent exterior cell. So, I think your youtube video and your github sourcecode are implementing the inter-cell pathgrid links perfectly.

Regarding the PGRL (Point to Reference) subrecords, these appear to only be used to link a node to some object within the same cell. At first, I thought they were used as hints for actors to reach the referenced object. However, based on several discussions I've read, the proper use of these records is to dynamically enable/disable a path node by passing the reference to which it is linked: https://cs.elderscrolls.com/index.php?t ... PathPoints. The use-case would be to avoid scripted traps, locked doors, etc. that dynamically change state through the course of a quest.

Based on this article: https://cs.elderscrolls.com/index.php?title=SubSpace, I believe you are correct regarding how the AI uses doors/teleport markers to figure out how to reach destinations in interior/disconnected cells.

User avatar
XJDHDR
Posts: 10
Joined: 12 Apr 2017, 07:39
Location: New Zealand
Gitlab profile: https://gitlab.com/XJDHDR
Contact:

Re: Elder-scrolls IV Oblivion

Post by XJDHDR » 11 May 2020, 01:01

My understanding of the pathgrid system (based on my observations of NPC behaviour, not based on looking at the game's code) is that NPCs do indeed use the doors and teleport markers themselves to determine how to move between interior and exterior cells. When an NPC wants to do so, the game finds the sequence of doors that take them to that new cell. For each door, the game finds the closest pathgrid node to that door. The game will then make the NPC walk from one node to another until they reach the closest node, preferring blue nodes until they stop bringing the NPC closer to the closest node then switching to red and orange. Once they reach that closest node, they then walk straight towards the door until they are close enough to activate it. Once they reach the teleport marker on the other side, they then walk to the nearest pathgrid node and continue their hop from one node to the next.
And I think this is all done at the time the AI package activates. If it was calculated at game start or inside the ESP, the system wouldn't be able to account for doors or markers being enabled, disabled or moved.

As Ponyrider said, I believe those orange links are used to connect two exterior cells together rather than connecting an interior to exterior. Said links in your video appear to correspond to the areas where the 4 IC Market exterior cells intersect with each other at their corners and sides.

And I was going to mention subspaces as well but Ponyrider beat me to it. Any pathgrid nodes inside a subspace are treated as being in a completely separate cell from the nodes outside it. I've never tried linking a node inside a subspace to one outside but assuming the editor allows it to begin with, I suspect the game would ignore the link.

Just as another piece of advice but if you have any questions about the way Oblivion's code works, it might be a good idea to get in contact with one of the OBSE devs or more renowned OBSE plugin creators, if you haven't already. Llde could be a good choice as they are still active and have contributed to OBSE.

User avatar
cc9cii
Posts: 511
Joined: 28 Mar 2013, 04:01

Re: Elder-scrolls IV Oblivion

Post by cc9cii » 11 May 2020, 01:25

ponyrider0 wrote:
11 May 2020, 00:12
PGRR subrecords are the links between nodes in the same cell/pathgrid.
Hi, I meant PGRR subrecords in ROAD records. They don't look the same to me. With regards to pathgrid implementation I believe OpenMW had it done many years ago so I don't think there'll be too many surprises.

User avatar
AnyOldName3
Posts: 1939
Joined: 26 Nov 2015, 03:25

Re: Elder-scrolls IV Oblivion

Post by AnyOldName3 » 11 May 2020, 03:25

You're not on the Skyrim SE RE discord server, are you? There's a lot of knowledge between all the people there, and even when something isn't exactly the same as the recent games, it can be good to bounce ideas around.
AnyOldName3, Master of Shadows

ponyrider0
Posts: 14
Joined: 11 Jun 2019, 23:53

Re: Elder-scrolls IV Oblivion

Post by ponyrider0 » 03 Jun 2020, 19:32

@cc9cii - Amazing work with your latest youtube videos.

Regarding the Landscape LOD performance, https://www.youtube.com/watch?v=gdcRzohlOaI&t, I have a few questions:

1. Are you using the pre-generated landscape meshes/textures (\Oblivion\Data\Meshes\Landscape\LOD\*.nif and \Oblivion\Data\Textures\LandscapeLOD\Generated\*.dds) or generating your own?

2. Similarly, are you using the pre-generated Distant Object LOD files (\Oblivion\Data\DistantLOD\*.lod) or generating your own Distant Object render list / scene graph? If generating your own, are you doing it real-time or pre-computing? Pre-computing a Distant Object render list per cell would allow you to limit the Distant Objects to a certain number of instances to improve performance. Then you could prioritize the size of the Distant Object vs distance from the cell (aka screen size of distant object from that cell) to determine which object to keep or cull.

3. Are you using separate graphics pipelines to render the Distant Lands and Distant Objects? You could probably cut out a number of graphics operations when rendering Distant Lands/Objects, ex: specular lighting, positional lighting, etc.

As you already mentioned in your youtube post, the Oblivion engine renders only a small number of distant objects; and even mods like Oblivion's RAEVWD only add 100-200 additional distant object instances into the entire worldspace. Most of the illusion of a fully populated LOD world is done via billboards of trees. Which brings me to my last idea: a good placeholder for a fully rendered speedtree is to use the speedtree billboards, these are just regular DDS files that use the scaling and coordinate information from the game records rather than any proprietary speedtree fileformat data. Here is a post with Oblivion engine being used to display non-Speedtree distant objects as billboards: https://www.patreon.com/posts/developer-2020-34313646.

User avatar
cc9cii
Posts: 511
Joined: 28 Mar 2013, 04:01

Re: Elder-scrolls IV Oblivion

Post by cc9cii » 03 Jun 2020, 22:51

ponyrider0 wrote:
03 Jun 2020, 19:32
@cc9cii - Amazing work with your latest youtube videos.

Regarding the Landscape LOD performance, https://www.youtube.com/watch?v=gdcRzohlOaI&t, I have a few questions:

1. Are you using the pre-generated landscape meshes/textures (\Oblivion\Data\Meshes\Landscape\LOD\*.nif and \Oblivion\Data\Textures\LandscapeLOD\Generated\*.dds) or generating your own?
Yes, using the pre-generated ones. I believe the CS can generate them, although I've never tried. I think I posted the naming convention in one of the earlier posts.

FO3/FONV also come with pre-generated meshes and textures but they are not in uniform size (32x32 cells) like Oblivion. They also have different naming convention.
2. Similarly, are you using the pre-generated Distant Object LOD files (\Oblivion\Data\DistantLOD\*.lod) or generating your own Distant Object render list / scene graph? If generating your own, are you doing it real-time or pre-computing? Pre-computing a Distant Object render list per cell would allow you to limit the Distant Objects to a certain number of instances to improve performance. Then you could prioritize the size of the Distant Object vs distance from the cell (aka screen size of distant object from that cell) to determine which object to keep or cull.
I was using the ".lod" files (which contain object references but without object id, so they're like spawned objects). If the object has a low-poly version (i.e. _far.nif) that is used, and no collision shapes are created. If I remember correctly the low-poly mesh also specify low-poly textures. (just checked, they do - e.g. CastleCheydinhalKeep01_far.NIF uses Architecture\\Castle\\Cheydinhal\\CheydinhalCastleLOD01.dds)

But I noticed a few of the static object references have incorrect positions and/or scale. So I changed to another way that I will describe below. For trees and any other object types - I've not decided yet.

Oblivion/FO3/FONV (and Skyrim? can't remember now) categorises each of the cell references as "persistent", "visible distant" and "temporary". The "visible distant" references match those in the ".lod" files (note: I've only checked static objects in Oblivion and none of the DLCs nor FO3/FONV).

Oblivion/FO3/FONV/Skyrim also has a way of indicating which cells have distant statics. From my notes/log:

Code: Select all

    Game     |  BSA              |  Folder     | filename
    ---------|-------------------|-------------|--------------
    Oblivion | Oblivion - Meshes | distantlod  | .cmp
    FO3/FONV | Fallout - Misc    | lodsettings | .dlodsettings
    Skyrim   | Skyrim - Meshes   | lodsettings | .lod
So, another way to get the visibly distant static object references, albeit a little less efficient, is to load the cell references into CellStore (which should have kept a preloaded Cell context) and grab the visibly distant references from the CellStores listed in the ".cmp" file.

But this way may not work so well for trees since there are so many of them and CellStore likes to index things which takes time (I thought about hacking it so that trees don't get indexed). So maybe use both methods in future? One for trees and another for statics, etc.

In terms of optimising the number of cells to grab - yes I am doing that. The video posted show distant statics from approx 20 cells from the player (also excludes 2 cells from the player, else the real mesh will overlap with the low-poly mesh but this bit needs more work to make it smoother). Again, using the "_far.nif" where possible.

Now, it is simple to extend this to show regular static objects from nearby cells, say up to distance of 5. The only thing is that these static objects won't have the low-poly mesh. I've thought about dynamically generating them and caching them somewhere (i.e. serialise to a file) but this kind of thing can wait till later. On the other hand, having the low-poly mesh pre-generated using a tool allows them to be kept in a MOD and gives the purists a choice to enjoy the vanilla experience if they wish.

I've not bothered with the object size for rendering, etc, since Ogre already does this. But I noticed that the vanilla Oblivion engine does it a little differently.
3. Are you using separate graphics pipelines to render the Distant Lands and Distant Objects? You could probably cut out a number of graphics operations when rendering Distant Lands/Objects, ex: specular lighting, positional lighting, etc.
Not doing any of that at the moment - probably something to do as performance optimisation once I get things to work properly. Actually I'm not even loading the normal maps for the landscape LOD currently (which might be the reason for them being too dark?)

I am using Ogre's static geometry for the landscape LOD which probably helps with the performance but I've not measured anything.
As you already mentioned in your youtube post, the Oblivion engine renders only a small number of distant objects; and even mods like Oblivion's RAEVWD only add 100-200 additional distant object instances into the entire worldspace. Most of the illusion of a fully populated LOD world is done via billboards of trees. Which brings me to my last idea: a good placeholder for a fully rendered speedtree is to use the speedtree billboards, these are just regular DDS files that use the scaling and coordinate information from the game records rather than any proprietary speedtree fileformat data. Here is a post with Oblivion engine being used to display non-Speedtree distant objects as billboards: https://www.patreon.com/posts/developer-2020-34313646.
Thanks for that I'll look into it.

User avatar
cc9cii
Posts: 511
Joined: 28 Mar 2013, 04:01

Re: Elder-scrolls IV Oblivion

Post by cc9cii » 03 Jun 2020, 23:20

Just noticed some discussion about script and active cells, etc. What I do is to maintain separate CellStore lists in MWWorld::Scene - active cells where everything is loaded and another where only the visible meshes (hopefully low-poly) are loaded without collision shapes. In fact I also keep yet another for persistent objects for the worldspace (aka "dummy" cell objects) which are indexed with cell grids.

1Mac
Posts: 56
Joined: 20 Feb 2019, 19:50

Re: Elder-scrolls IV Oblivion

Post by 1Mac » 05 Jun 2020, 19:52

CC, have you ever gotten in touch with Alenet? He's the developer for OblivionReloaded and knows a lot about Oblivion's engine. Currently he's experimenting with getting multithreading to work with Oblivion. It seems like you two are working in generally the same area.

Post Reply