Script Implementation Requirements

Everything about development and the OpenMW source code.
User avatar
Site Admin
Posts: 1193
Joined: 05 Aug 2011, 22:21
Location: Wroclaw, Poland

Script Implementation Requirements

Post by lgromanowski » 13 Aug 2011, 16:29

Zini wrote: We recently had a couple of people asking about participating in the development of OpenMW. But the current roadmap offers little opportunities and someone who is completely new to OpenMW probably doesn't have the necessary background to pick a task by him/herself.

So I thought I should make a list of tasks, that would be beneficial to the script implementation process, i.e. that allow me to implement/test new script features.

1. Render the NPCs.
Currently they are not rendered, because NPCs are more complex than ordinary references and the code is not complete yet. I don't need a full implementation. Just something, that can be interacted with.

2. Activate references.
When you focus a reference in the middle of the screen and press the activation key (Space), the game interacts with this reference. Also, the act of focussing alone brings up a floating text label, if the reference has a name.
This feature probably needs bullet-integration to be implemented first.

3. Mouse picking
While the in-game console is up, you can click on any reference in the scene. This reference will be used as implicit reference for script instructions entered into the console.

4. Support for multiple master/plugin files.
Currently we can only have one master file, which reduces the scope of things we can test quite a bit. This involves the infamous 255-plugin limit and the ability to change plugins without having to start a new game. I have some ideas on how to implement this. We can discuss it in the forum, if someone is interested.

Edit: Right now is not a good time to take up #4. I am working on related code, which most likely would result in a very nasty merge procedure.

5. Load all cell data at startup.
Currently cell data (the list of references, not the actual models) are only loaded on demand. That won't work, because the scripting language can access references outside the active cells. In fact a specific script construct requires searching the references of every single cell.

6. Fix the resources file handling.
Currently resources can only be loaded from bsa archives. That is okay for Morrowind/Tribunal/Bloodmoon. But not if we want a wider scope of testing (e.g. Redemption).
I see two problems with the current implementation:
- The case of resources names in the esm/esp files and the case of directory names on disc doesn't match (e.g. one starts with lower case, the other one with upper). This isn't a problem on Windows, but it breaks things on Linux.
- Morrowind uses a directory structure. File A/x.nif is different from file A/B/x.nif. Ogre on the hand has a a flat resources structure. (not 100% sure if this has been addressed already).


7. We have a GUI now, but most windows aren't implemented yet. That should be a rather easy task. Most useful at this point would be having the windows involved in character creation.
nicolay wrote: Little late in answering this, but I think this is a great list. I've linked it from the Roadmap itself.

A couple of questions about the list itself:

Nr 5: Is this true? I wasn't aware of this. In what kind of cases are references used outside their cells? Is it eg. just or NPCs or for other kinds of objects too? Keeping every cell in entirety in memory seems like overkill to me - those things are pretty huge. Could you give some details on the script construct you mentioned?

Nr 6: I thought just adding resource paths to Ogre will load resources just fine from the file system, or am I misunderstanding?
Zini wrote: #5: You can reference an ID via the -> operator and Morrowind will use the first reference for this ID that it can find, no matter what cell it is in. This feature is used in the startup script, the very moment you start a new game.

Edit: You could be right about it being only creatures and NPCs though. At least I can't find a different example. But even if that is true, does it do us any good?
Regarding wasting memory, I don't think that is a problem. How many references do we have in MW? A couple of 10000? With each maybe 1K long (at most, probably less). That makes at most a few dozen megabyte. Even if we are aiming for low spec boxes, that hardly matters anymore today.
That only problem I see is building the data structures, i.e. finding the IDs for each reference. But if that proves to be too slow, we can still think of some lazy evaluation strategy to speed it up.

#6: Actually, I might be wrong about the directory structure. But from what I remember Ogre really does use a flat file system (i.e. it knows it files only by leaf-name). I guess the only way to be sure is to try it out. But before we can do that, we have to get rid of the case issue, which definitely is a problem.
Zini wrote: #5: Okay, my last comment about it maybe being only NPCs and creatures was stupid. We use this feature in Redemption too even on activators. And it does work, at least if the inactive cell is nearby and was recently active (probably in a few other cases too).
nicolay wrote: Ok. Couple more questions: what do you actually DO with these references, what are they used for? Also I assume the references are named by id? (In which case they aren't technically referring to objects in cells but rather to the object type itself?)
ElderScroller wrote: I'm not sure if this will help, but there seems to be a running together of terms here. Object ID refers to Morrowind's internal name for each object in the game world. Once loaded, the game creates/loads a unique reference for each object (I think the default is for objects in the player's current cell and surrounding cells, but it may be different for people who have changed the Interior Cell Buffer and/or Exterior Cell Buffer settings in their morrowind.ini file; either way, I believe it's the cells already loaded in memory that determine whether an object can be "found" or not). That is to say, it loads the entire list of "object id's" from the master/plugin file list but only those references the player has accumulated.

As an example, we could look at Fargoth:
Name: "Fargoth"
Object ID: "fargoth"
Reference: "fargoth00000000"

The above information means that hovering the crosshair over him displays "Fargoth," scripting functions in vanilla Morrowind use "fargoth" to identify him, and since he's the first object Morrowind could find with that object id, he's been assigned "fargoth00000000" as a reference. If we used the console to place 2 additional copies of Fargoth in front of the player, each would display "Fargoth" and have "fargoth" as its object id, but each would have different references to distinguish them ("fargoth00000001" and "fargoth00000002").

For vanilla Morrowind, references are used internally and not explicitly by scripts for calling a function (the "->" operator is as close as you'll get to seeing a reference in vanilla Morrowind scripting). So, if I'm standing in Seyda Neen and wanted to set Fargoth's strength to 0, I would type,
fargoth->SetStrength 0
and the strength of the first "fargoth" the game could find, "fargoth00000000," would reduce to 0 while any others would remain the same.

It's not just npc's, creatures, and activators that have an object id. Just about anything accessible in the construction set does. Those objects the player encounters also have a unique reference stored in the player's saved game (so long as the object referred to has not been deleted/excluded implicitly by the game's hardcoding or explicitly deleted with the script function SetDelete; also, I believe use of the function DontSaveObject prevents the game from storing a reference to it, though I'm not sure if that's technically correct or if it does something else with a similar end result).

Unfortunately, relying on the object id alone has limitations. As with the example above with Fargoth, there might be more than one object with the same object id. Or, the modder might try to script something where we don't know the object id of what we want our script to target.

MWSE (Morrowind Script Extender) has helped overcome some of these and other hurdles, though the extended functions it offers are not without their own quirks and there are still functions modders would love to see added.
Zini wrote: Yes, I am aware of the situation and the terminology. But what I am not sure about is this part
I believe it's the cells already loaded in memory that determine whether an object can be "found" or not).
One of the startup scripts definitely uses creature IDs in cells not loaded yet.

btw. my numbers are wrong. By a whole magnitude of order. Vanilla MW has about ten times as many references as I thought. But on the other hand the data structure has only about a tenth of the size I expected, so this mistake does not affect my argumentation.

I noticed, that the CellRef class is rather wasteful. It has entries for all types of IDs, while most types use a fraction of them. We could reduce the CellRef data structure by another order of magnitude (on average) if we take advantage of this fact.
pogzy wrote: Hi all,

Sorry to be away for the project for some time. I've got a new job and had some (still) hard work time...

It may be useful to know really how many reference the game data actually contains. Is it possible to do some stats or anything that could give us a precise idea? If you need to access data from any reference within a script, you need at least a file pointer (direct access to the file position where is stored the data of the reference, or a ptr to the cell that contains the searched reference).
Reference name + pointer (file or cell) x number of reference (maybe 10000, may be much more or much less) = I do like to know :?: :?:

May be the script engine could have a dedicated mechanism for this usage, at startup read all references and their data file position (maybe it could be pre-computed and just read at startup) then store them in memory in a dedicated area (table, hash table ...)
Do we need the complete load of the cell or just load the data of the searched reference is enough? Is there the need to have a reference cache or some mechanism that preload data reference before they are needed?

I suppose cells are loaded or will be loaded before they are needed to avoid lags. Does a similar system for scripts reference is needed to avoid speed fall of.

It is just pure speculation from me, feel free to disagree or garbage my post.
Zini wrote: I suggest you read the post from ElderScroller again, because it describes the situation very well. Sorry, but I don't think what you propose makes much sense. If we store a file pointer for each reference, then we can as well store the whole reference in memory, since they are pretty small to begin with.

There are about 300000 references in vanilla Morrowind, most of them statics, so they have almost no data (except for position, orientation and probably scale).
Greendogo wrote: If it ends up making an impact how many references there are, even if they are small in size, it would be more beneficial in the long run to plan for a much larger number of references to compensate for a large number of references added through many small mods or a few large mods (such as Tamriel Rebuilt or one of the other large land-mass mods). If you planned for a number of references ten times the original amount it would probably be enough to compensate for the next ten years worth of large land-mass mods and many many small mods.
Zini wrote: With ten times as much as references as Morrowind we probably would end up with something around 100 MB, which is almost a trivial amount of memory, unless we are talking about ultra low spec boxes. I don't see any problem here.
Zini wrote: btw. I added another point to the list.
nicolay wrote: Ok. I'm still not totally convinced that this is how Morrowind does it, but I may be completely wrong about that. (I'm pretty sure the editor loads everything though.)

In any case, if we go this route there's room for some optimizations. The most obvious one is to skip the statics but store everything else, since they are by far the most common entry. (EDIT: unless the scripts access statics too? I vaguely remember one place in Tribunal where a rock wall is moved by a script.) And of course there's no point in loading path grids and landscape data until the cell is needed. Another optimization that can combine with this is to specialize CellRef as you suggested.

A third option is to do lazy evaluation, which you also hinted at. But that is pretty complicated, unless all the script actions using references are one-way (so they can be stored and applied later, ie. they don't require any immediate data to be returned to the script.)

In any case, the first rule of all optimization is "don't do it (yet)" :) I'm sure loading all the cells isn't that much of a burden for the time being. It should only require a few changes to esm_store/.
Zini wrote: Actually I am pretty sure that MW doesn't do it the way I suggested. MW seems to use a rather messy collection of special case implementations, that I have no idea of how they are working. Makes it really unpredictable to work with from a scripters point of view and would be a bitch to re-implement.

And yes, scripts can access statics.

btw. esm_store? I think mwworld/world would need to be modified instead. AFAIR the esm_store itself doesn't store the reference lists. It provides a class for it, which is used in mwworld/world. I could be wrong about that though.
nicolay wrote: Yes, I was thinking about adding more storage in CellList. But it should work just as well (or better) in mwworld, keeping the changes to the apps/ directory is a good idea.

Scripts moving statics around really kills another optimization opportunity too. Ogre has a StaticGeometry class, to which you could add all the scene nodes that make up the static geometry of the world. It takes all the meshes you give it and recreates them into one single object, that will (presumably) render in a much more optimized fashion. But if things can move around then that won't work.
ape wrote: I'm interested in the npc task (No 1).

But how do i start?

I thought of an class which contains npc data, for now i planned to implement the names of body parts meshes. (which i get from race and female flag from esm). That object is given to the CellRenderer which renders the meshes (new method: insertNPC or something).

But how do i concatenate these parts together?

How do i get the reference type (eg. NPC, Armor ... ) from a LiveCellRef Object?

If someone has some suggestions on implementing this task, please tell me.
Zini wrote: Take a look at openmw/apps/openmw/mrrender/cellimp.cpp.

Basically you have to specialise the insertObj function for ESM::LiveCellRef<ESM::NPC, MWWorld::RedData>. This part isn't very OOP, because Nico decided to have the basic esm objects as dumb structs instead of full classes (so they can be re-used for the editor). So making a new class for NPC rendering wouldn't fit in very well.

All the other data you need, you will get from the ESMS::ESMStore. Unfortunately this is a situation I didn't consider when writing cellimp. So, what you should do now is to add a const reference to the ESM store to the argument list of every function in cellimp.
Zini wrote: Discussion about NPC rendering continues here:
ape wrote: A question to task 2.

I've managed to get the name of an ogre MoveableObject after pressing space.

How do i get the LiveCellRef of this object by knowing only the ogre node name?
Zini wrote: That is complicated and it will change a bit once my poly branch is complete. Don't even try. For now get the handle (the ogre ID of the scene node, the movable object is attached to) and print it to std::cout for testing purpose. I will take it from there for a while.
Zini wrote: Edit: My guess about the mangle update requirements are wrong, apparently (removed text requesting Nico to make some changes).

@ape: For this feature I see the following sub-roadmap:

- a ray-casting function (or whatever you are using) to find out the handle of the object the player is looking at (you)
- an entry in the engine framelistener function, that calls this function every 10 frames or so and prints the handle to std::cout (you)
- getting an object pointer and a name for the object (me)
- show a label with this name slightly above the object; probably with MyGUI (you, optional)
- add a private member function to Engine, that is called whenever the player presses space (you)
- implement the actual activation-mechanism (me)

Is that okay for you?
ape wrote: Yeah, i will do that :).

I implemented the activate action in the imput manager. It prints out the nearest object by pressing space, but i can't properly test it because i can't match the ogre names to the mw objects.

I also think that the rayscene code is better placed in openengine or elsewhere than in the inputmanager. What would you suggest?
Zini wrote: The actual function for getting the handle of the focussed object could go into apps/openmw/mwrender/mwscene.

The rest should be placed in apps/openmw/engine for now.

Nico can pull parts into OpenEngine if he feels like it. But for now not having the code in a sub-module will make testing easier.
Zini wrote: Okay, I added the name lookup to your code. Please merge my fork before you continue with implementing this feature! Works nicely, btw.
ape wrote: good job :)
ape wrote: I noticed some strange invisible static objects that sometimes are between the objects that meant to be focused. The problem is that they can be walls and if i am completely ignoring them objects could be focused through them.

Here are two screenshots which are demonstrating the problem:

Static Object between the focused object
A little closer it's working

Do you know what these objects could be and how to ignore them?
Zini wrote: No idea what is going on. But I am not sure, that I can follow what you are doing. I suggest you simply continue with the code I started (a name tag, if !name.empty(), else no name tag). If it isn't perfect, it hardly matters at this point. After all Ogre raycasts only work with axis-aligned bounding boxes. That is most likely not exact enough for our purpose, but we can address this issue later.
ape wrote: Alright, my part seems to be done for now (except the mygui part which i'll eventually look into later).
Zini wrote: Activation is implemented now. Kinda ...

- Object picking is currently using an AABB-based raycasting algorithm. This is not precise enough and makes some object inaccessible. We have to see, if Bullet can offer a better solution. If it doesn't, we will have to implement a per-surface picking algorithm (*ugh*).

- The activation actions for most objects aren't implemented yet. The only thing that works right now are teleport doors, i.e. you can change from one interior cell to another by focussing the door and pressing space. But don't try this on a door, that leads to an exterior cell.

- The position after using a teleport door is wrong. This is a bug. Haven't looked into it yet. Also, the orientation is wrong (not implemented).
best regards,