Lua scripting in OpenMW

Everything about development and the OpenMW source code.
User avatar
AnyOldName3
Posts: 2056
Joined: 26 Nov 2015, 03:25

Re: Lua scripting in OpenMW

Post by AnyOldName3 » 10 Nov 2020, 16:17

If two scripts write to the same data and don't have some degree of negotiation between them to determine which wins, there are going to be problems whether or not they're running at the same time or one after the other. I don't believe we'd need more locking than with any other proposal.

As for dependencies resolving to a tree, if that's not possible, then the whole scripting system is impossible with any approach. It's guaranteed to be a directed graph, and they can be linearised. If there are cycles, they'd be there with any system, and there has to be aa terminating condition for the cycle otherwise you'd end up with an infinite loop and a freeze in any system.
AnyOldName3, Master of Shadows

User avatar
urm
Posts: 62
Joined: 02 Jun 2017, 16:05
Contact:

Re: Lua scripting in OpenMW

Post by urm » 10 Nov 2020, 16:36

AnyOldName3 wrote:
10 Nov 2020, 16:17
If two scripts write to the same data and don't have some degree of negotiation between them to determine which wins, there are going to be problems whether or not they're running at the same time or one after the other. I don't believe we'd need more locking than with any other proposal.

As for dependencies resolving to a tree, if that's not possible, then the whole scripting system is impossible with any approach. It's guaranteed to be a directed graph, and they can be linearised. If there are cycles, they'd be there with any system, and there has to be aa terminating condition for the cycle otherwise you'd end up with an infinite loop and a freeze in any system.
I think we might be using the term "script" differently. I think of it as a separate lua file (and probably a separate record in the .omwaddon file).
However, that file might subscribe to multiple events. So while the tree of functions to be ran is always a tree, I doubt that's a useful approach to running them sequentially. And while whatever circular dependencies between the scripts will always have a terminating condition, it might be arbitrarily complex.
The way I would expect this to work, is that during the normal OpenMW frame (or potentially multiple) we build up a queue of events, which are then processed by the Lua scripts, which have registered for them. So we don't really need a defined sequence of scripts to be processed at all.

User avatar
AnyOldName3
Posts: 2056
Joined: 26 Nov 2015, 03:25

Re: Lua scripting in OpenMW

Post by AnyOldName3 » 10 Nov 2020, 17:03

So what I'm basically proposing is that we have a thread pool that picks up the pending script functions whose events have triggered, but which haven't been run since then, and processes them, potentially in parallel if there are enough cores and multiple things that can be run at once. Pretty much every dependency relation can be modelled by having script functions only get triggered by events that set up their preconditions (either as part of the engine or other script functions) and themselves trigger events when they make changes other script functions might be interested in.
AnyOldName3, Master of Shadows

User avatar
urm
Posts: 62
Joined: 02 Jun 2017, 16:05
Contact:

Re: Lua scripting in OpenMW

Post by urm » 10 Nov 2020, 17:52

AnyOldName3 wrote:
10 Nov 2020, 17:03
So what I'm basically proposing is that we have a thread pool that picks up the pending script functions whose events have triggered, but which haven't been run since then, and processes them, potentially in parallel if there are enough cores and multiple things that can be run at once. Pretty much every dependency relation can be modelled by having script functions only get triggered by events that set up their preconditions (either as part of the engine or other script functions) and themselves trigger events when they make changes other script functions might be interested in.
Sure, that could work. A thread pool that goes through that queue and processes events in batches.
There are two more things to consider:
1. Making sure the order of events is consistent. Not sure if this is important to be honest. It needs to be consistent for the behavior to be 100% deterministic, but I doubt we can achieve it reasonably while utilizing many threads in singleplayer, or in multiplayer.
2. ptmikheev and I were discussing events returning a result. Although we can probably pause event's execution until the other event it triggered finishes, and return to it. However in that case we would need to pass values between different Lua instances, so not sure how well that would work.

User avatar
AnyOldName3
Posts: 2056
Joined: 26 Nov 2015, 03:25

Re: Lua scripting in OpenMW

Post by AnyOldName3 » 10 Nov 2020, 19:57

The sync stuff for events that return values can be fairly simple (either you trigger synchronously and block but can get something back later or trigger asynchronously and keep going but maybe get a return value from another event). If it's a built-in event, it's going to need to involve C++ anyway, so that's fairly easy as the Lua's not touching any other Lua, so worst-case scenario, we just marshal the Lua return value of a pure Lua event into C++ and then it won't have any interpreter instance specific stuff any more.

One thing to bear in mind, though, is that if events can return values, you need to strictly enforce that there's only one handler registered, otherwise results can be surprising.
AnyOldName3, Master of Shadows

User avatar
urm
Posts: 62
Joined: 02 Jun 2017, 16:05
Contact:

Re: Lua scripting in OpenMW

Post by urm » 10 Nov 2020, 20:03

AnyOldName3 wrote:
10 Nov 2020, 19:57
The sync stuff for events that return values can be fairly simple (either you trigger synchronously and block but can get something back later or trigger asynchronously and keep going but maybe get a return value from another event). If it's a built-in event, it's going to need to involve C++ anyway, so that's fairly easy as the Lua's not touching any other Lua, so worst-case scenario, we just marshal the Lua return value of a pure Lua event into C++ and then it won't have any interpreter instance specific stuff any more.

One thing to bear in mind, though, is that if events can return values, you need to strictly enforce that there's only one handler registered, otherwise results can be surprising.
Yeah I'm not sold on returns from events myself. One thing it's trying to accomplish is two-way communication with other mods, which doesn't involve two different events, as those would be very akward to use. Maybe we should just offer them as a different type of event, with the limitation of a single handler, as you are suggesting.

User avatar
AnyOldName3
Posts: 2056
Joined: 26 Nov 2015, 03:25

Re: Lua scripting in OpenMW

Post by AnyOldName3 » 10 Nov 2020, 23:49

Technically it's possible to set it up so they return an array/list/vector/set of results, i.e. one result from each handler, but that can be less obvious as you're writing a function that returns a value and when you call it it returns a collection of values.
AnyOldName3, Master of Shadows

User avatar
urm
Posts: 62
Joined: 02 Jun 2017, 16:05
Contact:

Re: Lua scripting in OpenMW

Post by urm » 11 Nov 2020, 14:17

AnyOldName3 wrote:
10 Nov 2020, 23:49
Technically it's possible to set it up so they return an array/list/vector/set of results, i.e. one result from each handler, but that can be less obvious as you're writing a function that returns a value and when you call it it returns a collection of values.
The main use case I see for events with results is when you would actually prefer to just call a function, but that function is actually running in a different script, and you don't have access to it. So I don't really how would multiple results mesh with this.
One thing I would want to handle are compatibility patches. So maybe instead of allowing a single handler, we should have some kind of a handler priority, where an event handler can pass the event further down the handler list.

Edit: another option is that all events return a result, and each handler gets access to result from the previous one. That would restrict how much we can split them into threads though.

ptmikheev
Posts: 45
Joined: 01 Jun 2020, 21:05
Gitlab profile: https://gitlab.com/ptmikheev

Re: Lua scripting in OpenMW

Post by ptmikheev » 14 Nov 2020, 18:03

I've written some ideas in a google doc. Open for comments.

Key points about threads and events:
  • Extracting the whole scripting to a separate thread is more important than multithreaded scripting. It should be safe because rendering and scripting work on different data.
  • Local scripts are always synchronized with the main thread. I.e. scripting rate is the same as frame rate.
  • In multiplayer global scripts are server-side and work asynchronous. I.e. there is no "frames" there.
  • Global scripts can not be distributed over several workers because for global scripts full access to the world state and direct communication with other global scripts is an essential functionality. So all global scripts are in a single thread. In future we can allow global scripts to start new worker threads, but such threads will not have any access to the game world and the event system.
  • Local scripts can be be distributed over several workers, but all scripts attached to the same actor should be processed by the same worker. Each worker sees the world state as it was at the beginning of the frame. All changes are synchronized by the main scripting thread after all local scripts are processed.
  • Sending an event is always a non-blocking operation that only adds the event to some queue. All local events will be delivered only after the end of the frame.
  • The engine itself (i.e. what is written in C++) doesn't support events with a result. But such functionality can be implemented as a Lua library on top of normal events.

User avatar
AnyOldName3
Posts: 2056
Joined: 26 Nov 2015, 03:25

Re: Lua scripting in OpenMW

Post by AnyOldName3 » 15 Nov 2020, 00:24

Extracting the whole scripting to a separate thread is more important than multithreaded scripting. It should be safe because rendering and scripting work on different data.
I'm not convinced this has been thought through. Rendering works on the world state after all processing has been done, so you're working on the script data for the next frame if you're running scripts in parallel with rendering. That means a frame of latency for anything involving scripts, which is bad, and probably breaks things. If you want scripts not to have latency, they need to be a distinct phase in the frame after non-script updates and before rendering.
AnyOldName3, Master of Shadows

Post Reply