Lua scripting in OpenMW

Everything about development and the OpenMW source code.
ptmikheev
Posts: 69
Joined: 01 Jun 2020, 21:05
Gitlab profile: https://gitlab.com/ptmikheev

Re: Lua scripting in OpenMW

Post by ptmikheev »

Progress report

A month ago I turned from discussions about Lua to the actual coding. Now it is time to talk about the progress.
Most of the main concepts (sandboxing, global/local/player scripts, handlers, events) are already implemented.
I think I've managed to build a clean and flexible basis that will be easy to extend. The code is here.
For now there is only a minimal API that I use to check interactions between local and global scripts.

Examples

I've updated the examples from my previous post to make it consistent with the part that is already implemented.
Here are also the scripts that I use for testing (and they actually work unlike the examples above): CI builds

At the moment there are only Debian CI builds.
Builds for other platforms will be available as soon as somebody adds prebuilt LuaJIT-2.1.0-beta3 to openmw-deps repo. I can barely imagine how C++ developers live in non-linux world, so I need help with it ;) .

Tests

All the stuff in components/lua is covered by tests.
Testing apps/openmw/mwlua is tricky because it requires the whole engine to run. In the future I plan to make a set of Lua scripts to test every command in Lua scripting API, and run it automatically on the example-suite data.

Todo

List of things that should be done next:
  • Saving/Loading scripts state (it will affect saves format).
  • Get a list of global scripts to run from omwgame/omwaddon (currently "test.lua" is hardcoded). Data files format will not change - I am going to reuse the same way how MWScript stores list of global scripts for auto start.
  • Extend current "minimal API" to "minimal valuable API" (just a few commands, but something useful).
  • Write documentation for Lua scripting.
When all of this is done, it will be possible to discuss merging it into master.
User avatar
AnyOldName3
Posts: 2666
Joined: 26 Nov 2015, 03:25

Re: Lua scripting in OpenMW

Post by AnyOldName3 »

I've built AMD64 LuaJIT-2.1.0-beta3 for Windows, but I'm not really sure what I'm supposed to be bundling when I package it. Normally, when I build stuff, I have a target I can build that leaves me with a nice bin, lib and include directory structure, and I zip those three up, and everything works perfectly. LuaJIT doesn't do that, though, so I've got a collection of headers and source files (where I don't know which need to be consumed by people using it as a library and which are internal), some DLLs and EXEs (I'm pretty sure you only want lua51.dll), some LIBs (you'll need lua51.lib to link to lua51.dll but there are others), and some Lua files, but far fewer than I'd expect as I was expecting some kind of standard library.

I'm going to make a guess as to which files you need, but might be wrong, especially with headers. Check what I just uploaded, please.
ptmikheev
Posts: 69
Joined: 01 Jun 2020, 21:05
Gitlab profile: https://gitlab.com/ptmikheev

Re: Lua scripting in OpenMW

Post by ptmikheev »

Thanks a lot!

These headers, lib51.lib, and lib51.dll are exactly what was needed.
*.lua files are not necessary, I don't know what they are needed for.

Now I have some compilation errors due to differences in compilers, I'll will investigate it tomorrow.
User avatar
akortunov
Posts: 899
Joined: 13 Mar 2017, 13:49
Location: Samara, Russian Federation

Re: Lua scripting in OpenMW

Post by akortunov »

Some notes:
1. Personally I do not like event names as string literals, they are error-prone:

Code: Select all

world.activeActors[i]:sendEvent("startJumping", {})
But I suppose than they are inevitable to support custom events. So it is important is to print an error message, which shows an exact location of instruction, which tries to register or send an invalid event.
2. I see an API to register handlers, but see no "mirror" API to un-register handlers (e.g. when player changes related in-game setting). Ideally it would be nice to have similar ways to do similar things.
3. Test scripts lack an example of writeable access to event data (except of "self" usage).
For instance, MWSE-Lua has a "damage" event, which get a writeable argument (amount of damage to apply), so event handler can change an amount of damage which Morrowind will apply, and it is unclear how such case will look and work with current implementation, especially in multiplayer.
User avatar
urm
Posts: 83
Joined: 02 Jun 2017, 16:05
Gitlab profile: https://gitlab.com/uramer

Re: Lua scripting in OpenMW

Post by urm »

akortunov wrote: 29 Dec 2020, 07:33 Some notes:
1. Personally I do not like event names as string literals, they are error-prone:

Code: Select all

world.activeActors[i]:sendEvent("startJumping", {})
But I suppose than they are inevitable to support custom events. So it is important is to print an error message, which shows an exact location of instruction, which tries to register or send an invalid event.
2. I see an API to register handlers, but see no "mirror" API to un-register handlers (e.g. when player changes related in-game setting). Ideally it would be nice to have similar ways to do similar things.
3. Test scripts lack an example of writeable access to event data (except of "self" usage).
For instance, MWSE-Lua has a "damage" event, which get a writeable argument (amount of damage to apply), so event handler can change an amount of damage which Morrowind will apply, and it is unclear how such case will look and work with current implementation, especially in multiplayer.
I can answer these from our previous discussions with ptmikheev. if something has changed, I'm sure he'll correct me

1. Unless we introduce a way to declare an event's existence, there isn't a way to know if a particular custom event exists or not at the time of registering a handler for it. We could also have different syntax for core engine events and custom script events, then we could display errors/warnings for invalid engine events. I'm not sure if this is necessary to be honest, at least in tes3mp scripts typos in event names weren't that big of a deal. Also I think so far it's technically possible for a script to register for an event which will only be triggered by a script that's later in the load order, in which case there is no way to check its validity regardless of what we do.
2. It is not planned to unregister event handlers. If you want to skip over a particular event, you can just do so if skip then return end. The only real downside is the performance cost of calling a few extra Lua functions, but if I recall correctly, having no way to unregister handlers allowed for some other optimization, you'd have to wait for ptmikheev to answer about that. Also keep in mind that most spammy events would be handled by local scripts, which are automatically unloaded and unregistered when the object they are attached is not in an active cell anymore.
3. The idea was to dehardcode as much game logic into Lua as possible. So ideally we would have an "attack collided with an object" event, rather than a damage event, and could then apply damage in whichever way we wanted with Lua. So likely damage logic would just be a global script listening to these events, which would calculate the damage and apply it to the object/actor/whatever. It could also generate an event for local scripts to display the stagger animation and play the hit sound.
If someone wanted to change how the damage is applied, they would just replace that script or parts of it using the interface concept.
Keep in mind that the above example if just one way to go about implementing the damage logic. It could also be done entirely with local scripts instead, or some other hybrid way, to reduce the latency of the hit effect and stagger animation in multiplayer.
User avatar
akortunov
Posts: 899
Joined: 13 Mar 2017, 13:49
Location: Samara, Russian Federation

Re: Lua scripting in OpenMW

Post by akortunov »

urm wrote: 29 Dec 2020, 13:06 Also I think so far it's technically possible for a script to register for an event which will only be triggered by a script that's later in the load order, in which case there is no way to check its validity regardless of what we do.
"Subscribe on non-existant event and hope that an another module will define it later" is not a good solution, IMO. In a common case client should not rely on 3d-party code, which may or may not be aware of specific client's needs.
urm wrote: 29 Dec 2020, 13:06 It is not planned to unregister event handlers. If you want to skip over a particular event, you can just do so if skip then return end.
Does not looks like a good solution for me as well - it may make code more messy. A common API to work with events is "subscribe on event" and "unsubscribe from event", and it would be better to have a way to do a cleanup rather than do not have it.
urm wrote: 29 Dec 2020, 13:06 The idea was to dehardcode as much game logic into Lua as possible. So ideally we would have an "attack collided with an object" event, rather than a damage event, and could then apply damage in whichever way we wanted with Lua.
It depends on if you want to make Lua optional and mandatory.
If it is aimed to be optional, you will need to spend a lot of time to write and debug a copy of all game mechanics, and after that you will have a lot of code duplication - two copies of code which does the same. Also for every change you will need to modify code in two places - in C++ implementation and in the Lua implementation, which is more error-prone (you have 2x more place for bugs to do the same job).
If it is aimed to be mandatory, you still will need to rewrite the whole game mechanics to Lua and then drop C++ implementation. The drawback is that if we ever decide to change scripting language (e.g. to Python) we will need rewrite the whole game mechanics again.
And in both cases modmakers will need to alter our scripts, and can be a real pain for modmaking (the same issue in S.T.A.L.K.E.R. mods), as for content creators (it is hard to install and combine scripts from different mods) as well for us (it will be much harder to debug game mechanics bugs).
IMO, it would be better to have larger amount of small event which do only one job per event (e.g. allow to skip collision or adjust applied damage) rather than lesser amount of large meta-events, which try to do everything.
User avatar
AnyOldName3
Posts: 2666
Joined: 26 Nov 2015, 03:25

Re: Lua scripting in OpenMW

Post by AnyOldName3 »

That sounds to me like leaving everything hardcoded if we're not replacing the hardcoded C++ with Lua.
User avatar
Capostrophic
Posts: 794
Joined: 22 Feb 2016, 20:32

Re: Lua scripting in OpenMW

Post by Capostrophic »

The drawback is that if we ever decide to change scripting language (e.g. to Python) we will need rewrite the whole game mechanics again.
That's not an actual drawback, we'd have to rewrite the game mechanics for a new scripting language either way.

I always assumed we'd replace relevant C++ parts of the mechanics with pure Lua.
User avatar
urm
Posts: 83
Joined: 02 Jun 2017, 16:05
Gitlab profile: https://gitlab.com/uramer

Re: Lua scripting in OpenMW

Post by urm »

akortunov wrote: 29 Dec 2020, 13:58 It depends on if you want to make Lua optional and mandatory.
If it is aimed to be optional, you will need to spend a lot of time to write and debug a copy of all game mechanics, and after that you will have a lot of code duplication - two copies of code which does the same. Also for every change you will need to modify code in two places - in C++ implementation and in the Lua implementation, which is more error-prone (you have 2x more place for bugs to do the same job).
If it is aimed to be mandatory, you still will need to rewrite the whole game mechanics to Lua and then drop C++ implementation. The drawback is that if we ever decide to change scripting language (e.g. to Python) we will need rewrite the whole game mechanics again.
And in both cases modmakers will need to alter our scripts, and can be a real pain for modmaking (the same issue in S.T.A.L.K.E.R. mods), as for content creators (it is hard to install and combine scripts from different mods) as well for us (it will be much harder to debug game mechanics bugs).
IMO, it would be better to have larger amount of small event which do only one job per event (e.g. allow to skip collision or adjust applied damage) rather than lesser amount of large meta-events, which try to do everything.
Regardless of what we do, unless we stick to a single language, a lot of code will be written multiple times. I think Lua is a great compromise of performance, simplicity and accessibility. The costs of having an API which supports multiple languages are very high, and interoperability between them always stays low. I could see the point of supporting different languages for OpenMW the game engine, but not so much OpenMW the way to play Morrowind.
There were some talks about adding Python scripting to OpenMW-CS, which personally I find a bit weird since we do seem somewhat commited with the Lua API for the main game. In any case, due to how the code is structured, the CS API will not share much if any code with the Lua API anyway.
ptmikheev
Posts: 69
Joined: 01 Jun 2020, 21:05
Gitlab profile: https://gitlab.com/ptmikheev

Re: Lua scripting in OpenMW

Post by ptmikheev »

Let's first define the terminology.

There are 4 separate things:
1) Engine API - Lua code calls C++ functions.
2) Engine handlers -- C++ code calls Lua functions.
3) Events -- Lua to Lua communication. One script sends, another script receives with a delay.
4) Interfaces (not implemented yet) -- Lua to Lua communication. One script uses an API, provided by another script. Both scripts should either be global scripts, or be attached to the same object. Have no delay, and unlike event handlers can return something.

To avoid ambiguity I suggest to use the word "event" only for the 3rd one. I.e. event is always a custom event.
For the same reason it is better to say either "engine handler" or "event handler" instead of just "handler". I will use just one word "handler" only if something applies both to engine handlers and to event handlers.

> 1
akortunov wrote: 1. Personally I do not like event names as string literals, they are error-prone:

Code: Select all

world.activeActors[i]:sendEvent("startJumping", {})
But I suppose than they are inevitable to support custom events. So it is important is to print an error message, which shows an exact location of instruction, which tries to register or send an invalid event.
`object:sendEvent` syntax always means Lua to Lua communication, so enums don't suit here.
If an object receives an event, but there are no handlers for this event, then a warning is logged (already implemented).

And it is quite easy to avoid string literals on the Lua side:

Code: Select all

*********** some_mod_defs.lua ***********
-- it is not attached to an object as a separate script; just a 'header' that used by other scripts as a library
return {
    startJumpingEvent = "startJumping",
    stopJumpingEvent = "stopJumping"
}

*********** some_mod.lua ****************
local defs = require('some_mod_defs.lua')

return {
    eventHandlers = {
        [defs.startJumpingEvent] = function(eventData) ... end,
        [defs.stopJumpingEvent] = function(eventData) ... end,
    }
}

*********** another_mod.lua *************
local some_mod = require('some_mod_defs.lua')

...
actor:sendEvent(some_mod.startJumpingEvent, {})
I'll add an example like this to the scripting documentation, and it is up to mod makers to decide, use this approach or live with literals.

> 2
akortunov wrote: 2. I see an API to register handlers, but see no "mirror" API to un-register handlers (e.g. when player changes related in-game setting). Ideally it would be nice to have similar ways to do similar things.
akortunov wrote: 29 Dec 2020, 13:58
urm wrote: 29 Dec 2020, 13:06 It is not planned to unregister event handlers. If you want to skip over a particular event, you can just do so if skip then return end.
Does not looks like a good solution for me as well - it may make code more messy. A common API to work with events is "subscribe on event" and "unsubscribe from event", and it would be better to have a way to do a cleanup rather than do not have it.
After detaching a local script from an object, we need to cleanup automatically the handlers it registered. If handlers can be registered or deregistered at any moment, tracing the list of handlers is hard. Also, since events are interceptable, it would add a lot of mess with calling order and debugging.
akortunov wrote: 29 Dec 2020, 13:58 "Subscribe on non-existant event and hope that an another module will define it later" is not a good solution, IMO. In a common case client should not rely on 3d-party code, which may or may not be aware of specific client's needs.
I prefer to look at it the other way around. Every script at the moment of creation defines a list of events it supports (a kind of API), and lifetime of every handler is always the same as lifetime of the script. It is not "subscribing to an event", but "registering an event". The main subject is the receiver rather than the caller.

> 3
akortunov wrote: 3. Test scripts lack an example of writeable access to event data (except of "self" usage).
Local scripts have a writeable access ONLY to "self" (except scripts attached to a player: it will also have an access to the UI and the camera). In order to change some other object, the script can send an event to this object (or to a global script; global scripts will have write access).

There are two reasons for this limitation:
- To avoid concurrent access from several threads. Local scripts of different objects can be processed by different thread (will be implemented in the future; important for performance).
- It is essential for multiplayer, because other objects can be even processed by another client.

As for global scripts, it will have write access, but I haven't implemented such commands yet.
akortunov wrote: For instance, MWSE-Lua has a "damage" event, which get a writeable argument (amount of damage to apply), so event handler can change an amount of damage which Morrowind will apply, and it is unclear how such case will look and work with current implementation, especially in multiplayer.
Calls of engine handlers (i.e. C++ to Lua calls) are not interceptable. Events and calls of a script interface (i.e. Lua to Lua communication) are interceptable.

To change the amount of damage we need dehardcoding. But the full dehardcoding is not necessary. On the first stage it is enough just to replace the C++ code that applies the damage with a call of an engine handler "applyDamage". The amount is still calculated in C++, but applied (and can be edited) by a Lua script. The script can also use the power of script interfaces to allow other scripts to intercept it.
urm wrote: So likely damage logic would just be a global script listening to these events, which would calculate the damage and apply it to the object/actor/whatever. It could also generate an event for local scripts to display the stagger animation and play the hit sound.
...
Keep in mind that the above example if just one way to go about implementing the damage logic. It could also be done entirely with local scripts instead, or some other hybrid way, to reduce the latency of the hit effect and stagger animation in multiplayer.
I think it is better to calculate and apply the damage by a local script on the object that receives the damage. It looks cleaner, and it is also important for performance, since global scripts cannot be parallelized.

P.S.
Builds for Windows now work.
Post Reply