Lua scripting in OpenMW

[Chris]

Dehardcoding

Dehardcoding means reimplementing game mechanics (which are currently implemented in C++) as built-in Lua mods.
Now it is one of the main priorities for development.

Not quite. Dehardcoding means supplying means to override default behaviors. This does not necessarily mean "reimplementing game mechanics ... as built-in Lua mods" that user mods can then replace. Game mechanics can be implemented as Lua or C++ or whatever is best, but to have the code look and go "if there's a mod hooking into this functionality, call that along side this bit of built-in code". It's a subtle but important difference. A "built-in Lua mod" could be completely overwritten by a user mod, causing any future changes or additions to the built-in code to be lost until and unless the mod incorporates the changes and additions itself. Whereas by providing hooks from an internal subsystem, we can still alter or add to built-in functionality to work with those existing hooks (so the hooks remain available and working, even as the subsystem changes and gets new features).

This way also means better compatibility, since future changes just need to ensure the hooks can do the work they're expected to, rather than interface redesigns causing previous "built-in Lua mod" replacers to no longer function. I can't imagine anything more aggravating than for a modding-focused engine to continually break mods because of internal changes, and require those mods to be updated before they can work on updated versions of the engine. This may mean starting small, since you'll only want to expose things that we can know will be supported forever, but it can grow with time in a way that both ensures continued improvements to the engine and better fitting with what modders need to get the results they want.

Public API and temporary internal API

Suppose that there is an important C++ function actorDoSomething(ptr) and we want this functionality to be available from Lua as soon as possible. But in future we plan to dehardcode this function and need a way to do it without breaking mods.

Then the steps are:

1. Design an interface that will still be relevant after dehardcoding.
   In our example we may decide that after dehardcoding "actorDoSomething" will be implemented in a builtin script 'basic_ai.lua' and will be available in local scripts via a script interface as interfaces.BasicAI.doSomething().
2. Add an internal function self:_doSomething() to the Lua package "openmw.self". This function doesn't need documentation and shouldn't be used by modders.
3. Create 'basic_ai.lua' and implement interfaces.BasicAI.doSomething() as a simple wrapper around self:_doSomething(). It is public interface that can be used in mods.
4. Later during dehardcoding we can remove self:_doSomething() and replace the wrapper with a real implementation without changing the public interface.

The problem here is that dehardcoding and exposing functionality that is currently hardcoded is going to depend heavily on the functionality in question. These two points, for instance:

1. Add an internal function self:_doSomething() to the Lua package "openmw.self". This function doesn't need documentation and shouldn't be used by modders.
2. Create 'basic_ai.lua' and implement interfaces.BasicAI.doSomething() as a simple wrapper around self:_doSomething(). It is public interface that can be used in mods.

Doesn't make much sense. Internal functions are only needed where they deal with internal data, which may change in future updates. If there's going to be a public function, it necessarily won't be the same as the internal function; if anything, a public function would use perhaps several internal functions to implement its functionality with additional logic. Or it may implement the functionality directly, depending on whether the data it needs is already public. But having a public interfaces.BasicAI.doSomething() as a simple wrapper around self:_doSomething() doesn't make much sense, since the whole point of an internal _doSomething() is that it can change, possibly drastically, without ever affecting the behavior of the public doSomething().

There isn't going to be a one-size-fits-all path to dehardcoding internal functionality. How some functionality gets exposed to mods is going to depend heavily on the functionality in question.

All functions and engine handlers with prefix "_" are internal API that shouldn't be used outside of built-in scripts. But note that it is a temporary thing. At the end there will be no internal API at all and built-in scripts will not have any "special powers" comparing to normal mods.

This is very much debatable. The engine isn't going to be a collection of Lua scripts that mods can replace as they want, and leave the engine as a glorified Lua script manager. The engine is going to be focused on playing a certain type of game, and the way it interfaces with subsystems and built-in scripts will necessarily change as it improves to get new features, but it's imperative that modded content continues to work regardless of what may be changed internally. As such, there must be internal APIs and built-in scripts that can access them, but which mods cannot. We need to be able to ensure new features can be added and work along side existing content, minimizing the amount of "gotchas", or "sorry, that new engine feature can't work with that mod", and doing everything reasonable to avoid "sorry, that mod doesn't work with newer versions of the engine. stay on the older version to use it".

[dmbaturin]

As long as Lua scripts can be stored together with other assets in mod directories, and I don't have to register them by hand in openmw.cfg, I'd be ok with it.

[AnyOldName3]

People cope just fine installing Skyrim mods and they keep their scripts in separate files. Depending on the mod, some are loose files, and some are in the BSA. I don't think there's any need to worry about scripts being in separate files, but I'd prefer if they were registered from within a content file, again, like Skyrim.

Not quite. Dehardcoding means supplying means to override default behaviors. This does not necessarily mean "reimplementing game mechanics ... as built-in Lua mods" that user mods can then replace. Game mechanics can be implemented as Lua or C++ or whatever is best, but to have the code look and go "if there's a mod hooking into this functionality, call that along side this bit of built-in code". It's a subtle but important difference. A "built-in Lua mod" could be completely overwritten by a user mod, causing any future changes or additions to the built-in code to be lost until and unless the mod incorporates the changes and additions itself. Whereas by providing hooks from an internal subsystem, we can still alter or add to built-in functionality to work with those existing hooks (so the hooks remain available and working, even as the subsystem changes and gets new features).

This way also means better compatibility, since future changes just need to ensure the hooks can do the work they're expected to, rather than interface redesigns causing previous "built-in Lua mod" replacers to no longer function. I can't imagine anything more aggravating than for a modding-focused engine to continually break mods because of internal changes, and require those mods to be updated before they can work on updated versions of the engine. This may mean starting small, since you'll only want to expose things that we can know will be supported forever, but it can grow with time in a way that both ensures continued improvements to the engine and better fitting with what modders need to get the results they want.

I don't think the distinction here is quite as meaningful as you make out. The behaviour you describe with the base mechanics implemented in C++ and multiple bits of surface being available to attach mod stuff to could also be implemented in Lua, but then there'd be the added advantage that a mod author wanting to make a small tweak to something would have the reference implementation they were starting with available in the language they were going to be writing the edited version in.

The problem here is that dehardcoding and exposing functionality that is currently hardcoded is going to depend heavily on the functionality in question. These two points, for instance:

Add an internal function self:_doSomething() to the Lua package "openmw.self". This function doesn't need documentation and shouldn't be used by modders.
Create 'basic_ai.lua' and implement interfaces.BasicAI.doSomething() as a simple wrapper around self:_doSomething(). It is public interface that can be used in mods.

Doesn't make much sense. Internal functions are only needed where they deal with internal data, which may change in future updates. If there's going to be a public function, it necessarily won't be the same as the internal function; if anything, a public function would use perhaps several internal functions to implement its functionality with additional logic. Or it may implement the functionality directly, depending on whether the data it needs is already public. But having a public interfaces.BasicAI.doSomething() as a simple wrapper around self:_doSomething() doesn't make much sense, since the whole point of an internal _doSomething() is that it can change, possibly drastically, without ever affecting the behavior of the public doSomething().

I think the point of the internal-function-matches-the-public-function idea is that if we ditch the internal function later, as long as we replace it with building blocks that could be used to reimplement it, we can reimplement the public function in terms of the new stuff. If we go for a contrived example, we could say the engine has a function called doubleNumber that doubles a number, and we expose it directly as openmw.self._doubleNumber and then have a public function interfaces.maths.doubleNumber, that lets mods double numbers. If we later decide it's more sensible to expose a deeper engine function, multiply, we can remove openmw.self._doubleNumber, add openmw.self._multiply and interfaces.maths.multiply, and reimplement interfaces.maths.doubleNumber using interfaces.maths.multiply with no impact on existing mods.

This is very much debatable. The engine isn't going to be a collection of Lua scripts that mods can replace as they want, and leave the engine as a glorified Lua script manager. The engine is going to be focused on playing a certain type of game, and the way it interfaces with subsystems and built-in scripts will necessarily change as it improves to get new features, but it's imperative that modded content continues to work regardless of what may be changed internally. As such, there must be internal APIs and built-in scripts that can access them, but which mods cannot. We need to be able to ensure new features can be added and work alongside existing content, minimizing the amount of "gotchas", or "sorry, that new engine feature can't work with that mod", and doing everything reasonable to avoid "sorry, that mod doesn't work with newer versions of the engine. stay on the older version to use it".

There's going to be behaviour that we don't want in the engine, but do want available as a mod, that will tightly integrate with the engine. If there're things people want, and they can only be added with a fork, we'll get a fork, and that's worse than a tightly coupled mod. People have made mods for the existing BGS games that require extensive new reverse engineering work with each game update (although it's less than it used to be as Meh made a cool IDA script to match offsets between different versions of the same executable, but still, it's without source access, so way harder than anything anyone will need to do with OpenMW), so there'll be a few people willing to make things with no expectation of forwards compatibility, and they'll know who they are and what they're doing when they do it. We should make every reasonable effort to ensure compatibility, but there's a point at which the effort stops being reasonable and does more harm than good. It's not like we'd go as far as to guarantee there'd be no observable behaviour differences going forward, for example, as that would basically stop all updates forever, but a mod that goes against our recommendations might end up on any arbitrary bit of observable behaviour, especially if it does something like attempt to rowhammer a bit to flip to change a bool when we don't want it to.

[Chris]

I don't think the distinction here is quite as meaningful as you make out. The behaviour you describe with the base mechanics implemented in C++ and multiple bits of surface being available to attach mod stuff to could also be implemented in Lua, but then there'd be the added advantage that a mod author wanting to make a small tweak to something would have the reference implementation they were starting with available in the language they were going to be writing the edited version in.

Some bits being implemented in Lua that can essentially call out to mods is what I was getting at. Internally, mechanics can be written in C++ or Lua or whatever makes the most sense to, but the interface mods get wouldn't be a reflection of the internal interface that built-in scripts work on; it wouldn't make sense, since built-in scripts have access to much more than mods should (as they can work with more volatile interfaces, or use data that mods shouldn't have access to).

I think the point of the internal-function-matches-the-public-function idea is that if we ditch the internal function later, as long as we replace it with building blocks that could be used to reimplement it, we can reimplement the public function in terms of the new stuff.

Right, that's what I meant by 'a public function would use perhaps several internal functions to implement its functionality with additional logic', it wouldn't be a public foo() as a simple wrapper around _foo() anymore. I also don't think having a public function be a simple wrapper around a private one is necessarily a good initial design for a new function. A public scripting function shouldn't be introduced until we have a problem (feature request) it's needed to solve, at which point we should already know how to implement it, no thin wrapper needed. Internal functions are only necessary where built-in scripts need internal data (e.g. connected peripherals and their state, screen resolution) or special access to things mods shouldn't (e.g. disk access), they shouldn't be what a public function strives to be.

If we go for a contrived example, we could say the engine has a function called doubleNumber that doubles a number, and we expose it directly as openmw.self._doubleNumber and then have a public function interfaces.maths.doubleNumber, that lets mods double numbers.

In this instance, I don't see the point in the openmw.self._doubleNumber internal interface. Couldn't interfaces.maths.doubleNumber be defined to directly call the engine's doubleNumber function? Then in the future, when the engine gets enhanced with more general multiply functionality and exposes interfaces.maths.multiply, interfaces.maths.doubleNumber could be redefined to use interfaces.maths.multiply. The internal openmw.self._doubleNumber and openmw.self._multiply don't seem necessary if they're just a thin wrapper.

There's going to be behaviour that we don't want in the engine, but do want available as a mod, that will tightly integrate with the engine. If there're things people want, and they can only be added with a fork, we'll get a fork, and that's worse than a tightly coupled mod.

If it's behavior we want to support, we can work on supporting it in a way that ensures compatibility. Forks are inevitable (we've already had some), and we shouldn't be afraid of them to the point of making the engine hostile to mod compatibility. Allowing mods to tightly integrate with the engine, to the point of making version-specific mods, will result in more fragmentation due to different people needing to stay with different versions for different mods, than a few forks who get a few flashy new features, while we take out time to get similar functionality with better guaranteed future compatibility. And even in the case where a fork comes out and implements something in a compatible way that OpenMW should be able to support... we can, then mods made for that fork and its new feature will work in OpenMW.

We should make every reasonable effort to ensure compatibility, but there's a point at which the effort stops being reasonable and does more harm than good.

That's basically what I'm aiming for. The issues I responded to in this post, I'd have thought, are reasonable steps to ensure compatibility (i.e. not moving everything to be "built-in Lua mods" that mods can replace willy-nilly, and not allowing modded scripts to do everything a built-in script could), and pointing out that some of the ideas presented are a bit too general (making a plan to dehardcode any given function, when the way to dehardcode something will very much depend on the behavior being dehardcoded).

Essentially, I'm saying dehardcoding doesn't have to come in the form of "turn C++ functions into Lua functions/scripts that mods can replace", or that "in the end there will be no internal API at all and built-in scripts will not have any "special powers" comparing to normal mods". I don't think it's an unreasonable effort to temper that expectation. There will always be internal APIs that built-in scripts can use and mods can't (unless we want to limit what scripts can do, so that built-in scripts can only do the things modded scripts can, which I doubt was the intent), and that modded functionality won't always (if not will rarely) come in the form of replacing built-in scripts, compared to utilizing hooks provided by built-in scripts and various engine components.

It's not like we'd go as far as to guarantee there'd be no observable behaviour differences going forward, for example, as that would basically stop all updates forever, but a mod that goes against our recommendations might end up on any arbitrary bit of observable behaviour, especially if it does something like attempt to rowhammer a bit to flip to change a bool when we don't want it to.

Sure, which is why I always specified in previous discussions "a properly written/made mod, following the interface specification we've set out" should remain compatible. Obviously if a mod does something that causes unintended behavior (rowhammer a bit to flip), and relies on that behavior, it's doesn't fall in that category, so an engine update that causes something else to happen that breaks the mod is out of our hands. At the same time I acknowledge that we're only human, we make mistakes, so there will be the rare occasion that what we thought we could provide to modders ends up being done badly, and will require us to break that functionality (along with properly made mods that used it). But we shouldn't set ourselves up to be in a situation where that's likely to happen.

[FiftyTifty]

Just going to chime in for a moment.

A public scripting function shouldn't be introduced until we have a problem (feature request) it's needed to solve, at which point we should already know how to implement it, no thin wrapper needed.

All sorts of functions should be added pre-emptively, so we have a large toolbox to begin with. Sure, more functions should be added over time, but there's no reason to skimp on obvious functions. A great reference would be the SKSE function docs, as well as the docs for various SKSE extension plugins, to get an idea of the wide variety of functions that modders will make use of.

It's a chicken and egg situation, and OpenMW is the chicken.

[wazabear]

Just going to chime in for a moment.

A public scripting function shouldn't be introduced until we have a problem (feature request) it's needed to solve, at which point we should already know how to implement it, no thin wrapper needed.

All sorts of functions should be added pre-emptively, so we have a large toolbox to begin with. Sure, more functions should be added over time, but there's no reason to skimp on obvious functions. A great reference would be the SKSE function docs, as well as the docs for various SKSE extension plugins, to get an idea of the wide variety of functions that modders will make use of.

It's a chicken and egg situation, and OpenMW is the chicken.

Agreed, though I think the easy solution is to carefully curate a list of must-haves for this toolbox and make issues for them on the Gitlab tracker.

Personally, this is what I see is needed before I can think about making any mods. Of course this list is not suggestive at all, it's literally just what I require. This is nowhere near an exhaustive list, that is well beyond where we are right now.

1. per player and per mod configuration with graphical interface
2. per player and per mod persistent storage
3. ray casting
4. openmw.weather for weather control and queries
5. openmw.audio for playing sounds and streaming music

[AnyOldName3]

Forks are inevitable (we've already had some), and we shouldn't be afraid of them to the point of making the engine hostile to mod compatibility.

We've had two meaningful forks. One was basically a personal tech demo to experiment with loading later games' assets, and now has its developer working on (among other things) porting things to upstream OpenMW (which would have happened years ago had the OGRE->OSG migration not blocked it) and another was another volatile experimental branch looking into multiplayer, and has its developer working on porting things to upstream OpenMW. Neither were mods, neither were expected to work with the other, both were too experimental to do on our main branch, and both were slated to eventually become built-in.

Forks that are mods don't have these features. Two low-level-behaviour-manipulating features could be relatively small and fully polished, too controversial to merge into the engine, and theoretically compatible, but if, to the vast majority of our users who don't build from source, they're available only as separate binaries, they can't be used together.

[C3pa]

There was a mention of SKSE here before. From what I have dug out SKSE hasn't got any real documentation on their page, but creationkit.com has some of the SKSE functions documented under Papyrus scripting. Here are the links:

Scripting
Events

MWSE can also be used as a reference. Here are their docs. (Not everything is documented yet):

APIs
Events

[Chris]

All sorts of functions should be added pre-emptively, so we have a large toolbox to begin with. Sure, more functions should be added over time, but there's no reason to skimp on obvious functions.

Yes. In fact, one of my sticking points in an earlier MR was on feeling any attempt to dehardcode now was premature due to Lua scripting not even being a capable substitute for vanilla scripting yet. Getting Lua to a point where it can be a viable full alternative to MWScript would be a better initial focus, and we can add on from there. Dehardcoding would come naturally as modding capabilities are expanded over time, rather than as a result of making internal functions public.

We've had two meaningful forks. One was basically a personal tech demo to experiment with loading later games' assets, and now has its developer working on (among other things) porting things to upstream OpenMW (which would have happened years ago had the OGRE->OSG migration not blocked it) and another was another volatile experimental branch looking into multiplayer, and has its developer working on porting things to upstream OpenMW. Neither were mods, neither were expected to work with the other, both were too experimental to do on our main branch, and both were slated to eventually become built-in.

Exactly. Some people wanted to add features to OpenMW which were deemed too early to do ourselves because it'd hinder development and be too volatile, so the codebase was forked to add the features separately because people really wanted them anyway. But it's not as if we remained ignorant of what people wanted or didn't want to do it, so we continued to work on getting into a state where those features could be added in with less disruption to the codebase, working with the people who made those forks where possible.

If someone forks the engine to add a feature, whether it's for an advanced mod feature or for people to play multiplayer or for supporting newer game content, that we think is too early and/or too volatile to do yet, I don't see the difference.

[AnyOldName3]

A screenshot posted to Discord today reminded me that we already have an example of the kind of mod I'm thinking of, so I don't have to be abstract: vtastek's shaders. These are very tightly coupled to how OpenMW's lighting and texturing and shadowing etc. all work and get broken reasonably often. Once post-processing is available, lots of the changes these shaders do, like making things oversaturated (and therefore too controversial to build into the engine), will be possible via that system, and will benefit from a stable API that we don't break, but others do genuinely need to poke around places we need to make frequent changes. There's an understanding that we'll avoid breaking compatibility when we don't gain anything from it, but will when we do, and that the maintenance burden resulting from that is on vtastek and users of his shaders.

Right now, vtastek's shaders are distributed as zips with GLSL files in them, and the recommended installation method is to copy the resources directory from the installation directory to somewhere else, add those shader files to the resources directory, and then point openmw.cfg at the modified resources directory. That's much more lightweight than a fork, and can be installed piecemiel, but if we change our policy to mods that don't use the stable API should be forks that would have to change.

The key distinctions between TES3MP, cc9cii's fork and OpenMW-VR versus vtastek's shaders and its ilk are:

• Any given version of vtastek's shaders are basically a finished product, whereas the forks aren't done, and won't be done until they're not forks anymore.
• We don't want vtastek's shaders in OpenMW itself (while lots of people like them, others, including me, find them less pretty than the built-in ones), whereas things like multiplayer, VR and Oblivion support are all objectively beneficial.