Lua scripting in OpenMW

[AnyOldName3]

I'm not in favour of letting mods provide native code. I didn't have a strong opinion against it in the original debate, but was convinced otherwise. One of the main justifications for Lua was that it was easy to guarantee it couldn't call native code except via the bindings we exposed for the engine.

If it's just stuff like libraries, I imagine they'd be useful for lots of mods, and it's better to bundle them with the engine so mods can guarantee they've got access to them rather than potentially conflicting with another mod's build of the same library or not being provided for a given platform. That way, we can also vet them for security issues.

[ptmikheev]

If we allow Lua mods to have dlls (I'd say we should), we have to at least warn players about potential safety issues. With platform compatibility, we could either have every mod provide dlls for multiple platforms, or attempt to build them during installation. There are two existing packet systems for Lua: LuaRocks and LuaDist, the former goes with the build approach, the latter supports both.

I agree with AnyOldName3.
Some modders will likely care only about their favourite platform. It would be better to disable the possibility of external libraries completely, than to allow such source of incompatibility.

However in some cases the possibility of calling external software can be very attractive. My favourite example is a mod that uses an external speech synthesis tool (let's image in a couple of year it has very good quality) to voice all dialogues in the game. To make it possible we can probably allow mods to communicate with other software via network connection (so user will have to manually start the tool).

[ptmikheev]

Obviously, we would still have local scripts. There is no reason not to, to be honest. Current tes3mp server is extremely lightweight, and performs well even despite having many extremely inefficient parts. So I don't think there would be a meaningful performance difference. I would say the main argument for not having any networking for singleplayer, is the theoretical possibility for inconsistent behavior even with 0 ping.

I mean not performance of the server, but performance cost of preparing, sending, and receiving the data on the client.

The problem is that they are designed for very different use cases. Game saves are supposed to be as small and efficient as possible, and don't need to be written/read in real time. The server, on the other hand, regularly loads/unloads new data from/to disk, so it needs a storage format that's optimized for speed, rather than size.

Why it can not keep all the data in memory and only make saves every hour?

I'm not sure how you see that automatic saving and loading to be honest. You certainly don't want to sync every script variable automatically (I don't think you can anyway), as they might depend on some non-permanent factors, such as currently connected players or even the current time. You would still need some API to save/load data from the Lua side, and some way to initialize scripts.

I think it is possible to make it transparent, but I need to do some research first. Let's return to this question later.

[darkbasic]

My favourite example is a mod that uses an external speech synthesis tool (let's image in a couple of year it has very good quality) to voice all dialogues in the game.

Wouldn't be better to implement such a thing directly into the engine?

[ptmikheev]

My favourite example is a mod that uses an external speech synthesis tool (let's image in a couple of year it has very good quality) to voice all dialogues in the game.

Wouldn't be better to implement such a thing directly into the engine?

Definitely no. Such tool is a big and complicated thing that means multiple dependencies (like tensorflow), non-free voice presets and problems with licenses.

[lysol]

Would it be possible to implement a native way for the engine to find the tool and use it then, without making the tool a dependency? I mean, instead of making it possible for scripts to find and run whatever app they want, just define in the engine that this certain app is ok and that scripts are allowed to use it?

[darkbasic]

Definitely no. Such tool is a big and complicated thing that means multiple dependencies (like tensorflow), non-free voice presets and problems with licenses.

Dependencies can be made optional, licensing issues are harder to deal with instead.

[silentthief]

My favourite example is a mod that uses an external speech synthesis tool (let's image in a couple of year it has very good quality) to voice all dialogues in the game.

I am crossing fingers and hoping that this kind of thing will be doable: Talky Morrowind

(More here https://www.youtube.com/watch?v=QMkwMlQcHOQ)

ST the crossed

[AnyOldName3]

Remember people have the source code for OpenMW, so at the least, proof-of-concept versions of big things can happen in forks before we decide they need to be native features. The biggest example of this is TES3MP, but effectively every other enhancement has been the same, even if the fork had a pull request open for its whole life. We don't need to make literally everything compatible with our release binaries as anything that needs native code has access to the native code.

[psi29a]

Let's not get ahead of ourselves and keep the scope of this bit of work concise and to the point, otherwise it'll never get done as we'll bike shed it to death and 'what about this?'

I think the approach outlined in the OP is more or less the most correct approach.

Are there are any pro/cons to the different sort of Lua backends? sol2, sol3, what tes3mp uses?

How this is time-lined however hasn't been touched on yet. To be clear, this feature is not a 1.0 blocker but a nice to have. The idea is that it will eventually replace our existing mwscript implementation. So we'll eventually need mwscript interpreter that maps on top of our Lua implementation.

We should also invest time/energy into documenting our Lua API.