Lua scripting in OpenMW

[urm]

I'm not in favour of letting mods provide native code. I didn't have a strong opinion against it in the original debate, but was convinced otherwise. One of the main justifications for Lua was that it was easy to guarantee it couldn't call native code except via the bindings we exposed for the engine.

If it's just stuff like libraries, I imagine they'd be useful for lots of mods, and it's better to bundle them with the engine so mods can guarantee they've got access to them rather than potentially conflicting with another mod's build of the same library or not being provided for a given platform. That way, we can also vet them for security issues.

Having only Lua code run would be an acceptable approach for singleplayer, but for multiplayer that will definitely not work. At the very least, we need to provide a convenient API for communicating with other applications (socket-based?), which would cover uses cases such as TTS dialogues and, say, logging to a cloud service. It would be very elegant if "mods" could also add significant new features, such as support for different databases, but I guess those could be forks. Although forks are very inconvenient to maintain compared to a proper module.

[urm]

Let's not get ahead of ourselves and keep the scope of this bit of work concise and to the point, otherwise it'll never get done as we'll bike shed it to death and 'what about this?'

I think the approach outlined in the OP is more or less the most correct approach.

Are there are any pro/cons to the different sort of Lua backends? sol2, sol3, what tes3mp uses?

How this is time-lined however hasn't been touched on yet. To be clear, this feature is not a 1.0 blocker but a nice to have. The idea is that it will eventually replace our existing mwscript implementation. So we'll eventually need mwscript interpreter that maps on top of our Lua implementation.

We should also invest time/energy into documenting our Lua API.

Here's a quote from NullCascade on updating MWSE to sol3:

Nothing terribly important TBH. But sol2 isn't getting updates anymore, all dev will go for sol3. So we should update at some point, but it's not critical.

(taken from the MMC discord).
Also he's posted this link: https://thephd.github.io/sol3-compile-t ... nary-sizes

Since we are starting more or less from scratch with this, I don't see why we wouldn't use the most up to date solution - sol3. AFAIK David was planning to update the TES3MP solution to it eventually in any case.

[urm]

I mean not performance of the server, but performance cost of preparing, sending, and receiving the data on the client.

That doesn't sound like performance loss of any significance. Tes3mp runs with very comparable performance to OpenMW, assuming not too many players. (keep in mind all tes3mp builds are RelWithDebInfo)

Why it can not keep all the data in memory and only make saves every hour?

With the current packet architecture it could be good enough, since I suspect the real bottleneck to the amount of players connected would be sending all the packets (it is roughly O(n^2)), and not running out of RAM.
Although a naive Lua representation of Morrowind esp files takes up over 1Gb of RAM, which is not reasonable. And multiplayer in general leads to much bigger "saves" (e. g. the largest server is already forced to remove custom potions from inactive accounts), so I could see some script setups hitting the RAM limit very easily (e.g. random dungeon generation).
Overall I feel like a database solution would be much more future proof.

[NullCascade]

Here's a quote from NullCascade on updating MWSE to sol3:

Nothing terribly important TBH. But sol2 isn't getting updates anymore, all dev will go for sol3. So we should update at some point, but it's not critical.

(taken from the MMC discord).
Also he's posted this link: https://thephd.github.io/sol3-compile-t ... nary-sizes

Since we are starting more or less from scratch with this, I don't see why we wouldn't use the most up to date solution - sol3. AFAIK David was planning to update the TES3MP solution to it eventually in any case.

Absolutely follow this information if using sol (and definitely use sol3 over sol2). Also note that support for sol is currently offline as the developer handles some larger C++ community issues. Either way, if using sol expect your binary size to bloat extraordinarily. In my experience, contrary to that link, sol3 drastically increased MWSE's binary size compared to sol2 (which was already rather bloated). If whoever ends up doing this ends up going with lua and sol, feel free to hit me up on Discord for some best practices for use with the library.

[ptmikheev]

Absolutely follow this information if using sol (and definitely use sol3 over sol2). Also note that support for sol is currently offline as the developer handles some larger C++ community issues. Either way, if using sol expect your binary size to bloat extraordinarily. In my experience, contrary to that link, sol3 drastically increased MWSE's binary size compared to sol2 (which was already rather bloated). If whoever ends up doing this ends up going with lua and sol, feel free to hit me up on Discord for some best practices for use with the library.

Thanks! I didn't know about differences in the binary size.

Sol3 has a very nice interface, but actually we need only a small subset of its features. I am going to write a prototype using Lua C API directly. After reading documentation it doesn't look complicated. I think a hundred lines of template magic will allow us to avoid an extra dependency.

I'm not sure how you see that automatic saving and loading to be honest. You certainly don't want to sync every script variable automatically (I don't think you can anyway), as they might depend on some non-permanent factors, such as currently connected players or even the current time. You would still need some API to save/load data from the Lua side, and some way to initialize scripts.

I think it is possible to make it transparent, but I need to do some research first. Let's return to this question later.

You was right. I haven't found a good way to snapshot Lua state.
There is Pluto Library, but it is not an official part of Lua and has hard dependency on a specific Lua version, so let's avoid it.

So every script will need to define "save() -> data" and "load(data)" functions to store it's local variables.
But I think that the world state itself should be stored on C++ side. Scripting should be simple, so let's keep all the stuff related to storage and formats in C++ as a basic part of the engine.

Having only Lua code run would be an acceptable approach for singleplayer, but for multiplayer that will definitely not work. At the very least, we need to provide a convenient API for communicating with other applications (socket-based?), which would cover uses cases such as TTS dialogues and, say, logging to a cloud service. It would be very elegant if "mods" could also add significant new features, such as support for different databases, but I guess those could be forks. Although forks are very inconvenient to maintain compared to a proper module.

Socket-based API without direct access to the file system seems to be a reasonable compromise between functionality on one side and security and being cross-platform on the other side.
My opinion is to disallow loading additional DLLs from mods. But of course we can provide some general-purpose libraries with the engine.

Although a naive Lua representation of Morrowind esp files takes up over 1Gb of RAM, which is not reasonable.

In the solution I am thinking of we don't need Lua representation. Lua can access C++ representation via "userdata".

[urm]

You was right. I haven't found a good way to snapshot Lua state.
There is Pluto Library, but it is not an official part of Lua and has hard dependency on a specific Lua version, so let's avoid it.

So every script will need to define "save() -> data" and "load(data)" functions to store it's local variables.
But I think that the world state itself should be stored on C++ side. Scripting should be simple, so let's keep all the stuff related to storage and formats in C++ as a basic part of the engine.

It's not so much whether you can, as whether you should. The lifecycle of a Lua script might not be exactly the same as one of a game save. For example, there is this MWSE mod: Echoes of the Fallen https://www.nexusmods.com/morrowind/mods/48603.

I would prefer a simple but convenient save/loading API, potentially even with a way to distinguish between general and per-save storage (e. g. some mod settings should stay between saves).

In the solution I am thinking of we don't need Lua representation. Lua can access C++ representation via "userdata".

Yep, this is more or less the default, and what MWSE uses. Converting them to Lua tables might still have value though (e. g. easier serialization).
One thing with data access, is that we probably want a convenient and uniform way to access both the currently loaded data and that which is only in .esp at the moment.

[ptmikheev]

Here is a prototype of an interface between C++ and Lua:
https://gitlab.com/-/snippets/2034622

Using sol2 or sol3 would mean adding more than 20'000 lines of code to OpenMW repo.
My prototype uses Lua C API directly and has only 300 lines of code. It has not so many features as sol3, but it seems to be enough for our needs.

[psi29a]

Using sol2 or sol3 would mean adding more than 20'000 lines of code to OpenMW repo.

How is that? Is that just generated interface stuff? Or are you trying to import sol2/3 library into /extern ?

Keep in mind, we can add more dep libraries as needed.

[ptmikheev]

Using sol2 or sol3 would mean adding more than 20'000 lines of code to OpenMW repo.

How is that? Is that just generated interface stuff? Or are you trying to import sol2/3 library into /extern ?

Keep in mind, we can add more dep libraries as needed.

sol2 and sol3 are header-only libraries. As I understand the only way to use it is to include its code to our repo and update it periodically.

For example look at "extern/sol2/sol.hpp" in the old pull request: https://github.com/OpenMW/openmw/pull/2523

[AnyOldName3]

If anything, header-only libraries are easier to have as dependencies than compiled ones. You're not going to get linker issues if you don't need to link with anything, and you just need to make sure CMake knows where the headers are, whether that's in the extern directory, a build/deps directory, or in the system include directory.