Lua scripting in OpenMW

Everything about development and the OpenMW source code.
ptmikheev
Posts: 68
Joined: 01 Jun 2020, 21:05
Gitlab profile: https://gitlab.com/ptmikheev

Re: Lua scripting in OpenMW

Post by ptmikheev »

Additionaly to the argument about forks it is worth to mention that in distant future we are going to support other games (Oblivion/Fallout/Skyrim) as well. It sounds reasonable if all mechanics that are different in different games are implemented as Lua mods and user can choose which one to use. It is incompatible with the idea of a default implementation that is protected from overriding.
So in future openmw.cfg can contain:

Code: Select all

# Defines morrowind-style mechanics, distributed with OpenMW (as well as skyrim-shim.omwgame and others).
content=morrowind-shim.omwgame

# Original Morrowind content, obviously not distributed with OpenMW, user should buy it.
content=morrowind.esm  
...
Chris wrote: That's basically what I'm aiming for. The issues I responded to in this post, I'd have thought, are reasonable steps to ensure compatibility (i.e. not moving everything to be "built-in Lua mods" that mods can replace willy-nilly, and not allowing modded scripts to do everything a built-in script could), and pointing out that some of the ideas presented are a bit too general (making a plan to dehardcode any given function, when the way to dehardcode something will very much depend on the behavior being dehardcoded).

Essentially, I'm saying dehardcoding doesn't have to come in the form of "turn C++ functions into Lua functions/scripts that mods can replace", or that "in the end there will be no internal API at all and built-in scripts will not have any "special powers" comparing to normal mods". I don't think it's an unreasonable effort to temper that expectation. There will always be internal APIs that built-in scripts can use and mods can't (unless we want to limit what scripts can do, so that built-in scripts can only do the things modded scripts can, which I doubt was the intent), and that modded functionality won't always (if not will rarely) come in the form of replacing built-in scripts, compared to utilizing hooks provided by built-in scripts and various engine components.
We have completely different views on how OpenMW Lua should look like. I don't agree that my approach is less safe or less compatible. We can spent a lot of time arguing on what is better, but there is an important technical detail that makes such discussion almost useless:
I started designing OpenMW Lua almost a year ago and the whole design is based on my view. It seems to you that your ideas are easier in implementation, but in fact they are not compatible with what is already done. I respect your opinion, but implementing your ideas would mean throwing away most of my work, so it is not an option for me. Even if you convince others that your approach is better, you would need to find someone to redesign the whole scripting system from scratch.

Now I will try to explain why in the current concept first steps of dehardcoding are so essential and has high priority. Actually it is exactly to ensure compatibility:
  • C++ and Lua are completely different languages and interaction between C++ part and Lua part can not be as flexible as interaction between Lua scripts.
  • Since modders should be able to change game mechanics in any way they want, these mechanics should be implemented in Lua. It means dehardcoding, at least in future.
  • Interactions Lua <-> C++ and interactions between Lua scripts are implemented in different ways:
    1. C++ -> Lua: Engine handlers. Lua scripts can subscribe to events that come from C++. C++ can not call Lua functions directly.
    2. Lua -> C++: API packages. Lua calls C++ functions.
    3. Lua -> Lua: Extensible script interfaces and event system.
    Scripts can change something provided by other scripts via interfaces, but API packages (i.e. C++ API) are fixed.
  • If we expose something via API packages, then later dehardcoding will break mods that depend on it.
  • So in order to ensure compatibility we need to think about dehardcoding first and provide most of functionality via wrappers in built-in scripts (i.e. using Lua-to-Lua interaction). Then the API can be stable from the beginning and we will not need to change it during dehardcoding.
User avatar
psi29a
Posts: 5148
Joined: 29 Sep 2011, 10:13
Location: Belgium
Gitlab profile: https://gitlab.com/psi29a/
Contact:

Re: Lua scripting in OpenMW

Post by psi29a »

I don't think Chris' view is incompatible with your view ptmlkheev, just that I think his concerns are different and also valid. So taking those into consideration would make our way forward in OpenMW more grounded.

Chris has this very important point: Lua needs to replace MWScript

This was the whole reason why Zini agreed to do this instead of mwscript++.
There are two very important aspects here to this:
* Supporting ALL the things that MWScripts support with a Lua equivelent
* Re-factor our MWScript to use the Lua equivelent.
* or Replace MWScript entirely with a Lua based interpretor.

The last one is interesting, because it makes the idea of supporting mwscript, obscript, papyrus modular and fully Lua. No need to change OpenMW and recompile it, since you can just modify Lua in your IDE or notepad++ and test.

Which is a great use-case and we can write tests for this.

I think that this, or something along this line, has a higher priority than de-hardcoding everything first. Don't get me wrong though, we'll likely have to de-hardcode some things in order to faccilate this.

We have time, there is no rush, let's do it right the first time. :)
ptmikheev
Posts: 68
Joined: 01 Jun 2020, 21:05
Gitlab profile: https://gitlab.com/ptmikheev

Re: Lua scripting in OpenMW

Post by ptmikheev »

My point is that the first step of dehardcoding (design an API that is available via Lua-to-Lua interfaces and will still make sense after dehardcoding) should be done first.
We can not just implement "ALL the things that MWScripts support" because if we don't think about dehardcoding now, we will have to change the API and break compatibility later.
The actual dehardcoding can be done later. But if we care about compatibility it is essential to decide from the very beginning how the interfaces will look like after dehardcoding.

I agree that replacing MWScript with a Lua based interpretor is very important. But I don't see a way to start from it. It can not be easy at least because OpenMW Lua cares a lot about multiplayer and MWscript is purely singleplayer. We will not be able to map every command directly from MWscript to Lua.
User avatar
psi29a
Posts: 5148
Joined: 29 Sep 2011, 10:13
Location: Belgium
Gitlab profile: https://gitlab.com/psi29a/
Contact:

Re: Lua scripting in OpenMW

Post by psi29a »

Sorry, I didn't mean to say ALL the things... :)

What I meant was that we can use MWScript replacement as a use-case we need for starting the de-hardcoding process. It important because it is a singleplayer use case which should be a priority over multiplayer. Don't get me wrong, multiplayer will come and keeping this in mind is important, but let's nail the single player experience first.

Unless you think that it should be reversed, that multiplayer-first approach is the goal and that single player experience comes on top of this.

By picking apart what we need to make this happen, it can then be used as a starting point for de-hardcoding.

In the mean time, if someone is interested, perhaps think about implimenting a MWScript parser in Lua. They could look at how we do it in C++.

Once those two are ready, then we can start wiring it all up.

What do you guys think of that proposal?
ptmikheev
Posts: 68
Joined: 01 Jun 2020, 21:05
Gitlab profile: https://gitlab.com/ptmikheev

Re: Lua scripting in OpenMW

Post by ptmikheev »

psi29a wrote: 06 Aug 2021, 14:43 What I meant was that we can use MWScript replacement as a use-case we need for starting the de-hardcoding process. It important because it is a singleplayer use case which should be a priority over multiplayer. Don't get me wrong, multiplayer will come and keeping this in mind is important, but let's nail the single player experience first.
Yes, of course singleplayer has higher priority. I just mean that it would be pity if for example we add bindings for player journal and then a year later realize that in multiplayer some journal records are per-player (e.g. joining a faction) and some are common for everyone (e.g. that a city A is in war with city B), so every journal command needs an additional argument "type". It is better to think in advance and add such argument at the beginning (for now implementation can ignore it), than to break compatibility later.
psi29a wrote: 06 Aug 2021, 14:43 In the mean time, if someone is interested, perhaps think about implimenting a MWScript parser in Lua. They could look at how we do it in C++.

Once those two are ready, then we can start wiring it all up.

What do you guys think of that proposal?
It would be great if somebody wants to work on it!
User avatar
AnyOldName3
Posts: 2248
Joined: 26 Nov 2015, 03:25

Re: Lua scripting in OpenMW

Post by AnyOldName3 »

I thought MWScript-in-Lua was our plan B in case MWScript-to-Lua-transpiler turned out to be too hard. One of the main arguments was that MWScript+ provided an easy migration path for legacy mods to be ported, whereas NewScript (Lua) wouldn't unless it could be transpiled to give an equivalent. Even if it was only good enough as a starting point, used MWScript idiosyncracies instead of Lua ones, and still needed some polish, it would be much easier to get started with for a mod author than some MWScript and the OpenMW-Lua manual.
AnyOldName3, Master of Shadows
Chris
Posts: 1605
Joined: 04 Sep 2011, 08:33

Re: Lua scripting in OpenMW

Post by Chris »

ptmikheev wrote: 06 Aug 2021, 11:25 [*] Interactions Lua <-> C++ and interactions between Lua scripts are implemented in different ways:
  1. C++ -> Lua: Engine handlers. Lua scripts can subscribe to events that come from C++. C++ can not call Lua functions directly.
  2. Lua -> C++: API packages. Lua calls C++ functions.
  3. Lua -> Lua: Extensible script interfaces and event system.
Yes, but whether a given interface is implemented as Lua->C++ or Lua->Lua is irrelevant to mods. A modded script can call a particular function or subscribe to a particular event, but whether that function or event is implemented in C++ or a built-in Lua script is simply an implementation detail. It shouldn't matter to mods using them, and it could even be changed without affecting the behavior of mods using it. The main restriction is the function or event must remain available to get the expected behavior as the engine is updated. As such, there needs to be a safety measure to ensure such built-in scripts can't be altered, or else things will break.
ptmikheev wrote: 06 Aug 2021, 12:21 The actual dehardcoding can be done later. But if we care about compatibility it is essential to decide from the very beginning how the interfaces will look like after dehardcoding.
Part of the issue I feel is that you're looking at "dehardcoding" as a thing in itself to accomplish. It's not. There is no ultimate goal for it, it's simply a concept to apply to a particular aspect of the engine. There is already a level of dehardcoding inherently done given that mods can create whole new objects and replace existing ones via the CS, or change the world map, or add or change quest behavior. Obviously we want to give modders more capabilities than they currently have, but that's kind of the point; no matter how much you "dehardcode" the engine, you can keep digging down and finding more hardcoded things to dehardcode.

Asking to define what interfaces will look like "after dehardcoding" is like asking to define what the interfaces will look like after all modding capabilities are added. It's not really meaningful, particularly for something we want to grow organically over time, and especially before we've really started. It's one thing to have a conceptual goal in mind (e.g. allowing mods to do XYZ with the camera), but another to preconceive the end result and make other design decisions that depend on it ending up like that.
User avatar
AnyOldName3
Posts: 2248
Joined: 26 Nov 2015, 03:25

Re: Lua scripting in OpenMW

Post by AnyOldName3 »

Other game engines have picked a level of customisability they want to allow and then aimed to make that level of customisability available across the board. It's not like Roblox is aiming to eventually let you take control of as much as you can in Unreal Engine, for example, or that Unreal Engine got to where it is via a Roblox-esque stage.
AnyOldName3, Master of Shadows
EvilEye
Posts: 27
Joined: 12 Feb 2014, 13:45
Gitlab profile: https://gitlab.com/Assumeru

Re: Lua scripting in OpenMW

Post by EvilEye »

I really hope when you guys mention MWScript in Lua you're talking about reimplementing components/interpreter and apps/openmw/mwscript and not components/compiler. The latter seems like it'd be a massive pain/waste of time to port.

Aside from that I have some concerns about the execution model. i.e. it's probably important that the code in apps/openmw/mwscript is executed at the same time it is now - something I'd say is incompatible with multiple Lua threads. Need not be a problem, but hey, let's mention it up front.

Honestly I'd be more in favour of defining a useful Lua API for new mods to use while leaving the MWScript stuff be for now. (And outright ignoring it while designing said API.) Once a stable Lua API emerges that doesn't make people ask for MWScript features every 5 minutes, it'd be interesting to see how many of the OpCodes in apps/openmw/mwscript can be implemented as Lua functions (which I imagine could end up fairly large to account for the ideosyncrasies of MWScript) and how many would still need additional API work added.

tl;dr

Ignore MWScript while designing the Lua API and retrofit it later instead of doing it all upfront and potentially polluting the API with MWScript-isms.
User avatar
sjek
Posts: 442
Joined: 22 Nov 2014, 10:51

Re: Lua scripting in OpenMW

Post by sjek »

Ignore MWScript while designing the Lua API and retrofit it later instead of doing it all upfront and potentially polluting the API with MWScript-isms.
+1
just a hunch. altho a thing to keep in mind in some degree.

also, sorry if discussed already, but from mod author or new game creator perspective. as in with unprotected route: they don´t necessarily need to make a fork for their largely different mod/game. other side is that the scripts are gonna be under license of sort and not necessarily free to use (if possible in practise?)

on the other hand tho if on the protective route, the fork´s gonna be GPL3 in any case so it might put some off and creates different scripting apis.

both have their + and - but would +1 open route and let the greators choose their licenses and modding scene to handle it instead on the forked engine side. if getting the license situation right ?

for modding capalities.... as a past time fun for majority of people i think the availibility of scripts to play with should be most plentifull.

dunno exactly how this situation generally would be easiest with less of foot guns tho. but those can be mostly avoided hopefully.
"life is crazy"
"craziness has beauty which only crazies understand" some movie clip in the head.
https://wiki.openmw.org/index.php?title=Testing
Post Reply