The Future of OpenMW Scripting

Everything about development and the OpenMW source code.
User avatar
Posts: 5538
Joined: 06 Aug 2011, 15:16

The Future of OpenMW Scripting

Post by Zini »

Last week we had a thread of heated, completely pointless discussion, full of trolling and overall a complete waste of time and it even resulted in one important developer leaving the project. This was such fun! So let's do it again!

Just kidding. Here is the promised discussion thread about the future of OpenMW scripting. I meant to post that a month ago, but as usual stuff got in the way. Sorry about that. RL bullshit, late but fairly important changes to the post-1.0 plans and for once even OpenMW-BS.

This is a discussion about continuing development of the existing language vs switching to a new scripting language. The topic is heated enough and the last thing we need is going off topic. So please stick with it. I am inclined to do heavy handed moderation for anything that goes on a tangent.
If there absolutely has to be a discussion about other script-related topics, please let's do that after this thread has run its course.

To be honest our previous discussions about this topic have been rather frustrating to me. That is because in my mind the decision for continued development of the existing scripting language is self-evident. The initial opposition came completely unprepared and had me stumped.

Of course that was stupid of me. If something appears to be self-evident, it needs to be reasoned for even more thoroughly. I don't know if we can reach any kind of consensus with this thread, but my hope is that we at least stop talking past each other so much. So let's get to it.

First a few definitions:

* oldscript: The scripting language as it is currently implemented in OpenMW. Basically a generalised version of the vanilla scripting language with better (or really any) error checking.
* oldscript+: The improved version of oldscript, as detailed in my upcoming 1.0 design document (almost ready, I promise). oldscript+ adds new features and removes all workarounds for bad scripts. Note that this does not mean that scripts that run in oldscript (with workarounds) need to be re-written to work in future versions of OpenMW (unless they want to take advantage of new features)
* newscript: A different scripting language (Lua has been proposed a lot, but let's stay abstract for now)

I see three possible routes how we could support newscript:

A) Develop newscript alongside of oldscript+. Basically support two fully functional scripting languages in parallel
B) Deprecate oldscirpt (meaning we keep it but no further development other than bug-fixes), drop oldscript+ all-together and then implement newscript.
C) Write a tool that translate oldscript into newscript. Implement newscript. Drop oldscript+ and also remove the oldscript engine from OpenMW.

I think all three options are bad, but for different reasons and to different degrees.

Option A) is the worst in my opinion and I have so many problems with it that a discussion about that could probably fill its own thread. Starting off, it would be a grand waste of developer time. Remember that in the future we want to provide advanced scripting tools in the editor (e.g. debugger). Both the initial script support and any advanced features would have to be duplicated. More work for us, more code complexity, more chances for things to go wrong.
I feel inclined to quote PEP 20 here: There should be one-- and preferably only one --obvious way to do it.
From a software engineering point of view having two scripting languages with the same purpose in one engine is a very bad idea ™.

But there is even more to it. It stands to reason that over time OpenMW scripting will generate a large body of documentation, e.g. in the form of tutorials, how-tos, forum-discussion threads, in text and possibly in video form. We see that in any other at least semi-widely used languages. Have a coding problem? Enter it into google and something will come up.

By having two languages we split the documentation in two camps. Now for us professional coders that isn't a problem. We can easily translate concepts between languages and learning a new language to at least a basic level of competency only takes a day or two.
But from my experience the majority of people writing scripts for MW are not professional software engineers. We should not forget that for an average person learning a new programming language can be a challenging task. For such a person having to deal with documentation in two languages can be a large disadvantage.

And then finally there is the question if we can keep the functionality of both languages in sync. I am most certain that we won't be able to do that always. Sooner or later there will come the time when we add a feature to one language and it gets delayed in the other one (by a feature release or more). As development will get increasingly more active after 1.0 (hopefully) this can only get worse. Sooner or later mod developers will arrive at the point where they have to think "Do I have to use this language or that language for this mod" which then culminates in "Oh, I have to rewrite this script in the other language if I want to use this feature."

Now to option B): Two years ago I would have said not just No!, but Hell, no! Now, with the Redemption project gone, I don't have a dog in this fight anymore, but I still think it is a bad idea.

Many existing scripts could benefit from new features we introduce after 1.0. Even for a small mod having to rewrite most scripts in a different language would be a significant barrier to adoption. For something like Redemption which had something of the order of several 10 000 lines of scripts, that would have been a deal breaker.

Now it is true that oldscript+ will disable all workarounds for bad scripts, meaning that many (maybe the majority of) existing scripts would have to be adjusted to take advantage of it. But there is a huge difference between having to re-write a script from scratch in another language and doing a small bit of syntactic housekeeping; especially when in the later case you have an editor that points out all the problems to you in a convenient list and lets you jump to the respective source line with just one click.

On its own merits this option is already bad. But there is more to it. Adding newscript is a major task. Yes, we seen a few examples where basic support for newscript was hacked in. But that won't do. If we do go to the trouble of getting a new scripting language, we won't half-ass it. Integration of a new scripting language will require careful design and extensive changes to the code base and that will require time.

Unfortunately that is at odds with the post 1.0 plan. The idea behind the post 1.0 plan is to cash in on the huge advantage we have with source access to a fully implemented game engine. We are (mostly) going for incremental improvements and low hanging fruits, that will allow us to dish out new modding opportunities at a rate that will make modders dizzy.
However nearly all of that depends on a script support. Not a problem with oldscript+, because that isn't actually that much work and we can do it incrementally. NewScript on the other hand would be a blocker. We can't do anything until it reaches feature parity with oldscript.
Now we could start early with newscript (some time before 1.0); assuming we actually have the manpower to do it. But that would distract us from getting to 1.0 as quickly as possible (quite significantly, I would guess). And since 1.0 is already late and people get unruly, that seems like a bad idea.

Now to option C): On its own merits this option seems the least objectionable to me. There is however a major issue with it. The oldscript compiler is relatively simple and it does cut a few corners. I don't think it would be up for the task. More likely we would end up with writing the translation tool from scratch while only re-using a few bits from the existing compiler. That would be a monumental task. I am not sure if we have the manpower for it and it has the same blocker/delay problems as option B), but cubed.

And that all for what? I don't get all the hate the vanilla language receives (my own feelings about it are rather neutral). IMO its problems fall in three categories:

1. No error checking worth speaking of and arbitrary limitations (e.g. no variables allowed for certain arguments)
2. Some glaring omissions (the most glaring ones being a proper function calling mechanism, several crucial variable types and some really basic stuff like boolean operators)
3. It will never win a beauty contest.

Problem #1 we have already addressed with oldscript. Oldscript+ will address problem #2. That leaves us only with problem #3. Yes, there is also a line about that in PEP 20, but it is really worth going to all that trouble over it? I don't think so.

Now there has been the problem brought up of scripts being of a "run-once-every-frame" nature. While some of the complains (coming from clearly accomplished software developers) are done with such vehemency that it makes me want to go like: "Show me on the doll where the bad state machine touched you" I basically agree with this complaint.

So, yes. You got an argument. But it is an argument for a different discussion. I have to argue that the questions of oldscirpt+ vs newscript and run-once-every-frame vs event/function-call-based execution are orthogonal to each other.

The run-once-every-frame construct is deeply embedded into the engine and the data structures. Using newscript will not automatically free us from that. Especially with option C) (but not limited to it) we will have to fully support run-once-every-frame and adding anything else will require additional work on top of that.
On the other hand continuing with oldscript does not stop us from offering alternatives to run-once-every-frame. In fact once you see the design for oldscript+, you will notice that most new features already moved away from it and there are also alternatives for existing functionality.

Well, that is all I have to say about this topic. Thank you for following my ramblings.
User avatar
Posts: 5538
Joined: 06 Aug 2011, 15:16

Re: The Future of OpenMW Scripting

Post by Zini »

It was pointed out to me that some implicit assumptions in this write up may not be obvious. I disagree with this position, but okay.

* I have a very strong preference for just going for oldscript+
* I have an extremely strong preference against option A)

My current assumption is still that we go with oldscript+. However if there is an overwhelming consensus that we should go with option B) instead (with an option to later translate it to C) ), I won't try to block it; that is, if we have the volunteers to do it.

As for the MWSE/Lua topic that has been discussed elsewhere, if we go for oldscript+ that would be a no go, because it would turn it into option A). If we go for option B), then we need to have a look at MWSE/Lua, examine it carefully and then decide, if it fits into our long term plans for OpenMW.
User avatar
Posts: 1175
Joined: 29 Jun 2015, 09:08

Re: The Future of OpenMW Scripting

Post by DestinedToDie »

Zini wrote: 14 Jun 2018, 15:02However if there is an overwhelming consensus that we should go with option B) instead (with an option to later translate it to C) ), I won't try to block it; that is, if we have the volunteers to do it.
If it's about consensus then I vote that we go for oldscript+ just my opinion I'm not a code commiter of course.
Posts: 8
Joined: 12 Apr 2015, 00:25

Re: The Future of OpenMW Scripting

Post by nerevar009 »

What about implementing oldscript+ (and dehardcoding in general) now/when the design document is finished instead of post-1.0? Could this attract new developers interested in new script capabilities/dehardcoding who might also help with bugfixes and remaining 1.0 features, helping 1.0 happen faster, or at the same time but with more capabilities?
User avatar
Posts: 5538
Joined: 06 Aug 2011, 15:16

Re: The Future of OpenMW Scripting

Post by Zini »

We can't really do that, because we need a fully function editor first.
User avatar
Posts: 2150
Joined: 26 Nov 2015, 03:25

Re: The Future of OpenMW Scripting

Post by AnyOldName3 »

On the pro-NewScript/Lua front, there are a few things that I feel this misses out:
  • We wouldn't have to implement/maintain the runtime.
  • People wanting to learn to do things aren't only restricted to OpenMW-specific documentation as the language exists already.
  • Most of the actual connecting-it-to-the-engine work should be telling it where some C++ functions are and how to expose them to Lua scripts and asking it to provide some other functions that can be called from C++. NullCascade has managed to do this for the original engine without even having source access to it.
  • Designing non-sucky programming languages is hard.
  • All of this makes me think that implementing NewScript might well be significantly easier than option C.
  • If we were to go for option A, I reckon most people would probably go for Lua, and it would quickly turn into a less bad version of option B.
  • I find it very unlikely that there'll be any particularly useful feature that we could implement in OldScript+ that wouldn't be at least as easy to implement in Lua as I don't hear people complaining that Lua itself is missing something important all the time, and any non-language-extension features would basically just involve exposing extra functions. If the unique features all end up in Lua instead of OldScript+, then we're still at option B.
AnyOldName3, Master of Shadows
Posts: 1597
Joined: 04 Sep 2011, 08:33

Re: The Future of OpenMW Scripting

Post by Chris »

I'd prefer Option C myself. A (built-in) tool to translate oldscript/mwscript to whatever other language we end up using, then use that language for script processing. Mods can write in either, and they'll be implicitly translated as needed. Although it's been okay, I don't imagine maintaining a high-performance scripting engine is easy, especially as enhancements get made to it (functions, callbacks/events, preemption, etc). As it is, there's a lot of stuff related to adding a new command that seems either nonsensical, conceptual but no functional design purpose, and various changes needed in multiple places to keep it synchronized.

Even with Option C though, we don't need to wholly drop oldscript+. It would just be a matter of adding the capabilities to the translator rather than adding it to the script processor.
Posts: 121
Joined: 16 Jan 2012, 07:58

Re: The Future of OpenMW Scripting

Post by NullCascade »

As for the MWSE/Lua topic that has been discussed elsewhere, if we go for oldscript+ that would be a no go, because it would turn it into option A). If we go for option B), then we need to have a look at MWSE/Lua, examine it carefully and then decide, if it fits into our long term plans for OpenMW.
If you have a change of heart and have any interest in collaboration, let me know. It doesn't have to be "take a hard look at MWSE-Lua and see if we want to support it" it could be "work together with MWSE-Lua to try to make its APIs compatible with our APIs". I'm not against making breaking changes to MWSE-Lua in its current state. It's far better to have some early on annoyance that is semi-easily fixed than it is to continue on towards 2090 with almost compatible scripting languages.

My vote, such that it is, of course is to go hard on newscript. oldscript is a nightmare to any modder who has actually had to live in it. oldscript+ is prone to break existing code (not that the project cares much about existing mod support). newscript lets the project break free of these restrictions and use a real language, rather than something cobbled together in-house.
MWSE 2.x lead.
User avatar
Posts: 502
Joined: 06 Jun 2017, 05:57

Re: The Future of OpenMW Scripting

Post by Thunderforge »

Whatever we do, I think that OpenMW must be able to run Morrowind mods written in pure oldscript. The modding community is one of the major reasons that Morrowind is still so popular 16 years after its release, and to say that we can't run even the most basic of the mods written over that time is a great way to kill community support.

That said, we don't have to actually have to use oldscript at runtime. If part of the mod installation process was "run this oldscript to newscript converter", then load the converted mod, I would be absolutely fine with that, so long as it is a reliable conversion process. My point is that someone should always be able to take a mod from 2002 and after installing it (whatever the process is), they should be able to play it in OpenMW.
User avatar
Posts: 155
Joined: 11 Aug 2011, 07:57
Location: Vivec City, MW

Re: The Future of OpenMW Scripting

Post by heilkitty »

Not an OpenMW developer, but I'd say, option B would be the most rational use of developers' time due to reasons AnyOldName3 stated.
In Soviet MoЯЯowind, Almalexia kills YOU!!