The Future of OpenMW Scripting
Posted: 14 Jun 2018, 14:16
Last week we had a thread of heated, completely pointless discussion, full of trolling and overall a complete waste of time and it even resulted in one important developer leaving the project. This was such fun! So let's do it again!
Just kidding. Here is the promised discussion thread about the future of OpenMW scripting. I meant to post that a month ago, but as usual stuff got in the way. Sorry about that. RL bullshit, late but fairly important changes to the post-1.0 plans and for once even OpenMW-BS.
This is a discussion about continuing development of the existing language vs switching to a new scripting language. The topic is heated enough and the last thing we need is going off topic. So please stick with it. I am inclined to do heavy handed moderation for anything that goes on a tangent.
If there absolutely has to be a discussion about other script-related topics, please let's do that after this thread has run its course.
To be honest our previous discussions about this topic have been rather frustrating to me. That is because in my mind the decision for continued development of the existing scripting language is self-evident. The initial opposition came completely unprepared and had me stumped.
Of course that was stupid of me. If something appears to be self-evident, it needs to be reasoned for even more thoroughly. I don't know if we can reach any kind of consensus with this thread, but my hope is that we at least stop talking past each other so much. So let's get to it.
First a few definitions:
* oldscript: The scripting language as it is currently implemented in OpenMW. Basically a generalised version of the vanilla scripting language with better (or really any) error checking.
* oldscript+: The improved version of oldscript, as detailed in my upcoming 1.0 design document (almost ready, I promise). oldscript+ adds new features and removes all workarounds for bad scripts. Note that this does not mean that scripts that run in oldscript (with workarounds) need to be re-written to work in future versions of OpenMW (unless they want to take advantage of new features)
* newscript: A different scripting language (Lua has been proposed a lot, but let's stay abstract for now)
I see three possible routes how we could support newscript:
A) Develop newscript alongside of oldscript+. Basically support two fully functional scripting languages in parallel
B) Deprecate oldscirpt (meaning we keep it but no further development other than bug-fixes), drop oldscript+ all-together and then implement newscript.
C) Write a tool that translate oldscript into newscript. Implement newscript. Drop oldscript+ and also remove the oldscript engine from OpenMW.
I think all three options are bad, but for different reasons and to different degrees.
Option A) is the worst in my opinion and I have so many problems with it that a discussion about that could probably fill its own thread. Starting off, it would be a grand waste of developer time. Remember that in the future we want to provide advanced scripting tools in the editor (e.g. debugger). Both the initial script support and any advanced features would have to be duplicated. More work for us, more code complexity, more chances for things to go wrong.
I feel inclined to quote PEP 20 here: There should be one-- and preferably only one --obvious way to do it.
From a software engineering point of view having two scripting languages with the same purpose in one engine is a very bad idea ™.
But there is even more to it. It stands to reason that over time OpenMW scripting will generate a large body of documentation, e.g. in the form of tutorials, how-tos, forum-discussion threads, in text and possibly in video form. We see that in any other at least semi-widely used languages. Have a coding problem? Enter it into google and something will come up.
By having two languages we split the documentation in two camps. Now for us professional coders that isn't a problem. We can easily translate concepts between languages and learning a new language to at least a basic level of competency only takes a day or two.
But from my experience the majority of people writing scripts for MW are not professional software engineers. We should not forget that for an average person learning a new programming language can be a challenging task. For such a person having to deal with documentation in two languages can be a large disadvantage.
And then finally there is the question if we can keep the functionality of both languages in sync. I am most certain that we won't be able to do that always. Sooner or later there will come the time when we add a feature to one language and it gets delayed in the other one (by a feature release or more). As development will get increasingly more active after 1.0 (hopefully) this can only get worse. Sooner or later mod developers will arrive at the point where they have to think "Do I have to use this language or that language for this mod" which then culminates in "Oh, I have to rewrite this script in the other language if I want to use this feature."
Now to option B): Two years ago I would have said not just No!, but Hell, no! Now, with the Redemption project gone, I don't have a dog in this fight anymore, but I still think it is a bad idea.
Many existing scripts could benefit from new features we introduce after 1.0. Even for a small mod having to rewrite most scripts in a different language would be a significant barrier to adoption. For something like Redemption which had something of the order of several 10 000 lines of scripts, that would have been a deal breaker.
Now it is true that oldscript+ will disable all workarounds for bad scripts, meaning that many (maybe the majority of) existing scripts would have to be adjusted to take advantage of it. But there is a huge difference between having to re-write a script from scratch in another language and doing a small bit of syntactic housekeeping; especially when in the later case you have an editor that points out all the problems to you in a convenient list and lets you jump to the respective source line with just one click.
On its own merits this option is already bad. But there is more to it. Adding newscript is a major task. Yes, we seen a few examples where basic support for newscript was hacked in. But that won't do. If we do go to the trouble of getting a new scripting language, we won't half-ass it. Integration of a new scripting language will require careful design and extensive changes to the code base and that will require time.
Unfortunately that is at odds with the post 1.0 plan. The idea behind the post 1.0 plan is to cash in on the huge advantage we have with source access to a fully implemented game engine. We are (mostly) going for incremental improvements and low hanging fruits, that will allow us to dish out new modding opportunities at a rate that will make modders dizzy.
However nearly all of that depends on a script support. Not a problem with oldscript+, because that isn't actually that much work and we can do it incrementally. NewScript on the other hand would be a blocker. We can't do anything until it reaches feature parity with oldscript.
Now we could start early with newscript (some time before 1.0); assuming we actually have the manpower to do it. But that would distract us from getting to 1.0 as quickly as possible (quite significantly, I would guess). And since 1.0 is already late and people get unruly, that seems like a bad idea.
Now to option C): On its own merits this option seems the least objectionable to me. There is however a major issue with it. The oldscript compiler is relatively simple and it does cut a few corners. I don't think it would be up for the task. More likely we would end up with writing the translation tool from scratch while only re-using a few bits from the existing compiler. That would be a monumental task. I am not sure if we have the manpower for it and it has the same blocker/delay problems as option B), but cubed.
And that all for what? I don't get all the hate the vanilla language receives (my own feelings about it are rather neutral). IMO its problems fall in three categories:
1. No error checking worth speaking of and arbitrary limitations (e.g. no variables allowed for certain arguments)
2. Some glaring omissions (the most glaring ones being a proper function calling mechanism, several crucial variable types and some really basic stuff like boolean operators)
3. It will never win a beauty contest.
Problem #1 we have already addressed with oldscript. Oldscript+ will address problem #2. That leaves us only with problem #3. Yes, there is also a line about that in PEP 20, but it is really worth going to all that trouble over it? I don't think so.
Now there has been the problem brought up of scripts being of a "run-once-every-frame" nature. While some of the complains (coming from clearly accomplished software developers) are done with such vehemency that it makes me want to go like: "Show me on the doll where the bad state machine touched you" I basically agree with this complaint.
So, yes. You got an argument. But it is an argument for a different discussion. I have to argue that the questions of oldscirpt+ vs newscript and run-once-every-frame vs event/function-call-based execution are orthogonal to each other.
The run-once-every-frame construct is deeply embedded into the engine and the data structures. Using newscript will not automatically free us from that. Especially with option C) (but not limited to it) we will have to fully support run-once-every-frame and adding anything else will require additional work on top of that.
On the other hand continuing with oldscript does not stop us from offering alternatives to run-once-every-frame. In fact once you see the design for oldscript+, you will notice that most new features already moved away from it and there are also alternatives for existing functionality.
Well, that is all I have to say about this topic. Thank you for following my ramblings.
Just kidding. Here is the promised discussion thread about the future of OpenMW scripting. I meant to post that a month ago, but as usual stuff got in the way. Sorry about that. RL bullshit, late but fairly important changes to the post-1.0 plans and for once even OpenMW-BS.
This is a discussion about continuing development of the existing language vs switching to a new scripting language. The topic is heated enough and the last thing we need is going off topic. So please stick with it. I am inclined to do heavy handed moderation for anything that goes on a tangent.
If there absolutely has to be a discussion about other script-related topics, please let's do that after this thread has run its course.
To be honest our previous discussions about this topic have been rather frustrating to me. That is because in my mind the decision for continued development of the existing scripting language is self-evident. The initial opposition came completely unprepared and had me stumped.
Of course that was stupid of me. If something appears to be self-evident, it needs to be reasoned for even more thoroughly. I don't know if we can reach any kind of consensus with this thread, but my hope is that we at least stop talking past each other so much. So let's get to it.
First a few definitions:
* oldscript: The scripting language as it is currently implemented in OpenMW. Basically a generalised version of the vanilla scripting language with better (or really any) error checking.
* oldscript+: The improved version of oldscript, as detailed in my upcoming 1.0 design document (almost ready, I promise). oldscript+ adds new features and removes all workarounds for bad scripts. Note that this does not mean that scripts that run in oldscript (with workarounds) need to be re-written to work in future versions of OpenMW (unless they want to take advantage of new features)
* newscript: A different scripting language (Lua has been proposed a lot, but let's stay abstract for now)
I see three possible routes how we could support newscript:
A) Develop newscript alongside of oldscript+. Basically support two fully functional scripting languages in parallel
B) Deprecate oldscirpt (meaning we keep it but no further development other than bug-fixes), drop oldscript+ all-together and then implement newscript.
C) Write a tool that translate oldscript into newscript. Implement newscript. Drop oldscript+ and also remove the oldscript engine from OpenMW.
I think all three options are bad, but for different reasons and to different degrees.
Option A) is the worst in my opinion and I have so many problems with it that a discussion about that could probably fill its own thread. Starting off, it would be a grand waste of developer time. Remember that in the future we want to provide advanced scripting tools in the editor (e.g. debugger). Both the initial script support and any advanced features would have to be duplicated. More work for us, more code complexity, more chances for things to go wrong.
I feel inclined to quote PEP 20 here: There should be one-- and preferably only one --obvious way to do it.
From a software engineering point of view having two scripting languages with the same purpose in one engine is a very bad idea ™.
But there is even more to it. It stands to reason that over time OpenMW scripting will generate a large body of documentation, e.g. in the form of tutorials, how-tos, forum-discussion threads, in text and possibly in video form. We see that in any other at least semi-widely used languages. Have a coding problem? Enter it into google and something will come up.
By having two languages we split the documentation in two camps. Now for us professional coders that isn't a problem. We can easily translate concepts between languages and learning a new language to at least a basic level of competency only takes a day or two.
But from my experience the majority of people writing scripts for MW are not professional software engineers. We should not forget that for an average person learning a new programming language can be a challenging task. For such a person having to deal with documentation in two languages can be a large disadvantage.
And then finally there is the question if we can keep the functionality of both languages in sync. I am most certain that we won't be able to do that always. Sooner or later there will come the time when we add a feature to one language and it gets delayed in the other one (by a feature release or more). As development will get increasingly more active after 1.0 (hopefully) this can only get worse. Sooner or later mod developers will arrive at the point where they have to think "Do I have to use this language or that language for this mod" which then culminates in "Oh, I have to rewrite this script in the other language if I want to use this feature."
Now to option B): Two years ago I would have said not just No!, but Hell, no! Now, with the Redemption project gone, I don't have a dog in this fight anymore, but I still think it is a bad idea.
Many existing scripts could benefit from new features we introduce after 1.0. Even for a small mod having to rewrite most scripts in a different language would be a significant barrier to adoption. For something like Redemption which had something of the order of several 10 000 lines of scripts, that would have been a deal breaker.
Now it is true that oldscript+ will disable all workarounds for bad scripts, meaning that many (maybe the majority of) existing scripts would have to be adjusted to take advantage of it. But there is a huge difference between having to re-write a script from scratch in another language and doing a small bit of syntactic housekeeping; especially when in the later case you have an editor that points out all the problems to you in a convenient list and lets you jump to the respective source line with just one click.
On its own merits this option is already bad. But there is more to it. Adding newscript is a major task. Yes, we seen a few examples where basic support for newscript was hacked in. But that won't do. If we do go to the trouble of getting a new scripting language, we won't half-ass it. Integration of a new scripting language will require careful design and extensive changes to the code base and that will require time.
Unfortunately that is at odds with the post 1.0 plan. The idea behind the post 1.0 plan is to cash in on the huge advantage we have with source access to a fully implemented game engine. We are (mostly) going for incremental improvements and low hanging fruits, that will allow us to dish out new modding opportunities at a rate that will make modders dizzy.
However nearly all of that depends on a script support. Not a problem with oldscript+, because that isn't actually that much work and we can do it incrementally. NewScript on the other hand would be a blocker. We can't do anything until it reaches feature parity with oldscript.
Now we could start early with newscript (some time before 1.0); assuming we actually have the manpower to do it. But that would distract us from getting to 1.0 as quickly as possible (quite significantly, I would guess). And since 1.0 is already late and people get unruly, that seems like a bad idea.
Now to option C): On its own merits this option seems the least objectionable to me. There is however a major issue with it. The oldscript compiler is relatively simple and it does cut a few corners. I don't think it would be up for the task. More likely we would end up with writing the translation tool from scratch while only re-using a few bits from the existing compiler. That would be a monumental task. I am not sure if we have the manpower for it and it has the same blocker/delay problems as option B), but cubed.
And that all for what? I don't get all the hate the vanilla language receives (my own feelings about it are rather neutral). IMO its problems fall in three categories:
1. No error checking worth speaking of and arbitrary limitations (e.g. no variables allowed for certain arguments)
2. Some glaring omissions (the most glaring ones being a proper function calling mechanism, several crucial variable types and some really basic stuff like boolean operators)
3. It will never win a beauty contest.
Problem #1 we have already addressed with oldscript. Oldscript+ will address problem #2. That leaves us only with problem #3. Yes, there is also a line about that in PEP 20, but it is really worth going to all that trouble over it? I don't think so.
Now there has been the problem brought up of scripts being of a "run-once-every-frame" nature. While some of the complains (coming from clearly accomplished software developers) are done with such vehemency that it makes me want to go like: "Show me on the doll where the bad state machine touched you" I basically agree with this complaint.
So, yes. You got an argument. But it is an argument for a different discussion. I have to argue that the questions of oldscirpt+ vs newscript and run-once-every-frame vs event/function-call-based execution are orthogonal to each other.
The run-once-every-frame construct is deeply embedded into the engine and the data structures. Using newscript will not automatically free us from that. Especially with option C) (but not limited to it) we will have to fully support run-once-every-frame and adding anything else will require additional work on top of that.
On the other hand continuing with oldscript does not stop us from offering alternatives to run-once-every-frame. In fact once you see the design for oldscript+, you will notice that most new features already moved away from it and there are also alternatives for existing functionality.
Well, that is all I have to say about this topic. Thank you for following my ramblings.