Never thought of this. Very good point actually.Zini wrote: Python is a no go in my opinion, because fully sandboxing it is very hard to impossible. Call me paranoid, but I see even the possibility of having viruses in OpenMW content files as a major disaster.
[post-1.0] Scripting enhancemnts
Re: [post-1.0] Scripting enhancemnts
-
maqifrnswa
- Posts: 180
- Joined: 14 Jan 2013, 03:57
Re: [post-1.0] Scripting enhancemnts
Also, from a community POV, you don't want fracturing with some people only working with one or the other and unable to interface with other members of the community.Zini wrote:Also, I think it will be harmful to have more than one scripting language.
My repo pretty much shows how you can finish implentation if anyone needs to in the future (console is done, just do the same thing for the run script opcode, and continue wrapper function work) in case someone wants to incorporate dynamic web content or other advanced things that are easier in python than c++ into their own games that use the openmw engine (using python)
No sanboxing is a problem (how do you determine what is trusted script), but you also get python editors and debuggers for no work. The current implementation doesn't put python code on morrowind content, but does allow one to run arbitrary python scripts on the system - which is a security risk.
Re: [post-1.0] Scripting enhancemnts
I've built a proof-of-concept hack that invokes Lua 5.1 in the console. The benefit is, since Lua is designed to interop with C/C++, it uses a stack just like the OpenMW MWScript runtime. So I don't even need to build bindings for all of the Morrowind methods: I just made a generic "call" binding, which invokes the correct Opcode, and the opcode->execute method reads the arguments like it would normally do, except it's been handed a subclassed Runtime that feeds it them from the Lua parameter stack*.
Then, to make it pretty, I hacked together a Lua script which adds sugar for calling them. So
player:additem("gold_001",10) actually invokes
call(<integer opcode>, "player", "gold_001", 10) which is bound to Console::call(), which looks up opAddItem() and executes it in the Console's current context.
I haven't got it in a repository yet but I'll put it somewhere soon. My favorite part is how little actually needs to be bound in C, so you can play with the sugar syntax just by reloading your base Lua script. It should be easy enough to sandbox it.
* anything that peeks past the top element in the parameter stack would fail, but I don't think any of them do
Then, to make it pretty, I hacked together a Lua script which adds sugar for calling them. So
player:additem("gold_001",10) actually invokes
call(<integer opcode>, "player", "gold_001", 10) which is bound to Console::call(), which looks up opAddItem() and executes it in the Console's current context.
I haven't got it in a repository yet but I'll put it somewhere soon. My favorite part is how little actually needs to be bound in C, so you can play with the sugar syntax just by reloading your base Lua script. It should be easy enough to sandbox it.
* anything that peeks past the top element in the parameter stack would fail, but I don't think any of them do
-
maqifrnswa
- Posts: 180
- Joined: 14 Jan 2013, 03:57
Re: [post-1.0] Scripting enhancemnts
awesome! (academic discussion at this point, but I'm still interested)rwig wrote:I've built a proof-of-concept hack that invokes Lua 5.1 in the console...
that's actually what I was thinking of doing what you did for python too, but couldn't figure out how to hand the runtume off. I'd be interested in seeing how you did it and if it could be generalized to SWIG. It would theoretically be able to be extended to any scripting language automatically. As zini said, this might not be usfeull for morrowind, but could be for someone wishing to use the openmw engine for a different game.
Security is still an issue; I was thinking of only running signed/trusted scripts. How is security with lua and your implementation?
Re: [post-1.0] Scripting enhancemnts
RegisterExtension associates keywords with opcodes; RegisterOpcode associates the opcodes with an Opcode0 object that actually executes the command.
So, first I just dump all of the registered Functions and Instruction keywords into an associative array in Lua, so Lua knows all the opcodes. They're just integers (well, floats, since Lua5.1 doesn't have ints), not pointers. Then there's a C function which takes an opcode as a parameter on the Lua stack, finds the associated Opcode that was registered, and executes it and returns the result. That function is bound to a Lua function.
I don't know how performance compares with swig, but it has the benefit of not requiring writing headers for Swig that it can understand. It might be possible to call swig programmatically inside RegisterExtension and have it build the bindings for us as if it were parsing a header file?
It might be possible to wall off the unsafe parts of the Lua stdlib (ones that can talk to the outside world, like "os") just with Lua. You can pick what environment to execute Lua code in, and I think I can build one that will just not have those library symbols. Otherwise, I'd have to build a custom Lua. Which I think is common with Lua, it's nothing like building a custom Python.
So, first I just dump all of the registered Functions and Instruction keywords into an associative array in Lua, so Lua knows all the opcodes. They're just integers (well, floats, since Lua5.1 doesn't have ints), not pointers. Then there's a C function which takes an opcode as a parameter on the Lua stack, finds the associated Opcode that was registered, and executes it and returns the result. That function is bound to a Lua function.
I don't know how performance compares with swig, but it has the benefit of not requiring writing headers for Swig that it can understand. It might be possible to call swig programmatically inside RegisterExtension and have it build the bindings for us as if it were parsing a header file?
It might be possible to wall off the unsafe parts of the Lua stdlib (ones that can talk to the outside world, like "os") just with Lua. You can pick what environment to execute Lua code in, and I think I can build one that will just not have those library symbols. Otherwise, I'd have to build a custom Lua. Which I think is common with Lua, it's nothing like building a custom Python.
-
maqifrnswa
- Posts: 180
- Joined: 14 Jan 2013, 03:57
Re: [post-1.0] Scripting enhancemnts
rwig, that is awesome - thank you. I'd like to see the repo some time if you get a chance!
EDIT:
moved implementation idea here:
https://wiki.openmw.org/index.php?title ... _Extension
EDIT:
moved implementation idea here:
https://wiki.openmw.org/index.php?title ... _Extension
Re: [post-1.0] Scripting enhancemnts
I will try and make it available this weekend. It's, ah, messy. There's a way to do it pretty I'm sure, but it's nowhere near yet.
As an aside, I think it may be possible to use MetaLua to compile MWScript into lua. Metalua lets you build extensions to the lua grammar, and the differences with MW scripting are not too great. The only big one is that MWScript lets you call functions without putting parentheses around the parameters. If we're lucky it may be possible to hack that into Lua without making an ambiguous grammar and then we can compile existing scripts into lua
Not sure what benefit that would give us, but it would be cool.
As an aside, I think it may be possible to use MetaLua to compile MWScript into lua. Metalua lets you build extensions to the lua grammar, and the differences with MW scripting are not too great. The only big one is that MWScript lets you call functions without putting parentheses around the parameters. If we're lucky it may be possible to hack that into Lua without making an ambiguous grammar and then we can compile existing scripts into lua
Not sure what benefit that would give us, but it would be cool.-
maqifrnswa
- Posts: 180
- Joined: 14 Jan 2013, 03:57
Re: [post-1.0] Scripting enhancemnts
Update:
I've created a tool "extensiontool" and added it to my python-scripting branch.
It links to libcomponents.a, and from there generates bindings for all 471 extensions (both header and implementations). The bindings don't work completely yet (return definetely won't work, haven't implemented explicit scope yet), but wanted to share progress.
Each binding generates a compiled MW script and passes it through an interpreter (seperate from the scriptmanager or console interpreter so don't need to worry about stack collisions). This way everything stays encapsulated and compatible with existing implementation and scripting improvements/bug fixes. This interpreter object uses the running script's interpreter context, which will allow us to treat this like any other script.
Why do this?
1) I'm learning A LOT about openmw. I may be new to coding, but the current compiler, interpretter, and scripting system is exteremly impressive - at least to me!
2) These are generic bindings, in theory you can use them to do everything from creating SWIG bindings to any scripting system to actually creating your own c++ plugins.
Example binding implementations: http://pastebin.com/mF5xuNA4
I've created a tool "extensiontool" and added it to my python-scripting branch.
It links to libcomponents.a, and from there generates bindings for all 471 extensions (both header and implementations). The bindings don't work completely yet (return definetely won't work, haven't implemented explicit scope yet), but wanted to share progress.
Each binding generates a compiled MW script and passes it through an interpreter (seperate from the scriptmanager or console interpreter so don't need to worry about stack collisions). This way everything stays encapsulated and compatible with existing implementation and scripting improvements/bug fixes. This interpreter object uses the running script's interpreter context, which will allow us to treat this like any other script.
Why do this?
1) I'm learning A LOT about openmw. I may be new to coding, but the current compiler, interpretter, and scripting system is exteremly impressive - at least to me!
2) These are generic bindings, in theory you can use them to do everything from creating SWIG bindings to any scripting system to actually creating your own c++ plugins.
Example binding implementations: http://pastebin.com/mF5xuNA4
Re: [post-1.0] Scripting enhancemnts
A few ideas, apologies if these have been suggested before, I imagine they may well have been thought of before.
A way to declare globals programmaticaly would be nice, and perhaps to modify GMSTs similarly. Perhaps such statements could be included in an individual script but outwith the begin end block. Programmatic declaration of other things (everything) could be useful, and an option to serialise an omwaddon to code that could then be used to regenerate the omwaddon. These features would make the use of VCS as discussed in this thread much more useful.
Event scripts, which run for one frame only before stopping themselves and are started by the engine under specific circumstances if they exist at run time. A not very useful example would be an evOnDaybreak script which could be executed at dawn. I can see problems with the same script being run multiple times in the same frame, some kind of queueing or locking mechanism may be required.
It occurred to me that given the prevolence of state variables and big if-elseif structures in scripts, jump tables may be handy. This may be best implemented as a compiler optimisation rather than with syntax. If all the conditions in an if-elseif are equality tests for known quantities (immediate values or functions with known and limited return values like MenuMode) then it should be possible to implement a jump table in the bytecode. I don't know how the scripting engine works, so perhaps I'm wrong. Even if I'm right it may have limited utility for legacy scripts given the block size limit of vanilla Morrowind's scripting engine. Perhaps this is covered by the planned case statements.
A way to declare globals programmaticaly would be nice, and perhaps to modify GMSTs similarly. Perhaps such statements could be included in an individual script but outwith the begin end block. Programmatic declaration of other things (everything) could be useful, and an option to serialise an omwaddon to code that could then be used to regenerate the omwaddon. These features would make the use of VCS as discussed in this thread much more useful.
Event scripts, which run for one frame only before stopping themselves and are started by the engine under specific circumstances if they exist at run time. A not very useful example would be an evOnDaybreak script which could be executed at dawn. I can see problems with the same script being run multiple times in the same frame, some kind of queueing or locking mechanism may be required.
It occurred to me that given the prevolence of state variables and big if-elseif structures in scripts, jump tables may be handy. This may be best implemented as a compiler optimisation rather than with syntax. If all the conditions in an if-elseif are equality tests for known quantities (immediate values or functions with known and limited return values like MenuMode) then it should be possible to implement a jump table in the bytecode. I don't know how the scripting engine works, so perhaps I'm wrong. Even if I'm right it may have limited utility for legacy scripts given the block size limit of vanilla Morrowind's scripting engine. Perhaps this is covered by the planned case statements.
Re: [post-1.0] Scripting enhancemnts
I am really intrigued by maqifrnswa's work with Python and want to investigate Cython and PyPy as possible variants for a Python sandbox. PyPy is even interesting for having a custom interpreter option. Is it crazy to imagine a Papyrus-compatible frontend to a Stackless Python sandbox?
However, there's much to be said for cleaning up Papyrus. As a former MUD coder, I've worked on in-game scripting languages and there are things that they do particularly well, especially when you expect novice coders to be involved. Counting loops and preventing more than, say, a thousand loops per tick in a script is a particularly good feature - you never know when someone is going to write an infinite loop!
One thing, though, is that OO is highly helpful to make sure that clean, efficient code is written for common patterns, especially for things like signal/slot behavior.
More specific ideas when I've actually reviewed more code!
However, there's much to be said for cleaning up Papyrus. As a former MUD coder, I've worked on in-game scripting languages and there are things that they do particularly well, especially when you expect novice coders to be involved. Counting loops and preventing more than, say, a thousand loops per tick in a script is a particularly good feature - you never know when someone is going to write an infinite loop!
One thing, though, is that OO is highly helpful to make sure that clean, efficient code is written for common patterns, especially for things like signal/slot behavior.
More specific ideas when I've actually reviewed more code!