SWIG Scripting Extensions project

Everything about development and the OpenMW source code.
maqifrnswa
Posts: 180
Joined: 14 Jan 2013, 03:57

SWIG Scripting Extensions project

Post by maqifrnswa »

This is a thread on extending OpenMW scripting to SWIG (e.g., python)
For information on how to install and test:
https://wiki.openmw.org/index.php?title ... _Extension

I think a post-1.0 goal, of mine at least, will still be to see if I can get an easy generic scripting interface.

It will be a seperate fork from openmw, as I think a unified scripting language and zini's enhancements to the existing implementation would be best for the community. It's generally better to improve one implementation rather than create several new ones. If, in the future, such extension of scripting is found desireable, then there is no problem incorporating it into openmw.

There are some things that python/other languages can do better and can be implemented faster than writing a new scripting engine from scratch. For example, you can use all of python's existing modules and capabillities that could enable things like dynamic web content/interaction in games. Some people are already used to lua and python scripting, they could potentially write more complicated scripts or more easily integrate scripts. For discussion on python versus lua, I found this very helpful:http://lua-users.org/wiki/LuaVersusPython -- I think both could be usefull in different cases.

Existing scripting IDEs can be used, most scripting language already has built-in verbose debugging. It's possible to generate embedded consoles for testing scripts on the fly. Scripts can be edited and recompiled on the fly, in game.

Security will be an issue (running untrusted scripts), but it something people already deal with in the modding community (e.g, Blender Python Scripting)Security. Concerns can be addressed at a minimum by making users opt in to using python, and can be extended to only allow running scripts signed by a trusted certificate/key.
Last edited by maqifrnswa on 26 Jan 2015, 19:03, edited 1 time in total.
maqifrnswa
Posts: 180
Joined: 14 Jan 2013, 03:57

Re: SWIG Scripting Extensions project

Post by maqifrnswa »

Weekly update:

New update: all morrowind variables can be set and get with the new "omwset" and "omwget" command in python
set("localvariablename", value)
set("globalvariablename", value)
set("objectID.variablename", value)
set("globalID.variablename", value)

to avoid function name collisions, I changed the commands to:
omwset("localvariablename", value)
omwset("globalvariablename", value)
omwset("objectID.variablename", value)
owmset("globalID.variablename", value)

and omwget("localvariablename")
etc.

Overall status:
[*]all extension instructions work in python
[*]all extension functions work in python
[*]global, local variables can be set and get in python
[*]You can run python scripts from the console as global scripts

Code: Select all

StartExternalScript, "name.py"
[*]You can run python scripts from scripts attached to objects in the editor

Code: Select all

begin SomethingScript
short hello
StartExternalScript, "test.py"
end
that allows both the python script to access "hello" as a local and allow other scripts to interact with the python script

On going:
[*] performance improvements (subclassing interpreter to create python environment on initialization, and method to reset environment when new script runs; create presistant interpreter object rather than installing opcodes everytime)
[*] "get" function, just like "set"
[*] a generic "call" function that will expose all non-extension instructions/functions (like messagebox). It will compile whatever is in the argument of "call" like MWGui::Console.compile does.
[*] implement tcl, lua
maqifrnswa
Posts: 180
Joined: 14 Jan 2013, 03:57

Re: SWIG Scripting Extensions project

Post by maqifrnswa »

Update:
All functions and instructions now work in Python!
python functions are mapped to all of these:
https://github.com/maqifrnswa/openmw/bl ... ndings.hpp
https://github.com/maqifrnswa/openmw/bl ... dings0.hpp

and every other function/instruction not on that list is callable with "omwcall()" (see wiki below)

A PPA has been set up for testing and demonstration!


Detailed info here:
https://wiki.openmw.org/index.php?title ... _Extension

SECURITY warning:
Python, for better or worse, exposes the whole python system/modules/everything. That's powerful, but a security risk. Sandboxing is considered difficult as well, as someone can always figure out how to out-hack you. So, for now, I'm keeping it open - which gives game developers power, but also possible security risks. We can think of how to do script signing/trusting later (this is the same issue blender has, and it allows users to run python scripts). So, only run scripts you trust!

TO DO:
LUA scripting is coming next. That is totally sandbox-able, and can be made secure. It's tiny and lightweight, faster than python as well.
wheybags
Posts: 207
Joined: 21 Dec 2012, 19:41

Re: SWIG Scripting Extensions project

Post by wheybags »

PLZ merge this into mainline openmw
User avatar
sjek
Posts: 442
Joined: 22 Nov 2014, 10:51

Re: SWIG Scripting Extensions project

Post by sjek »

Maybe after 1.0 in some form or as totally togglable / as a barebone.

http://www.ogre3d.org/tikiwiki/PyOgre I take that libraries like this or any other can be used to alter he game functionality on top of it .?

http://creepypasta.wikia.com/wiki/Jvk1166z.esp .. .p has also good youtube storytelling
There's even reddit group for making that...
.... watch the sky.....
Chris
Posts: 1625
Joined: 04 Sep 2011, 08:33

Re: SWIG Scripting Extensions project

Post by Chris »

"Python, for better or worse, exposes the whole python system/modules/everything."

This ensures it won't ever be merged.
maqifrnswa
Posts: 180
Joined: 14 Jan 2013, 03:57

Re: SWIG Scripting Extensions project

Post by maqifrnswa »

Lua is done and totally implemented! See https://wiki.openmw.org/index.php?title ... _Extension for examples and how to use, I'll try to update the PPA shortly.
Chris wrote:"Python, for better or worse, exposes the whole python system/modules/everything."

This ensures it won't ever be merged.
The fastest way to get a scientist to do something is to tell them it is impossible ;)

Lua can be sandboxed, so I don't see that as being a barrier for merging.

We really can pick-and-choose how to best implement this.

mainline openmw could pick and choose settings for each of:
1) default build options
2) available build options
3) default runtime options
4) available opt-in runtime options

from any of the following python choices
a) untrusted full functionallity python
b) trusted full functionallity python (only run cryptographically signed scripts)
c) no python

and with any of the following lua choices
a) untrusted fully functional lua
b) untrusted sandboxed lua (limited to safe functions)
c) trusted fully functional lua (only run cryptographically signed scripts)
d) no lua

My opinion, as of now:
1) default build options: python (c), lua (b)
2) available build options: python (a,b,c), lua (a,b,c,d)
3) default runtime options: python (c), lua (d)
4) available opt-in runtime options: python (a,b,c), lua (a,b,c,d)

That is, only allow build with secure options by default and keep them disabled unless a user opts-in. However, people are free to build binaries that allow insecure code in case they are developing their own games and want it.

For precedence, this is what blender chose: http://wiki.blender.org/index.php/Doc:2 ... 6_Security
1) default build options: python (a)
2) available build options: python (a,c)
3) default runtime options: python (c)
4) available opt-in runtime options: python (a,c)

AND/OR we could continue to have two seperate forks, one totally locked down (python (c), lua (d)) and one totally open (python (a), lua (a))

Either way, i'll keep tweaking the fork to see if it could be useful.
Chris
Posts: 1625
Joined: 04 Sep 2011, 08:33

Re: SWIG Scripting Extensions project

Post by Chris »

maqifrnswa wrote:The fastest way to get a scientist to do something is to tell them it is impossible ;)
It's not that it isn't possible, it certainly is able to happen. But allowing user mods unfettered system access not only introduces security holes (I can just imagine viruses being unknowingly spread that way), but it also allows for compatibility problems for differing systems (Windows-only mods, X11-only mods, etc..). Sandboxed with a pre-set (but user-extendable) API is really the only way to go.
For precedence, this is what blender chose: http://wiki.blender.org/index.php/Doc:2 ... 6_Security
Blender isn't a game engine*, it's a production tool. It has different needs than OpenMW.

* Yes, it technically has a game engine, but it's still first and foremost a modeling tool.
User avatar
psi29a
Posts: 5356
Joined: 29 Sep 2011, 10:13
Location: Belgium
Gitlab profile: https://gitlab.com/psi29a/
Contact:

Re: SWIG Scripting Extensions project

Post by psi29a »

Chris, he knows that. :)

Python isn't on his plan to be merged into OpenMW, yet.

Did you miss the part where he has LUA working. 8-)

Nothing stopping LUA from being included, aside from Zini of course. Dude is like a brick wall!
maqifrnswa
Posts: 180
Joined: 14 Jan 2013, 03:57

Re: SWIG Scripting Extensions project

Post by maqifrnswa »

Chris wrote:
maqifrnswa wrote:The fastest way to get a scientist to do something is to tell them it is impossible ;)
It's not that it isn't possible, it certainly is able to happen. But allowing user mods unfettered system access not only introduces security holes (I can just imagine viruses being unknowingly spread that way), but it also allows for compatibility problems for differing systems (Windows-only mods, X11-only mods, etc..). Sandboxed with a pre-set (but user-extendable) API is really the only way to go.
I think the highlighted part is crucial; I totally agree with you. I'm thinking of keeping python in some other repo solely for those that actually want to break things (i.e., use openmw as an engine in some yet-to-be-conceived game, and write their content in other scripting langauges besides mwscript).

Lua, on the other hand, will allow for a stripped down interpreter where only safe and reasonable basic included functions will be white-listed (e.g. math & string functions).
Post Reply