The new MWSE-Lua interface

Everything about development and the OpenMW source code.
User avatar
SeaFox
Posts: 34
Joined: 29 Feb 2016, 17:30

Re: The new MWSE-Lua interface

Post by SeaFox » 09 Oct 2018, 02:31

Chris wrote:
09 Oct 2018, 01:55
SeaFox wrote:
08 Oct 2018, 23:03
That seems like an example of a false dichotomy. The end goal of a software project should, as you say, be a rigorously planned and sustainable piece of software. But the process of reaching that goal can, and should, include the use of software prototypes, such as the binary plugins within mods.
But mods aren't prototypes. A mod can be prototyped, but also has official releases, just like an engine. How would you ensure official releases of mods, which are expected to work for a wide audience well into the future, aren't using native code plugins?
Mods are generally the work of individuals who love the game and have ideas about how to improve it. They express those ideas by creating mods. They share those mods with the community as an expression of love for the game and the community that surrounds it. While these mods are not necessarily intended to be prototypes for future improvements to the OpenMW code base, they can serve that function. Mod creators obviously cannot spend their entire life maintaining that mod and providing support for it. Eventually the mod must be abandoned, and newer, better, mods will take it's place, and possibly an OpenMW developer will use the mod as a prototype for improving OpenMW.

User avatar
AnyOldName3
Posts: 1123
Joined: 26 Nov 2015, 03:25

Re: The new MWSE-Lua interface

Post by AnyOldName3 » 09 Oct 2018, 03:14

If it's something which can be done through a mod, it's probably something which should be done through a mod. This idea goes both ways, meaning we shouldn't be implementing things into the engine which work fine as a mod, and we shouldn't provide tools to let mods do things which are going to break literally everything else if they aren't handled by the engine.

A native code plugin is always going to be more work to create and maintain than a single-feature fork created with the intention of it (or a derivative of it) being merged back into master. If you want to prototype a feature, whether or not we offer unsandboxed mod support, forking is going to be easiest. The only benefit is if you want to do something which you know there's absolutely no chance you'll ever get merged, and even then, a fork might still be easier to maintain.
AnyOldName3, Master of Shadows

Chris
Posts: 1500
Joined: 04 Sep 2011, 08:33

Re: The new MWSE-Lua interface

Post by Chris » 09 Oct 2018, 03:52

SeaFox wrote:
09 Oct 2018, 02:31
While these mods are not necessarily intended to be prototypes for future improvements to the OpenMW code base, they can serve that function. Mod creators obviously cannot spend their entire life maintaining that mod and providing support for it.
All the more reason to ensure mods are given an engine where they can remain stable and compatible long after work on them stops.

User avatar
Ravenwing
Posts: 262
Joined: 02 Jan 2016, 02:51

Re: The new MWSE-Lua interface

Post by Ravenwing » 09 Oct 2018, 07:15

SeaFox wrote:
08 Oct 2018, 23:03
That seems like an example of a false dichotomy.
That's not how I mean it at all. They are simply both very good examples of each end of the spectrum. I'm sure that NullCascade plans many of his features extensively, but his driving principle is to get modders tools now rather than later. Reverse engineering the preexisting engine and hacking into it has proven a successful strategy for a great number of features, which the current modding community has benefited from. But there will come a time when OpenMW finally exceeds MWSE's functionality, it just takes a lot longer because we have to build the whole engine from scratch.
SeaFox wrote:
08 Oct 2018, 23:03
You seem to be suggesting that creators of mods should fork the entire OpenMW project whenever they wish to create a software prototype of some new feature which is not possible with the current code base. Binary plugins would be a far superior way of creatiing such prototypes.
I think this may be the source of our disconnect. First, forking a project is not as horrible as it sounds and we have all been using it in different contexts. Apologies if you know all this already, but hopefully this will clear some stuff up. In Git, it just means you've copied the project to your own directory so you can work on it without screwing with the master branch. It's literally required for any contributor to fork the project, it's just that the devs fork with the intention of working on features that will immediately be integrated back into the master branch upon completion. This is where the rapid prototyping happens! (We are 100% in agreement on this being a beneficial development strategy)

The other "fork" we've been talking about is if someone absolutely cannot work with the project as is and will fork it with no intention of ever merging back to the main branch. This is where things evolve so the projects become incompatible. Nobody wants this because it theoretically splits up effort and slows overall progress, but if enough of the community hates the direction Zini is taking with it, there is absolutely nothing stopping them from doing so.

David's TES3MP fork is something of a hybrid of the two as it won't be merged in anytime soon, but it is meant to maintain compatibility with the main project and eventually be merged back in. I'm under the impression that mod authors who are capable of developing their own features would fit into either this category or the first. Petethegoat brought up an excellent point, that there probably exists cases where features may not be added into the master branch. I hope my response was helpful in allaying concerns, but as I say, worrying about fringe cases at this point is a bit premature, and probably even more "fringe-y" than they might initially seem.

Your last statement that binary plugins are a far superior way of creating prototypes is, I think, a product of this misunderstanding. Under no circumstances would this ever be a superior way of doing anything. The only reason it's used in non-FOSS games is because it's literally the only way to do some things. The thing is, anyone with the skills required to make binary plugins also has the skills to work on OpenMW directly. The knowledge of the programming language is the same or similar. Both must understand the OpenMW codebase fairly well (at least the part they're trying to change). Forking and pull requests are trivial. And both OpenMW and the binary plugin would have to be compiled in order to test the feature in-game. Someone correct me if I'm wrong, but the difference in speed of compilation would also be negligible.
SeaFox wrote:
08 Oct 2018, 23:03
I support OPTIONAL sandboxing. I don't support MANDATORY sandboxiing.
I think we all understand the distinction you're trying to make. And I think we all believe in options as much as possible! I even saw a post from Zini elsewhere saying more options are generally better! It's just that the option of sandboxing is tacitly the same as having no sandboxing at all. Some people have argued that "just because some stupid people will screw things up doesn't mean we should block other people from using it". But this is not precisely what we're trying to do. We have plenty of settings people can change that will fuck up their setup, and they do! We (generally) try to help as best we can, but explaining for the 10th time that trying to load every object in a 20 cell radius will lower your FPS gets tedious, so sorry if we come off a little short sometimes :)

The difference is that all it takes is for one or two mods that require some unsandboxed stuff to become popular to royally screw us over. Forking and pull requests may be just as easy technically to make as unsandboxed plugins, but they are a bit slower because they require someone else to review them. This means it is a certainty mods that use these will exist if allowed. It is also a certainty that some people will just breeze through any warnings, regardless of how dire. Now if one of these mods is also popular, exponentially more people will want to use it. Just think of something like SkyUI for Skyrim. People categorically refused to play the 64-bit update because they couldn't play with SkyUI until SKSE had been updated for the new engine. If a mods gains this level of popularity, there isn't really anything we can do to aid in supporting it, and a ton of potential issues that are likely to occur because of it. Since we're the new kid on the block, literally everything that goes wrong with anything can and does get blamed on us. We don't have the time and energy for that kind of negativity.

The devs are not willing to take that kind of risk unless there are some very good reasons to do so, and so far, we have still been given none. Not a single concrete example has been presented for reasonable discussion in this whole thread (this is directed at others more than yourself). I'm not saying this to be argumentative, but the circular discussions are getting us nowhere. I want you to feel like we're taking you seriously and hope that we are allaying some or all of your concerns. And if you don't understand how something works, just ask! Most of our devs will take the time to answer questions asked in good faith.

User avatar
Ravenwing
Posts: 262
Joined: 02 Jan 2016, 02:51

Re: The new MWSE-Lua interface

Post by Ravenwing » 09 Oct 2018, 07:31

Jemolk wrote:
08 Oct 2018, 07:42
I just wish this eternity would hurry up and pass already.
Just because we understand the need to take things slow to keep things organized doesn't mean we have to be happy about it! :lol:
AnyOldName3 wrote:
09 Oct 2018, 03:14
A native code plugin...
Do you (or others) mind explaining a little bit more what exactly you mean by a native code plugin? Can you also say what precisely sandboxing will block or rather, what unsandboxing will allow over a sandboxed option. I think I understand for the most part but want to make sure I understand when I'm explaining elsewhere. Does sandboxing keep you from accessing just system files? Third party programs? Both? The discussion on this thread has been somewhat... disjointed in it's explanations.

Also, if there are any inaccuracies in my explanations, please correct them!

User avatar
psi29a
Posts: 4259
Joined: 29 Sep 2011, 10:13
Github profile: https://github.com/psi29a/
Contact:

Re: The new MWSE-Lua interface

Post by psi29a » 09 Oct 2018, 08:16

We're working on that now, I stated earlier that we're going to make a post explaining and update the FAQ accordingly.

The problem is pinning down the words to mean the same thing for everyone.

Modders create 'mods' these mods can also be called plugins. Most, if not all mods, use scripting of some form. There are third-party binaries that people call mods, like MGE and MWSE, that are 'native code' meaning they are executables just like Morrowind.exe with libraries that are injected into Morrowind.exe to extend functionality giving modders additional scripting capabilities.

Now, correct me if I'm wrong... or using the wrong terminology, because we really want to nail this, as far as I'm aware, MWSE's new scripting extensions are in Lua and the Lua interpreter that they are using is unfettered, meaning _not_ sandboxed so that any language feature of Lua is available to the modder, including unlimited access and privileges are available to the user who is running MWSE.

This is both a strength and a weakness. The modder is free to use whatever Lua has to offer in addition to what MWSE chooses to expose to Lua for modders to play with. However, if there was ever a mistake or a bug, the onus is on the end-user for installing a mod that got a relative path wrong somewhere and started deleting the wrong files which it can totally do because unfettered Lua has direct access to everything the user has access to.

OpenMW is going to do the same thing as MWSE, exposing the internals of OpenMW to Lua for modders to use in the form of 'newscript' and then a limited version of Lua that is sandboxed to protect the user and OpenMW from a misbehaving mod. That doesn't mean that mods are malicious, but shit happens. If we can't even trust multi-billion dollar companies to get it right, with updates that delete files from your Documents directory, then it is safe to assume that all humans can make mistakes. See: https://arstechnica.com/gadgets/2018/10 ... -loss-bug/ So erring on the side of caution, having a reasonable best-effort way to handle mods in a safe, sane and stable way isn't asking for much.

We would even encourage MWSE to add a bit of protection themselves. That is sandboxing.

Now providing modders with ways to do things on the filesystem or with networking is totally possible but OpenMW has to provide those in a safe way and if Lua and Lua based plugins (additional Lua libraries, think like "#import something" from Python) provide this capability, we can include those as well.

Allowing third-party libraries into OpenMW via mods was something that OpenMW has always tried to avoid, by virtue of the fact that unlike Morrowind.exe, OpenMW is an open-source project. If there is functionality that needs to be added, it can be requested and implemented by those willing and able to do so. If modders want to get their hands dirty, they are more than welcome but that isn't the point and, at no point, are we asking modders to learn C++, but to articulate their needs/wants in feature requests so that we have something to implement.

Does this help?

User avatar
AnyOldName3
Posts: 1123
Joined: 26 Nov 2015, 03:25

Re: The new MWSE-Lua interface

Post by AnyOldName3 » 09 Oct 2018, 13:16

The main things you can do with unsandboxed Lua which we won't allow in sandboxed Lua are:
  • Arbitrary network access. If malware intentionally disguises itself as a mod, this is something it'll want to be able to do. If a malware author wants to hijack an existing safe mod, this is the part of the mod it will target. We might be able to figure out a safe subset of network commands which can be done without causing other problems, but as far as I'm aware, TES3MP covers all cases where a legitimate mod would need network access (except things like Discord Rich Presence integration, which we're trying to figure out a reasonable system to allow), so this isn't anything to worry about.
  • Arbitrary file access. Either accidentally or intentionally, things can be broken very easily with arbitrary file access. However, anything a mod might want to do can be achieved safely if we allow one directory per mod where anything can be done (potentially with a system to allow mods to let other mods access their directory), so there isn't anything to worry about.
  • Calling native code. This is a major problem for both intentional and unintentional bad behaviour, and also makes things a massive pain to maintain across multiple OSes and when anything gets updated and when two mods both want to change the same system. However, a lot of potential modding power is gained when native code can be called, so if a really good example of something which couldn't be replicated by a well-designed engine-side alternative could be found, it would be a good argument against sandboxing. After months of debate, nothing convincing has been suggested, so there's probably nothing to worry about, but if it turns out there is, we can always backtrack.
There are a bunch of things which can be done with a well-designed sandboxed system which can't be done with Morrowind's built-in sandboxed system, too. Some of the best MWSE-Lua mods would fit this category, such as the UI tweaks, weapon sheathing and glowing windows, both of which aren't done by executing native code, but instead because NullCascade has provided an API to change these parts of the engine from within Lua (if you look, these mods don't include a DLL or .exe, just Lua, Nifs and textures. We can open up parts of the engine to mods without sacrificing the sandbox in a very similar way.
AnyOldName3, Master of Shadows

User avatar
Ravenwing
Posts: 262
Joined: 02 Jan 2016, 02:51

Re: The new MWSE-Lua interface

Post by Ravenwing » 09 Oct 2018, 15:52

Yes, very helpful! This was more or less my impression, but it helps organize it in my mind better. And I did see that you’re planning an FAQ update, which I think is an excellent idea. We have discussed all these things in some capacity at some point, but it is very helpful to put them all in one spot. I think it’s unreasonable to expect people just joining us to piece it all together from the various posts. Plus, it’s nice to have something the devs in general have agreed on being an “official” position.

Also, not all of this is totally relevant, but found it interesting nonetheless:
https://stackoverflow.com/questions/343 ... embly-code

I imagine you’re already covering this, but just in case, it would be good to distinguish that we are planning on implementing all (with exception of unsandboxed features) of MWSE functionality but mods that use MWSE may not be compatible out of the box. The compatibility of such will be dependent on volunteers to implement. At least that is my current understanding and the question most people will want answered.

Equinox
Posts: 15
Joined: 31 Jul 2018, 20:13

Re: The new MWSE-Lua interface

Post by Equinox » 15 Oct 2018, 13:43

The way i see this, openmw might have 2 forks after version 1.0.

The main fork(sandbox, Zini)
And
The old school modders fork(no sandbox,MWSE)

Post Reply

Who is online

Users browsing this forum: Baidu [Spider] and 8 guests