The new MWSE-Lua interface

Everything about development and the OpenMW source code.
User avatar
Capostrophic
Posts: 794
Joined: 22 Feb 2016, 20:32

Re: The new MWSE-Lua interface

Post by Capostrophic »

and will be protected from binary plugins.
They won't use such plugins either way! :|
User avatar
SeaFox
Posts: 34
Joined: 29 Feb 2016, 17:30

Re: The new MWSE-Lua interface

Post by SeaFox »

Capostrophic wrote: 06 Oct 2018, 23:17
and will be protected from binary plugins.
They won't use such plugins either way! :|
Not sure what your point is.
User avatar
Capostrophic
Posts: 794
Joined: 22 Feb 2016, 20:32

Re: The new MWSE-Lua interface

Post by Capostrophic »

I reiterate upon a sandboxing toggle being completely useless.
User avatar
SeaFox
Posts: 34
Joined: 29 Feb 2016, 17:30

Re: The new MWSE-Lua interface

Post by SeaFox »

Capostrophic wrote: 06 Oct 2018, 23:25 I reiterate upon a sandboxing toggle being completely useless.
No it's not. Because the purpose of sandboxing is to protect your computer from malware. Malware is software created by malicious hackers to do nasty things to you. The whole purpose of sandboxing in OpenMW is thus to protect users from malware in your mods. Many people want such protection, but others do not. They are willing to take a small risk so that they can make use of mods with binary plugins, which can be very powerful.
Chris
Posts: 1625
Joined: 04 Sep 2011, 08:33

Re: The new MWSE-Lua interface

Post by Chris »

SeaFox wrote: 06 Oct 2018, 23:38 No it's not. Because the purpose of sandboxing is to protect your computer from malware.
That's only one (actually small) facet. And while it's certainly an issue to take seriously, there are more issues than that. To go over it again:

Compatibility. When you have native code, it can rely on whatever it wants from the system. Make a call to CoCreateInstance, and suddenly it's Windows-only, and no Linux, Mac, or Android system can use the plugin. Make some DX12 call, and it's Windows 10-only, no Win7 or Win8 support. Such plugins would be distributed as shared libraries, which inherently embed platform-specific data and code, meaning even if you personally don't make any system-specific calls, you still have to build and provide multiple shared libs, one for each platform you want to support. You can also look forward to different CPUs. A plugin for 32-bit OpenMW and a plugin for 64-bit OpenMW, for each supported OS. Then you can look forward to different CPUs, some OSs supporting ARM (32-bit and 64-bit flavors).

With sandboxed-only code, we can ensure a script that works on one system can work on the others, no extra work on anyone's part to support. The same mod and same code would work everywhere. But as many other projects have shown, most people have a habit of just targeting the system they use which is likely also the most widely used, because it's easier to use what they're already familiar with (for most developers, non-portable Windows code) than to look for or request something new. So even though there are people using other systems, they'd be locked out from using those mods.

Stability. When the engine oversees everything a mod does, it can ensure correctness, warn of and workaround problems, and just generally handle errors gracefully. When a mod can call out to native code, the engine has no say in anything that goes on. The mod may work 95% of the time, so it's not a deal breaker but has an annoying tic. Add another mod that also works 95% of the time, and another. Then people complain how the engine is so unstable, while the engine can't do anything to fix the problems because it has no control over what those mods are doing.

Forward compatibility. Make a non-sandboxed plugin now, you can obviously only target systems that exist now because you can only build for systems that exist. When mods only use sandboxed code, any platform OpenMW itself can be ported to in the future will automatically inherit all existing mods. Even if the modder is long-gone, no changes are needed to the mod to support new systems.

Backwards compatibility. Systems aren't static, and code is not always bug-free. It's not hard to find cases where some code works while inadvertently relying on a bug or undefined behavior in the system. The system updates, and even though neither OpenMW or the mod changed, the plugin stops working because the undefined behavior changed. With sandboxed code, only OpenMW is responsible for ensuring the plugin works, so if a mod stops working, we can pinpoint what in OpenMW changed to make it stop working, and provide a built-in fix or some other workaround.

So 20 years down the line, when you want to revisit that old mod you remembered having a lot of fun with, you can be relatively sure it'll work regardless of all the system and engine updates that've occurred since the mod was last updated. And if there is an issue with the mod, you can report it to whoever's working on the engine to get it fixed and make it work again.

Or if you're getting fed up with whatever OS you're currently on, you don't have to be concerned about your mods being compatible with other platforms you're thinking about changing to.

Or if you hear about an awesome new mod coming out, you don't have to wonder if it'll support your system.
User avatar
SeaFox
Posts: 34
Joined: 29 Feb 2016, 17:30

Re: The new MWSE-Lua interface

Post by SeaFox »

Chris wrote: 07 Oct 2018, 00:45
SeaFox wrote: 06 Oct 2018, 23:38 No it's not. Because the purpose of sandboxing is to protect your computer from malware.
there are more issues than that.
Perhaps. But none of the issues that you list change my argument that an option should exist for disabling sandboxing for those who wish to do so.
User avatar
AnyOldName3
Posts: 2666
Joined: 26 Nov 2015, 03:25

Re: The new MWSE-Lua interface

Post by AnyOldName3 »

If you want sandboxing, then you can simply just not download any mods which require unsandboxed access (provided they're labelled) and get exactly the same effect. Having an option to disable sandboxing is functionally identical and has all of the same problems as having no sandboxing. Your average not-particularly-savvy user is just going to see 'make sure to change this ini setting' in the installation instructions for a mod and not consider that there's a reason the setting started off disabled, just like how every Fallout 4 mod says to make sure you've changed a particular ini setting before installing it (as Bethesda decided that loading mods should be disabled in the ini by default) and users do that.

The discussion should be moot at this stage anyway - if sandboxing genuinely causes problems, it can be removed fairly easily, but if it's not there, it can never be added as that's breaking backwards compatibility and we'll lose compatibility with a load of mods which used native code even though they didn't strictly need to and also never be allowed to put that update on any debian-based Linux.
mychaelo
Posts: 10
Joined: 08 Aug 2014, 10:55

Re: The new MWSE-Lua interface

Post by mychaelo »

So I was bored on a train and read this entire thread, and I didn't see one good example of why would you need to turn off the sand box in OpenMW-lua. I get why you would need to do that in Morrowind - you can't do crap about it since it's an untouchable blob of copyrighted bits, so instead you hack around, hook it up and beat into submission.

But why would you program a C code around OpenMW instead of pull-requesting the API stuff you need, or feature-request it, and then code the stuff that uses it in LUA? Isn't writing C for existing codebase a lot easier than making a renderer hook from scratch?

Give me one example of a mod that cannot be done in openmw-lua, even with an augmented API
User avatar
Ravenwing
Posts: 335
Joined: 02 Jan 2016, 02:51

Re: The new MWSE-Lua interface

Post by Ravenwing »

Well, at least only one of these threads has cropped up in the time I've been away :(
mychaelo wrote: 07 Oct 2018, 12:42 So I was bored on a train and read this entire thread, and I didn't see one good example of why would you need to turn off the sand box in OpenMW-lua.
This is literally what happened to me lol!

@SeaFox, I'm going to talk directly to you as I feel like you asked your first several questions in good faith, but have been led astray in the ensuing drama.

The strife between NullCascade and Zini boils down to a fundamental difference of design philosophy: get modders as many tools as possible, as soon as possible using any means necessary VS spend a little more time and energy planning a more rigorous and sustainable piece of software. Or in other words, users have waited this long for interesting features, so let's not make them wait any longer VS users have waited this long for interesting features, they can wait a little bit longer.

The first is a totally valid and understandable point of view, however it is an unsustainable engineering practice that will always eventually trip you up and screw up development and functionality. NullCascade is no idiot, he knows this and has admitted himself that eventually OpenMW will overtake anything that MWSE is capable. The keyword for him is eventually, hence why he continues to focus on MWSE.

The sandboxing decision really comes down to the fact that there are several good reasons for having sandboxing, and essentially no concrete reason for not having it. The reasons for were best explained by Chris:
Chris wrote: 07 Oct 2018, 00:45 Compatibility. When you have native code, it can rely on whatever it wants from the system. Make a call to CoCreateInstance, and suddenly it's Windows-only, and no Linux, Mac, or Android system can use the plugin. Make some DX12 call, and it's Windows 10-only, no Win7 or Win8 support. Such plugins would be distributed as shared libraries, which inherently embed platform-specific data and code, meaning even if you personally don't make any system-specific calls, you still have to build and provide multiple shared libs, one for each platform you want to support. You can also look forward to different CPUs. A plugin for 32-bit OpenMW and a plugin for 64-bit OpenMW, for each supported OS. Then you can look forward to different CPUs, some OSs supporting ARM (32-bit and 64-bit flavors).

With sandboxed-only code, we can ensure a script that works on one system can work on the others, no extra work on anyone's part to support. The same mod and same code would work everywhere. But as many other projects have shown, most people have a habit of just targeting the system they use which is likely also the most widely used, because it's easier to use what they're already familiar with (for most developers, non-portable Windows code) than to look for or request something new. So even though there are people using other systems, they'd be locked out from using those mods.

Stability. When the engine oversees everything a mod does, it can ensure correctness, warn of and workaround problems, and just generally handle errors gracefully. When a mod can call out to native code, the engine has no say in anything that goes on. The mod may work 95% of the time, so it's not a deal breaker but has an annoying tic. Add another mod that also works 95% of the time, and another. Then people complain how the engine is so unstable, while the engine can't do anything to fix the problems because it has no control over what those mods are doing.

Forward compatibility. Make a non-sandboxed plugin now, you can obviously only target systems that exist now because you can only build for systems that exist. When mods only use sandboxed code, any platform OpenMW itself can be ported to in the future will automatically inherit all existing mods. Even if the modder is long-gone, no changes are needed to the mod to support new systems.

Backwards compatibility. Systems aren't static, and code is not always bug-free. It's not hard to find cases where some code works while inadvertently relying on a bug or undefined behavior in the system. The system updates, and even though neither OpenMW or the mod changed, the plugin stops working because the undefined behavior changed. With sandboxed code, only OpenMW is responsible for ensuring the plugin works, so if a mod stops working, we can pinpoint what in OpenMW changed to make it stop working, and provide a built-in fix or some other workaround.

So 20 years down the line, when you want to revisit that old mod you remembered having a lot of fun with, you can be relatively sure it'll work regardless of all the system and engine updates that've occurred since the mod was last updated. And if there is an issue with the mod, you can report it to whoever's working on the engine to get it fixed and make it work again.

Or if you're getting fed up with whatever OS you're currently on, you don't have to be concerned about your mods being compatible with other platforms you're thinking about changing to.

Or if you hear about an awesome new mod coming out, you don't have to wonder if it'll support your system.
I'm confused by your response that none of these are enough to persuade you that optional sandboxing is an unacceptable solution. These are all very real and significant issues from a user and modder standpoint. I believe the distinction between optional sandboxing and not sandboxing at all has been discussed enough and I really have nothing to add as I don't know how to be more convincing. Having an option means we sanction both options and for the above listed reasons we don't believe in not sandboxing, therefore we can't in good faith sanction it at this time.

The reasons against sandboxing essentially boil down to:
1) mod authors won't be able to do literally anything they could possibly want, and
2) mod development is slower if we have to request or create new features in the engine itself.

The first point is philosophical at this stage, and the reason we keep saying that we haven't seen a single good example that requires, and is best dealt with, using unsandboxed mods.
The second point is a legitimate concern to have if speed of mod production is your main/only concern. However, as I state above, this isn't really a sustainable design philosophy and shouldn't be desired above a more rigorous solution. Fundamentally, this really only offsets the labor anyway. Either the feature is implemented universally in the engine, allowing other modders to benefit from the framework as well, or the modder has to cobble together something for their mod only. We aren't even to the point where sandboxed/unsandboxed matters, and people are this concerned about waiting a few months for a hypothetical feature to be implemented!?!? I'm sorry, but this is all fear mongering.

A lot of past drama stems from the fact that Zini has categorically blocked many feature and pull requests that don't fall within the established 1.0 goals. Sometimes these decisions seem arbitrary, but the point is to prevent feature creep, a problem that has plagued product development in every industry across time. If he doesn't limit scope changes, the project will never hit the 1.0 milestone. This is literally the job of a project manager.

This is relevant, because it's one of several reasons more MWSE functionality has not been implemented yet. Past the 1.0 mark, there are still some first order of business items that need to be focused on, so he will probably still block a lot of features up through the Grand Dehardcoding. Once this is completed however, we finally have a solid game engine and everyone will be much more open to playing around with implementing new and interesting features.

Zini and the rest of the devs are much less capricious than NullCascade and others would have you believe. The decision to use Lua being the most recent and obvious example of him being convinced to move the project in a direction he did not originally want to.

There has been a lot of fear mongering in regards to features blocked by sandboxing the scripting language. You seem to have been led to believe that lots of really cool things won't be possible. This just isn't true. As davidcernat said:
davidcernat wrote: 06 Oct 2018, 00:54 Those can be done just fine in sandboxed Lua with a few script functions that let you attach or replace meshes, i.e. you added some mesh-related C++ code to MWSE that could be called from your Lua scripts and now OpenMW can add some somewhat similar (but much more elegant) C++ code that can be called from its Lua scripts to achieve the same functionality.

Similarly, the mod this thread was initially about (Tamrielic Lore Tooltips) was made possible by your addition of GUI-related C++ code to MWSE that allowed for GUI manipulation in your Lua scripts, which OpenMW can have its own equivalent C++ code for that can be called from its own Lua scripts.

Can you come up with some actual examples that really require a non-sandboxed environment? The examples I remember from you seemed so lightweight so as to make your whole obsession with the topic seem a little silly.
I think you understand what sandboxing is blocking people from doing based on your more recent posts, so I'm confused as to what you think those blocked actions will allow people to do that we aren't already planning on implementing. If it's GUI stuff, it's better to have a framework setup within the engine for modders to use. If you want to be able to tie into online stuff, it's not difficult to implement a safe and limited framework for that. There are lots of really good reasons to not let modders mess around with things willy-nilly and so far no compelling reasons to allow it.

Moreover, as AnyOldName3 says, we can always decide to un-sandbox if we realize we've fucked up, but you can't do it backwards. If we get to that stage and mean ol' Zini-the-Gatekeeper-of-All-Good-Ideas can't be convinced, then you can worry about forking the project. No need to act like it's my-way-or-the-highway until then.


I also wanted to point out that Blender plugins are much more akin to OpenMW-CS plugins. Something that we're planning on allowing with a much more open framework than the sandboxed Lua scripting (unless things have changed significantly while I've been away). So no need to worry there as well.
Petethegoat
Posts: 3
Joined: 08 Dec 2017, 04:29

Re: The new MWSE-Lua interface

Post by Petethegoat »

I think there's a lot of good reasons in favour of exclusive sandboxing.

But there's this assumption that anything a modder wants to use that isn't supported or exposed in the engine, will and should be integrated into the core engine.

It seems really obvious to me that this won't be what happens in practice, as some desirable features for modders are just too specialized and specific to be integrated while keeping the engine maintainable. While jiggling titty physics might be more generally useful for procedurally animating cloth or making wobbling netch jelly, it's not obvious to me that it'd be merged into the main branch just because a curvy body mod wants it. Maybe I'm wrong, and even niche stuff will eventually be merged or created, and then maintained, but it seems like a big presumption to make.
Post Reply