The new MWSE-Lua interface

Everything about development and the OpenMW source code.
User avatar
Ravenwing
Posts: 266
Joined: 02 Jan 2016, 02:51

Re: The new MWSE-Lua interface

Post by Ravenwing » 08 Oct 2018, 00:07

Well, at least only one of these threads has cropped up in the time I've been away :(
mychaelo wrote:
07 Oct 2018, 12:42
So I was bored on a train and read this entire thread, and I didn't see one good example of why would you need to turn off the sand box in OpenMW-lua.
This is literally what happened to me lol!

@SeaFox, I'm going to talk directly to you as I feel like you asked your first several questions in good faith, but have been led astray in the ensuing drama.

The strife between NullCascade and Zini boils down to a fundamental difference of design philosophy: get modders as many tools as possible, as soon as possible using any means necessary VS spend a little more time and energy planning a more rigorous and sustainable piece of software. Or in other words, users have waited this long for interesting features, so let's not make them wait any longer VS users have waited this long for interesting features, they can wait a little bit longer.

The first is a totally valid and understandable point of view, however it is an unsustainable engineering practice that will always eventually trip you up and screw up development and functionality. NullCascade is no idiot, he knows this and has admitted himself that eventually OpenMW will overtake anything that MWSE is capable. The keyword for him is eventually, hence why he continues to focus on MWSE.

The sandboxing decision really comes down to the fact that there are several good reasons for having sandboxing, and essentially no concrete reason for not having it. The reasons for were best explained by Chris:
Chris wrote:
07 Oct 2018, 00:45
Compatibility. When you have native code, it can rely on whatever it wants from the system. Make a call to CoCreateInstance, and suddenly it's Windows-only, and no Linux, Mac, or Android system can use the plugin. Make some DX12 call, and it's Windows 10-only, no Win7 or Win8 support. Such plugins would be distributed as shared libraries, which inherently embed platform-specific data and code, meaning even if you personally don't make any system-specific calls, you still have to build and provide multiple shared libs, one for each platform you want to support. You can also look forward to different CPUs. A plugin for 32-bit OpenMW and a plugin for 64-bit OpenMW, for each supported OS. Then you can look forward to different CPUs, some OSs supporting ARM (32-bit and 64-bit flavors).

With sandboxed-only code, we can ensure a script that works on one system can work on the others, no extra work on anyone's part to support. The same mod and same code would work everywhere. But as many other projects have shown, most people have a habit of just targeting the system they use which is likely also the most widely used, because it's easier to use what they're already familiar with (for most developers, non-portable Windows code) than to look for or request something new. So even though there are people using other systems, they'd be locked out from using those mods.

Stability. When the engine oversees everything a mod does, it can ensure correctness, warn of and workaround problems, and just generally handle errors gracefully. When a mod can call out to native code, the engine has no say in anything that goes on. The mod may work 95% of the time, so it's not a deal breaker but has an annoying tic. Add another mod that also works 95% of the time, and another. Then people complain how the engine is so unstable, while the engine can't do anything to fix the problems because it has no control over what those mods are doing.

Forward compatibility. Make a non-sandboxed plugin now, you can obviously only target systems that exist now because you can only build for systems that exist. When mods only use sandboxed code, any platform OpenMW itself can be ported to in the future will automatically inherit all existing mods. Even if the modder is long-gone, no changes are needed to the mod to support new systems.

Backwards compatibility. Systems aren't static, and code is not always bug-free. It's not hard to find cases where some code works while inadvertently relying on a bug or undefined behavior in the system. The system updates, and even though neither OpenMW or the mod changed, the plugin stops working because the undefined behavior changed. With sandboxed code, only OpenMW is responsible for ensuring the plugin works, so if a mod stops working, we can pinpoint what in OpenMW changed to make it stop working, and provide a built-in fix or some other workaround.

So 20 years down the line, when you want to revisit that old mod you remembered having a lot of fun with, you can be relatively sure it'll work regardless of all the system and engine updates that've occurred since the mod was last updated. And if there is an issue with the mod, you can report it to whoever's working on the engine to get it fixed and make it work again.

Or if you're getting fed up with whatever OS you're currently on, you don't have to be concerned about your mods being compatible with other platforms you're thinking about changing to.

Or if you hear about an awesome new mod coming out, you don't have to wonder if it'll support your system.
I'm confused by your response that none of these are enough to persuade you that optional sandboxing is an unacceptable solution. These are all very real and significant issues from a user and modder standpoint. I believe the distinction between optional sandboxing and not sandboxing at all has been discussed enough and I really have nothing to add as I don't know how to be more convincing. Having an option means we sanction both options and for the above listed reasons we don't believe in not sandboxing, therefore we can't in good faith sanction it at this time.

The reasons against sandboxing essentially boil down to:
1) mod authors won't be able to do literally anything they could possibly want, and
2) mod development is slower if we have to request or create new features in the engine itself.

The first point is philosophical at this stage, and the reason we keep saying that we haven't seen a single good example that requires, and is best dealt with, using unsandboxed mods.
The second point is a legitimate concern to have if speed of mod production is your main/only concern. However, as I state above, this isn't really a sustainable design philosophy and shouldn't be desired above a more rigorous solution. Fundamentally, this really only offsets the labor anyway. Either the feature is implemented universally in the engine, allowing other modders to benefit from the framework as well, or the modder has to cobble together something for their mod only. We aren't even to the point where sandboxed/unsandboxed matters, and people are this concerned about waiting a few months for a hypothetical feature to be implemented!?!? I'm sorry, but this is all fear mongering.

A lot of past drama stems from the fact that Zini has categorically blocked many feature and pull requests that don't fall within the established 1.0 goals. Sometimes these decisions seem arbitrary, but the point is to prevent feature creep, a problem that has plagued product development in every industry across time. If he doesn't limit scope changes, the project will never hit the 1.0 milestone. This is literally the job of a project manager.

This is relevant, because it's one of several reasons more MWSE functionality has not been implemented yet. Past the 1.0 mark, there are still some first order of business items that need to be focused on, so he will probably still block a lot of features up through the Grand Dehardcoding. Once this is completed however, we finally have a solid game engine and everyone will be much more open to playing around with implementing new and interesting features.

Zini and the rest of the devs are much less capricious than NullCascade and others would have you believe. The decision to use Lua being the most recent and obvious example of him being convinced to move the project in a direction he did not originally want to.

There has been a lot of fear mongering in regards to features blocked by sandboxing the scripting language. You seem to have been led to believe that lots of really cool things won't be possible. This just isn't true. As davidcernat said:
davidcernat wrote:
06 Oct 2018, 00:54
Those can be done just fine in sandboxed Lua with a few script functions that let you attach or replace meshes, i.e. you added some mesh-related C++ code to MWSE that could be called from your Lua scripts and now OpenMW can add some somewhat similar (but much more elegant) C++ code that can be called from its Lua scripts to achieve the same functionality.

Similarly, the mod this thread was initially about (Tamrielic Lore Tooltips) was made possible by your addition of GUI-related C++ code to MWSE that allowed for GUI manipulation in your Lua scripts, which OpenMW can have its own equivalent C++ code for that can be called from its own Lua scripts.

Can you come up with some actual examples that really require a non-sandboxed environment? The examples I remember from you seemed so lightweight so as to make your whole obsession with the topic seem a little silly.
I think you understand what sandboxing is blocking people from doing based on your more recent posts, so I'm confused as to what you think those blocked actions will allow people to do that we aren't already planning on implementing. If it's GUI stuff, it's better to have a framework setup within the engine for modders to use. If you want to be able to tie into online stuff, it's not difficult to implement a safe and limited framework for that. There are lots of really good reasons to not let modders mess around with things willy-nilly and so far no compelling reasons to allow it.

Moreover, as AnyOldName3 says, we can always decide to un-sandbox if we realize we've fucked up, but you can't do it backwards. If we get to that stage and mean ol' Zini-the-Gatekeeper-of-All-Good-Ideas can't be convinced, then you can worry about forking the project. No need to act like it's my-way-or-the-highway until then.


I also wanted to point out that Blender plugins are much more akin to OpenMW-CS plugins. Something that we're planning on allowing with a much more open framework than the sandboxed Lua scripting (unless things have changed significantly while I've been away). So no need to worry there as well.

Petethegoat
Posts: 2
Joined: 08 Dec 2017, 04:29
Github profile: https://github.com/Petethegoat

Re: The new MWSE-Lua interface

Post by Petethegoat » 08 Oct 2018, 01:00

I think there's a lot of good reasons in favour of exclusive sandboxing.

But there's this assumption that anything a modder wants to use that isn't supported or exposed in the engine, will and should be integrated into the core engine.

It seems really obvious to me that this won't be what happens in practice, as some desirable features for modders are just too specialized and specific to be integrated while keeping the engine maintainable. While jiggling titty physics might be more generally useful for procedurally animating cloth or making wobbling netch jelly, it's not obvious to me that it'd be merged into the main branch just because a curvy body mod wants it. Maybe I'm wrong, and even niche stuff will eventually be merged or created, and then maintained, but it seems like a big presumption to make.

User avatar
AnyOldName3
Posts: 1131
Joined: 26 Nov 2015, 03:25

Re: The new MWSE-Lua interface

Post by AnyOldName3 » 08 Oct 2018, 01:24

I think there's a pretty good chance that we'll have a system for attaching physics objects to character meshes eventually - if we don't all die of old age by the time we add Fallout 4 support, we have to do it in order to support the hair physics, and then non-hair body parts can just use that system. I think you've accidentally picked an example where the potential fan backlash and technical justification both mandate that the feature happens eventually.

There's a good chance people will want systems in place to mod things which are unfeasible to maintain, but in reality, they're likely to be pretty few and far between - bear in mind that after Morrowind's done and things have been de-hardcoded, we're hoping that all subsequent Bethesda Gamebryo/Creation Engine games can eventually be supported, ideally with all differences between their engines managed by scripts. That's almost certainly going to force us to open the engine up to modification in ways which surpass almost anything any modder could need.
AnyOldName3, Master of Shadows

User avatar
Ravenwing
Posts: 266
Joined: 02 Jan 2016, 02:51

Re: The new MWSE-Lua interface

Post by Ravenwing » 08 Oct 2018, 05:32

Petethegoat wrote:
08 Oct 2018, 01:00
But there's this assumption that anything a modder wants to use that isn't supported or exposed in the engine, will and should be integrated into the core engine.

It seems really obvious to me that this won't be what happens in practice, as some desirable features for modders are just too specialized and specific to be integrated while keeping the engine maintainable. While jiggling titty physics might be more generally useful for procedurally animating cloth or making wobbling netch jelly, it's not obvious to me that it'd be merged into the main branch just because a curvy body mod wants it.
That's a reasonable concern to have and I can see how some of our arguments have sounded that way. I won't say there is absolutely no possibility of this ever happening, however, I think even most niche/specialized needs are actually things that require a much more generalized solution. Your example of the jiggling titties is the niche need, with (apparently) physics objects to meshes being the general solution. I'm sure the vast majority of seemingly niche needs can actually be solved with generalized solutions. As I say here:
Ravenwing wrote:
08 Oct 2018, 00:07
Either the feature is implemented universally in the engine, allowing other modders to benefit from the framework as well, or the modder has to cobble together something for their mod only.
So the best approach is for someone, either the mod author or the devs, to submit a pull request implementing physics objects attached to meshes. Now any modder can use the supported framework to make jiggling titties, jangling wangs, long flowing hair, or just cool fabric banners, whatever floats their boat. There has been some discussion that modders shouldn't be expected to know how to modify the code base for a pull request. The thing is, if they were going to go with the unsandboxed route, they'd already have to have the level of knowledge required to implement their own separate feature. If they don't, they were already at the mercy of someone else doing it for them, so it's really a moot point.

Now for the cases where the devs don't really feel something can be reasonably maintained.

First, it's the modder's responsibility to be at least somewhat convincing in asking for a new feature if they can't make it themselves.
If Mr. Jiggling Titties just goes on the forum and shouts, "I WANT JIGGLING TITTIES!" And the devs say, "lol y tho." And Mr. JT says, "BECAUSE I WANT THEM! ALSO ZINI IS AN EVIL DICTATOR AND YOU GUYS ARE ALWAYS MEAN" I think the devs have every right to say, "nah, you're an idiot."
If instead he says "Yo, I think jiggling titties would be cool, plus we could make all kinds of jiggly other things" the devs are much more likely to realize there's a nifty general solution and help the guy out.

Second, if, even after a thorough and convincing argument about how great a feature would be, the devs still say they won't merge something, forking isn't the most horrible other thing to happen. Look at TES3MP, it's a fork of OpenMW. Multiplayer is a specifically post-1.0 thing that Zini would never allow to merge as it stands. But davidcernat gets along well with the core project and keeps his branch updated with each version of OpenMW proper. Now it's influencing the main project (a large reason we decided on Lua) and has even become our goal to merge for version 2.0 (I believe). This is perhaps not the best example because multiplayer is obviously a seriously cool, highly desired feature. It is also not ideal to have people making forks left and right just to allow their pet mods, because ain't nobody got time for that. However, this is a great way to show rather than tell the devs, hey look, I've got a cool thing going, maybe this should be merged into the master. Even if the mod authors don't want to support the mod themselves, the mod author who forked it can do the maintenance they'd have to do anyway for a binary solution and just keep the two projects synced.

Once we actually get to the point where people want to be merging things that would best be left out of the core project, it's entirely possible that Zini will want a more generalized plugin system that makes everyone happy. If he doesn't he probably has a damn good reason for why and it would be a much more productive discussion at that time.

User avatar
Jemolk
Posts: 106
Joined: 20 Mar 2017, 02:12
Location: Sadrith Mora

Re: The new MWSE-Lua interface

Post by Jemolk » 08 Oct 2018, 07:42

Thanks for the calm reasoning, guys. I don't know about any of the others who ended up contributing to this becoming a firestorm, but I really appreciate it. And taking a step or two back, I can see why I felt that way, and also why it's not exactly useful or even reasonable. I'm really stressed right now, and due in no small part to that, it feels like things are taking an absolute eternity to me. But I can see the merit in what you all are saying. I just wish this eternity would hurry up and pass already. Part of it also is that I just don't have the skill to help, despite really, really wanting to, and with my current classes, I don't feel like I have enough free time to learn. :(

User avatar
psi29a
Posts: 4275
Joined: 29 Sep 2011, 10:13
Github profile: https://github.com/psi29a/
Contact:

Re: The new MWSE-Lua interface

Post by psi29a » 08 Oct 2018, 08:46

Jemolk wrote:
08 Oct 2018, 07:42
I'm really stressed right now, and due in no small part to that, it feels like things are taking an absolute eternity to me. But I can see the merit in what you all are saying. I just wish this eternity would hurry up and pass already. Part of it also is that I just don't have the skill to help, despite really, really wanting to, and with my current classes, I don't feel like I have enough free time to learn. :(
Many people feel this way. Some of us dealing with classes, work, family, kids, commitments and more which limit people in contributing more time to whatever project they want to work on. So having these kinds of discussions, while important in trying to address the issues, can sometime draw away from everyone's time which would have been better spend actually working on the project.

The good news is, we'll be releasing a statement and updating the FAQ accordingly to help users, modders and developers with where OpenMW is going, how they can help and how we can help them.

That way, when future discussions happen, no one has to repeat themselves and we can jump right into the technical issues and how best to deal with them. Players, modders, game makers and content creators are 1st class citizens, they are the people we want to help create great things. :)

As cliche as that sounds, help us help you. We have a commitment to ourselves to try to help, where we can, be kind and patient. We ask that also in return. If we feel that that is being abused, then we'll stop providing help. This is our hobby too. We'd hope that this would go without saying, but sometimes it needs to be reiterated.

silentthief
Posts: 299
Joined: 18 Apr 2013, 01:20
Location: Currently traversing the Ascadian Isles

Re: The new MWSE-Lua interface

Post by silentthief » 08 Oct 2018, 15:34

psi29a wrote:
08 Oct 2018, 08:46
As cliche as that sounds, help us help you. We have a commitment to ourselves to try to help, where we can, be kind and patient. We ask that also in return. If we feel that that is being abused, then we'll stop providing help. This is our hobby too. We'd hope that this would go without saying, but sometimes it needs to be reiterated.
Best advice right there. I have as much passion and excitement over this project as any - even though I lurk and don't get too much into the debates. I have seen more than one project get halted due to developers deciding that (due to arguments and/or unsatisfied users asking for more features) it simply was more trouble than its worth.

I saw the progress from where this was barely above an ESM world viewer where the character model was a ball that flew around showing the features of Vivec, and none of the NPCs could be interacted with. This was prior to the rewrite from D to C. There have been trolls (and no - under no circumstances should the arguments for or against sandboxing be considered a troll. Passions fly for a game we all love - but we all share the same ultimate goal). There has been people who came and then said after a few weeks "This project is dead". And still - the devs have continued to deliver even when real life is a pita. TY to you devs for continuing with this

ST - the MW fan, the MSWE fan and the openMW fan
"Hurry, hurry! Last boat to Solstheim! Until the next one. Hah-ha-hah."

User avatar
Jemolk
Posts: 106
Joined: 20 Mar 2017, 02:12
Location: Sadrith Mora

Re: The new MWSE-Lua interface

Post by Jemolk » 08 Oct 2018, 15:54

psi29a wrote:
08 Oct 2018, 08:46
As cliche as that sounds, help us help you. We have a commitment to ourselves to try to help, where we can, be kind and patient. We ask that also in return. If we feel that that is being abused, then we'll stop providing help. This is our hobby too. We'd hope that this would go without saying, but sometimes it needs to be reiterated.
Of course! I meant my post by way of explanation rather than excuse. And it's great to hear the reassurances. Thanks. And sorry for adding fuel to that rather...explosive fire. You guys all do great work, and I do appreciate it. Stress just can bring out the worst in me, as there. I do try to do better, but, well.... Obviously it doesn't always work as well as I'd like. Mea culpa.

User avatar
SeaFox
Posts: 34
Joined: 29 Feb 2016, 17:30

Re: The new MWSE-Lua interface

Post by SeaFox » 08 Oct 2018, 23:03

Ravenwing wrote:
08 Oct 2018, 00:07
The strife between NullCascade and Zini boils down to a fundamental difference of design philosophy: get modders as many tools as possible, as soon as possible using any means necessary VS spend a little more time and energy planning a more rigorous and sustainable piece of software.
That seems like an example of a false dichotomy. The end goal of a software project should, as you say, be a rigorously planned and sustainable piece of software. But the process of reaching that goal can, and should, include the use of software prototypes, such as the binary plugins within mods. The use of such sofware prototypes is part of the process of Rapid-application development (RAD). Once the prototype has served it's purpose by acting as a proof of concept, a more rigorous and sustainable piece of software can then be constructed to replace it.
Ravenwing wrote:
08 Oct 2018, 00:07
I'm confused by your response that none of these are enough to persuade you that optional sandboxing is an unacceptable solution. These are all very real and significant issues from a user and modder standpoint.
I support OPTIONAL sandboxing. I don't support MANDATORY sandboxiing.
Ravenwing wrote:
08 Oct 2018, 05:32
It is also not ideal to have people making forks left and right just to allow their pet mods, because ain't nobody got time for that. However, this is a great way to show rather than tell the devs, hey look, I've got a cool thing going, maybe this should be merged into the master. Even if the mod authors don't want to support the mod themselves, the mod author who forked it can do the maintenance they'd have to do anyway for a binary solution and just keep the two projects synced.
You seem to be suggesting that creators of mods should fork the entire OpenMW project whenever they wish to create a software prototype of some new feature which is not possible with the current code base. Binary plugins would be a far superior way of creatiing such prototypes.

Chris
Posts: 1500
Joined: 04 Sep 2011, 08:33

Re: The new MWSE-Lua interface

Post by Chris » 09 Oct 2018, 01:55

SeaFox wrote:
08 Oct 2018, 23:03
That seems like an example of a false dichotomy. The end goal of a software project should, as you say, be a rigorously planned and sustainable piece of software. But the process of reaching that goal can, and should, include the use of software prototypes, such as the binary plugins within mods.
But mods aren't prototypes. A mod can be prototyped, but also has official releases, just like an engine. How would you ensure official releases of mods, which are expected to work for a wide audience well into the future, aren't using native code plugins?

Post Reply

Who is online

Users browsing this forum: Jerome89 and 8 guests