Post 1.0 Planning Update

Anything related to PR, release planning and any other non-technical idea how to move the project forward should be discussed here.
Post Reply
NullCascade
Posts: 121
Joined: 16 Jan 2012, 07:58

Re: Post 1.0 Planning Update

Post by NullCascade »

Zini wrote: 05 Apr 2018, 22:40I would have to look into each of these features individually to answer that. There may be some room for further abstraction. Or not. Needs more research (and not a priority right now).

Unless there is a license problem with a third party API, of course (also needs to be looked at). But this point is independent of the implementation path anyway.
So you don't know the features, but you're confident that they'll be possible? :(

Hey, if you have something up your sleeve that is somehow better than LuaJIT's speed, versatility, and completeness, cool. Lua is built to be sandboxed by default. But it's going to take you years of R&D to turn mwscript into a legitimate competitor to LuaJIT. If you want to reinvent that wheel, all the power to you. But there's a solution out there already. It does amazing things for modding scenes other than OpenMW's. Two other Morrowind-related projects (tes3mp, MWSE) are using it, so there's great synergy for developers. You have one, right here, willing to do the work to make it happen. Bam -- two birds, one stone, no heavy competition from MWSE.

I just don't see why expanding mwscript is the more appealing option for the project.
User avatar
AnyOldName3
Posts: 2668
Joined: 26 Nov 2015, 03:25

Re: Post 1.0 Planning Update

Post by AnyOldName3 »

Zini hasn't said that we'll only ever have MWScript available. You're still conflating the debate of whether or not we let other non-shitty scripting languages be used with the entirely separate issue of whether we let the scripting languages we do support call arbitrary code in written in arbitrary other languages and compiled to an arbitrary binary.

Allowing other scripting languages is very likely and I don't feel that we need to continue debating the merit of the idea in general as making scripting not suck is a very obviously good idea with no real downsides. Regardless of any discussion here, something will be done and the results will eventually be good.

Allowing arbitrary external code to be run or access to system APIs like arbitrary file IO and networking is less likely, potentially worth debating, and something that Zini doesn't think is a good idea. In an ideal world, every feature of merit that this would allow could also be implemented by making a PR against the engine and then having someone with merge rights agree that it's a good thing and merging it. The questions are ones like whether anything exists that would be wanted by certain users but seem obviously stupid to developers, would it result in unwanted feature creep and would there be enough such features that users who wanted to do something abnormal would have serious problems versus is letting a modder kill your computer a real risk.
NullCascade
Posts: 121
Joined: 16 Jan 2012, 07:58

Re: Post 1.0 Planning Update

Post by NullCascade »

AnyOldName3 wrote: 05 Apr 2018, 23:26Zini hasn't said that we'll only ever have MWScript available. You're still conflating the debate of whether or not we let other non-shitty scripting languages be used with the entirely separate issue of whether we let the scripting languages we do support call arbitrary code in written in arbitrary other languages and compiled to an arbitrary binary.
Zini hasn't said that we will get that thing either. Unless I'm reading it wrong, it seems like there's some brainstorming and engineering cycles being aimed at improving mwscript, not implementing an alternative? You can say it's very likely, but my concern comes from thinking that it is very unlikely from Zini's posts. He hasn't clarified either way. Can we get clarification?

Arbitrary code can be toggled on and off. Ship the product with FFI and core Lua utils disabled. Lua is sandboxed by default until you opt in to features. That works 100% fine. But let people enable them so that cool new mods can be made.

Again, I've used Lua professionally in a cross-platform engine. It can do everything we need, and more. And I'm super happy to help. I really don't want OpenMW to fail to live up to its name, and to need LibreMW instead.
Chris
Posts: 1625
Joined: 04 Sep 2011, 08:33

Re: Post 1.0 Planning Update

Post by Chris »

NullCascade wrote: 05 Apr 2018, 23:57 Zini hasn't said that we will get that thing either. Unless I'm reading it wrong, it seems like there's some brainstorming and engineering cycles being aimed at improving mwscript, not implementing an alternative?
Things like this are done in steps. It's possible to make improvements to mwscript more quickly and easily than it is to get another scripting language implemented. An extended mwscript also works as an avenue for mods that use vanilla mwscript to add small enhancements easily, since they don't have to redo all their scripts. That doesn't mean another scripting language can't be implemented, but you don't just halt all forward progress as you wait for it.
Arbitrary code can be toggled on and off.
That's no consolation for mods that don't actually need arbitrary code, but use it anyway for.. arbitrary reasons. Or because they want to shortcut ahead and do things without waiting for the engine to get proper built-in support for what they need (so when the engine does support what they need, the mod is still locked behind the arbitrary code execution requirements despite not really needing it to get the desired effect).
NullCascade
Posts: 121
Joined: 16 Jan 2012, 07:58

Re: Post 1.0 Planning Update

Post by NullCascade »

Chris wrote: 06 Apr 2018, 01:05Things like this are done in steps. It's possible to make improvements to mwscript more quickly and easily than it is to get another scripting language implemented. An extended mwscript also works as an avenue for mods that use vanilla mwscript to add small enhancements easily, since they don't have to redo all their scripts. That doesn't mean another scripting language can't be implemented, but you don't just halt all forward progress as you wait for it.
... what? Incrementally working on potentially breaking changes to mwscript over time is easier than doing...

Code: Select all

sol::state lua;
lua.open_libraries( sol::lib::base );
lua.do_script( script->code );
Congrats. There you have a lua script running, with no networking, no file I/O, no system calls, no JIT, no package support, no string support, no math functions, no table functions, no debug functions, no FFI. You can be as bland as you want. (I advocate using sol2 because it's amazing, but you can do raw Lua bindings if you are feeling masochistic/educational. MWSE 2.1 uses sol2, and it looks like tes3mp does as well. It's a matter of hours for me to bind native Morrowind engine data structures, instead of days.)

Never, ever will reinventing the wheel be faster than using a stable and reliable project that has already been shipped.
Chris wrote: 06 Apr 2018, 01:05That's no consolation for mods that don't actually need arbitrary code, but use it anyway for.. arbitrary reasons. Or because they want to shortcut ahead and do things without waiting for the engine to get proper built-in support for what they need (so when the engine does support what they need, the mod is still locked behind the arbitrary code execution requirements despite not really needing it to get the desired effect).
So for every new piece of software that comes out, every new API that people want to call to, every piece of custom code modders want... they have to do a pull request. That's insane. You want the bottleneck of modding to not be the potential of modders, but the relatively extremely small main dev team? What?

Zini said that things like Discord RPC would be possible with his solution. Without third party extensions or bloating Discord into the project for people that may not even want it, I just don't see how that is possible. But if you give modders some freedom, they'll make this project worth using.
User avatar
AnyOldName3
Posts: 2668
Joined: 26 Nov 2015, 03:25

Re: Post 1.0 Planning Update

Post by AnyOldName3 »

Either way, if Zini goes completely mad and doesn't implement necessary features, I'll make a fork of OpenMW which integrates Mono (including its implementation of the Common Language Interface). I'll add an MWScript -> .NET bytecode compiler to the CS so existing stuff will still work and can be called from and can call stuff written in other .NET languages (if we have multiple scripting languages, we better be able to share data between them without caring or necessarily even knowing what the other scripts were written in). I might leave out chunks of the standard library by default if I think that they're too dangerous, but you'll at least be able to turn them on in CMake. If you ask nicely, I might even send you a link to NeoLua so you can get .NET bytecode from Lua source.

Why .NET? I like C# and wouldn't mind having its full power at my disposal when creating scripts, it's very simple to write something that compiles an existing language to .NET and additionally in my opinion, the CLI is the absolute best way to call things written in other languages as if they were written in the language you were using. Also, if I write a ridiculously large mod, I'd rather JIT optimised bytecode than the source code.

This sounds like quite a lot of work, right? From my somewhat limited experience fixing Mod Organizer's Python plugin bindings, I think it might turn out to be less work than the abandoned fork of OpenMW that supported Python scripts was, and someone already did that, so it's somewhat feasible. However, my biggest work reduction strategy is that I'm pretty confident that once Zini finishes his planning document, has had it reviewed by Scrawl, and then has let the rest of us tear holes in it, the result will be something that's either good or flexible enough that if it turns out not to be good after implementation it won't be too hard to redesign it to make it good.
NullCascade
Posts: 121
Joined: 16 Jan 2012, 07:58

Re: Post 1.0 Planning Update

Post by NullCascade »

AnyOldName3 wrote: 06 Apr 2018, 01:39Either way, if Zini goes completely mad and doesn't implement necessary features, I'll make a fork of OpenMW which integrates ...
Image
I really, really don't want modders, let alone players, to have to deal with compatibility between Morrowind, MWSE, OpenmW, and tes3mp. With just an ounce of collaboration, we can make APIs that are safe to use between all those modded projects.

If OpenMW fails to satisfy the community, and forks start popping up, it's going to make the issue even worse.
Chris
Posts: 1625
Joined: 04 Sep 2011, 08:33

Re: Post 1.0 Planning Update

Post by Chris »

NullCascade wrote: 06 Apr 2018, 01:25 ... what? Incrementally working on potentially breaking changes to mwscript over time is easier than doing...

Code: Select all

sol::state lua;
lua.open_libraries( sol::lib::base );
lua.do_script( script->code );
If you think that's all that's necessary for OpenMW to recognize Lua code, and provide access to the game state in a way to ensure future compatibility, and work correctly along side mwscript (the game and preexisting mods will still be using mwscript, so Lua code has to work alongside mwscript), then you apparently don't understand all the issues involved.
So for every new piece of software that comes out, every new API that people want to call to, every piece of custom code modders want... they have to do a pull request.
More than that: they have to ask themselves if it's relevant/necessary to do, if there's other preexisting methods that can already allow what they want, and if it's safe for the user's machine to allow*. Just like not every CPU feature is accessible or relevant to user applications, not every system API needs to be accessible or relevant to mods. Sometimes it'll take a bit more work to achieve, and sometimes it'll come with compromises, but if the result is a secure, stable mod that everyone can use and will still work in 10 years, then IMO that's better than giving them arbitrary code access that can get it done faster while locking out systems the modder doesn't care about and exposing the mod and users to potential future problems the engine can't help with.

* like it or not, scripts are code, and wanting features to allow your code to do more can have implications far beyond what you want to do. Even if you don't know of anyone personally that would abuse it, you need to consider the possibility that someone will want to, so the question is, is the short-term benefits worth the potential long-term problems; the security, stability, future-proofing, and platform lock-in issues.
Zini said that things like Discord RPC would be possible with his solution. Without third party extensions or bloating Discord into the project for people that may not even want it, I just don't see how that is possible.
We know (or can figure out) how Discord and such work. We can also design an interface that would allow mods to access Discord-like functionality. Then it's just a matter of getting the two interfaces to work together. It won't be an overnight thing, but it's certainly a doable prospect. As a benefit, an abstract mod-side interface would allow other backends to be plugged in in the future, so instead of being Discord-only, such a mod could work with Discord or something else that provides similar functionality, and work on different platforms since the mod won't rely on a particular platform's RPC methods.
NullCascade
Posts: 121
Joined: 16 Jan 2012, 07:58

Re: Post 1.0 Planning Update

Post by NullCascade »

Chris wrote: 06 Apr 2018, 02:06If you think that's all that's necessary for OpenMW to recognize Lua code, and provide access to the game state in a way to ensure future compatibility, and work correctly along side mwscript (the game and preexisting mods will still be using mwscript, so Lua code has to work alongside mwscript), then you apparently don't understand all the issues involved.
Yeah, duh, it's more complicated than 3 lines of code. But if in a few days I could add Lua scripting overrides to MWSE, without any source code, just how big of a project do you possibly think it could be to do it to OpenMW? This isn't exactly rocket science here, and you have all the source code available. You don't have to do hacky things or do any reverse engineering, just basic data binding and script calls.
Chris wrote: 06 Apr 2018, 02:06More than that: they have to ask themselves if it's relevant to do, if there's other pre-existing methods that can allow them to do what they want, and if it's safe for the user's machine to allow*. Just like not every CPU feature is accessible or relevant to user applications, not every system API needs to be accessible or relevant to mods.

* like it or not, scripts are code, and wanting features to allow your code to do more can have implications far beyond what you want to do. Even if you don't know of anyone personally that would abuse it, you need to consider the possibility that someone will want to, so the question is, is the short-term benefits worth the potential long-term problems; the security, stability, future-proofing, and platform lock-in issues.
Do you have any actual factual backing to this? Lua is sandboxed. Lua is shipped in major PC (and console/mobile) games. Where's this giant scandal that Lua is insecure? There's so much FUD in this thread, with no evidence backing any of it up.

You know a lot of the games you play let you extend them if you know how, right? And if you choose to download and install a mod, you're already compromised. You're arguing to padlock the back door when your front door is wide open.
Chris wrote: 06 Apr 2018, 02:06We know (or can figure out) how Discord and such work. We can also design an interface that would allow mods to access Discord-like functionality. Then it's just a matter of getting the two interfaces to work together. It won't be an overnight thing, but it's certainly a doable prospect. As a benefit, an abstract mod-side interface would allow other backends to be plugged in in the future, so instead of being Discord-only, such a mod could work with Discord or something else that provides similar functionality, and work on different platforms since the mod won't rely on a particular platform's RPC methods.
So let me get this straight. Instead of giving modders enough power that they can in in 30 minutes or less create a Discord RPC mod like I did with MWSE 2.1, you're advocating... some generic API? And how would you connect with this API? You're saying there are two interfaces, and the two can connect, and that's... what, exactly? You have Discord's API. You don't want to bundle that with OpenMW. How do you work with a native API, perfectly sandboxed code? You can't. You have to, at some point, have someone, somewhere, write native code. You either have to bundle it into the exe, bloating OpenmW to basically include every OBSE/SKSE/MWSE-style mod's library. Or, you change the code I posted to include sol::libs::ffi if the user has a config option set. Which is more viable? Which is more useful?
Chris
Posts: 1625
Joined: 04 Sep 2011, 08:33

Re: Post 1.0 Planning Update

Post by Chris »

NullCascade wrote: 06 Apr 2018, 01:44 If OpenMW fails to satisfy the community, and forks start popping up, it's going to make the issue even worse.
How familiar are you with the Doom modding scene? It had a very similar issue to this hypothetical situation when the source code was released and everyone and their mother started creating forks. Boom, Doom Legacy, Doomsday, ZDoom (itself getting forked to ZDoomGL, GZDoom, Skulltag, and Zandronum), and a host of others I can't think off off the top of my head. They all came about because they wanted to add features their progenitors didn't have. Plenty of incompatibilities between them, and various ones being "king of the hill" at various times. However, over time developers came to see which features modders actually used and what players actually wanted. The different forks would end up implementing common feature sets because that's what was needed to support what players were using. Ones that didn't either died out, or started focusing on a niche that didn't need those features. The term 'Boom-compatible map' came about because features provided by Boom were ubiquitously supported among various forks, and UDMF (Universal Doom Map Format) was a collaboration between forks for another common/standard base that leaves open options for future expandability. Individual forks still have and develop their own unique features, but that doesn't mean they're forever-incompatible.

Different people have different opinions on what should be prioritized, or how to do something. The forks that stand the test of time are ones that have a solid, stable foundation to continue working on. Just because a given fork isn't the first at doing something doesn't mean it never does it; often, said fork can do it better because they can learn from the mistakes of the ones that rushed ahead.

Another thing to consider is that multiple forks can remain in continuous development. TES3MP was forked from a much earlier version of OpenMW, but improvements OpenMW makes still feed back to TES3MP, and OpenMW isn't blind to the improvements TES3MP makes. But just because TES3MP hasn't been fully reintegrated back into OpenMW doesn't mean either is a failure.
Post Reply