MWSE, Lua and OpenMW compatibility

General discussion regarding the OpenMW project.
For technical support, please use the Support subforum.
NullCascade
Posts: 121
Joined: 16 Jan 2012, 07:58

Re: Windows Pre-build Script Documentation Updated

Post by NullCascade »

Zini wrote: 08 Jun 2018, 15:34 Just for the record TES3MP is not going its own direction. I am in regular PM contact with davidcernat.

But yeah, basically. I don't oppose MWSE backwards compatibility per see (except for the security relevant issues) but not at the expense of our own post 1.0 plans.

The only way I can think of to achieve (almost-) backwards compatibility with MWSE would be a translation tool that turns MWSE into OpenMW script. A Python script (because post 1.0 OpenMW-CS will use Python as scripting language) would do the job. It would have to be a fairly complex script though and that would still require running all relevant content files through OpenMW-CS once before they can be used by OpenMW.
mwscript bytecode is super simple and easy to work with. I'd advocate using bytecode for legacy mods whenever possible, with how varied the compilers are. About the only thing reliable is the bytecode, while the script texts can be all over. But still, that won't help actually add the important mwse features.
User avatar
AnyOldName3
Posts: 2667
Joined: 26 Nov 2015, 03:25

Re: Windows Pre-build Script Documentation Updated

Post by AnyOldName3 »

because post 1.0 OpenMW-CS will use Python as scripting language
As I've accidentally ended up as the maintainer for Mod Organizer's Python plugin bridge, I might have some things to say about this when it comes to the implementation, especially if we use SIP/PyQt to let the scripts access Qt types but use Boost::python as the actual interpreter (as neither system can do both jobs on its own).
davidcernat
Posts: 256
Joined: 19 Jul 2016, 01:02

Re: Windows Pre-build Script Documentation Updated

Post by davidcernat »

psi29a wrote: 08 Jun 2018, 15:21
NullCascade wrote: 08 Jun 2018, 15:15tes3mp is going off in its own direction.
@TeamTES3MP: care to share?

As far as I was aware, we were on the same line and the two leads, DavidC and Koncord are here on the forums. Just talked with David a few days ago about an important feature in TES3MP that would have implications for OpenMW users as well in which he also talked with Zini and we more or less hammered out something that should work. I mean, one doesn't work this closely with others only to then go off in another direction.
The only direction I'm interested in is the one where we meet up with OpenMW. I'll rework anything that goes against OpenMW's wishes, once the wishes regarding any given aspect become clear to me.



That being said, I do have an unique perspective on compatibility with mods. Yes, a lot of mods have been made with MWSE in mind, and NullCascade wants to somehow solve for that. Every single Morrowind mod ever made up to 2017 has been made with singleplayer in mind, and I need to solve for that.

In practice, the vast majority of mods work fine in multiplayer because they are not overly complex, but – for complex ones – almost every single scripted sequence ever created is going to assume the existence of a single player and potentially break hard if there are more players going around where they shouldn't.

There's no way to "automagically" fix these situations because of their infinite variety, but I can give server owners the ability to tweak or override client scripts, the ability to listen to specified variable changes, the ability to synchronize only certain variables across players, and the ability to reimplement anything they want as part of server scripts, the idea being that – through collaboration – we can put together a collection of multiplayer fixes to complicated singleplayer scripts.

NullCascade's problem seems a lot easier, because it may well be possible to "automagically" fix all MWSE scripts by putting them through a MWSE-to-OpenMW 1.x converter, or by having an MWSE script emulator that only runs for scripts that are identified to use MWSE functions.

As such, his attitude seems unnecessarily brash. He could go a long way with better people skills.
User avatar
AnyOldName3
Posts: 2667
Joined: 26 Nov 2015, 03:25

Re: Windows Pre-build Script Documentation Updated

Post by AnyOldName3 »

Putting aside the points of questionable merit that NullCascade has made, there is still the debate of whether mods executing native code is a really bad idea or a really good idea. Zini thinks it should never be necessary as mod authors can just improve the engine directly and letting users add random bits of native code is going to lead to someone's computer getting infected. NullCascade believes that there's no real-world security risk as every Bethesda script extender has implemented this feature and no one seems to have ever created a malicious mod, but if individual mod authors need to do something the engine doesn't support we'll end up with ten zillion forks and loads of mods that work on master-branch OpenMW but not on whatever the latest release is at any given point in time.

My personal opinion is that both sides have some merit, but if Zini turns out to be wrong, if we were to make an engine plugin system (which loads .DLLs/.SOs) separate from the game plugin system (which loads ESPs) we could keep the majority of mods sandboxed and be more intrusive with warnings about the few that weren't. I'm pretty sure at some point there's going to be some feature that we won't want to include (because it's niche, hard to maintain or seems really stupid) or won't be able to include (e.g. because of licencing issues) but a subset of users will be keen on. Once something like that comes up, it might be worth discussing this in earnest.
davidcernat
Posts: 256
Joined: 19 Jul 2016, 01:02

Re: Windows Pre-build Script Documentation Updated

Post by davidcernat »

AnyOldName3 wrote: 09 Jun 2018, 01:08 Putting aside the points of questionable merit that NullCascade has made, there is still the debate of whether mods executing native code is a really bad idea or a really good idea.
NullCascade has previously given examples of mods that need to execute native code:
NullCascade wrote: 05 Apr 2018, 17:52
fraang wrote: 05 Apr 2018, 17:45In what use case do you need to access stuff outside the engine from mwscripts?
I linked one example, interfacing with programs like Discord. There's also Fliggerty's Census and Excise Office, which was a sort of social network for characters that posted events from the game and let you view your character's stats on the site. Then there's Sky Diversity, which loads different sky textures as the game is running and uses imagemagick to create new ones so you have dynamic and custom skies. These don't directly manipulate game data, they manipulate files and online services. And, these are just for Morrowind, which is not the best example of a game that has these features. There are better examples for Oblivion and Skyrim.
They're not persuasive examples to me because most functionality requiring online services can be implemented using a server script in TES3MP that cannot cause any harm to a client's computer.

Here's an example. Download the latest public release of TES3MP, join the German server called "[RKD] Ashfall - https://ashfall.de" and then watch as you appear on this live map based on your location ingame.

(Sadly, the server owner seems to have introduced a Lua bug that keeps crashing the server every 30 seconds, but I'll talk to him and get him to fix it when I see him around.)
User avatar
AnyOldName3
Posts: 2667
Joined: 26 Nov 2015, 03:25

Re: Windows Pre-build Script Documentation Updated

Post by AnyOldName3 »

The sky thing isn't necessarily the best approach to getting really good sky so the engine could do something completely different to implement that. For the networky ones, it's possible that we might allow mods network access without giving them access to native code. They'd be able to leak any data available within the sandbox into the wild, but that's not necessarily scary as it's just game state, and they'd be able to collect data from the wild and let it influence the game, but any mod could ruin the game if it wanted anyway. The main risk is that some nutcase will write a crypto miner in MWScript, but at that point, they've earned the pittance it could make them.

Alternatively, others have suggested that all that kind of stuff could get merged and we'd be fine.

My point is that every example so far has been something that could be done some other way except adding VR support (as we could make the interface LGPL and link against existing SDKs), but that will become a non-issue when OpenXR releases, which should be any day now.
davidcernat
Posts: 256
Joined: 19 Jul 2016, 01:02

Re: Windows Pre-build Script Documentation Updated

Post by davidcernat »

If we had a client-server model (even for singleplayer, like in Neverwinter Nights) where only the serverside scripting system could interface with online services, that would require servers to make sure they don't use scripts harmful to themselves, but the clients would be at no risk.

That seems way better to me than dealing with client mods providing strange and risky DLL files. Instead, server owners would mostly just have to contend with cross-platform vetted libraries and human-readable scripts.

Everything else could be done as an engine feature (or optional engine plugin).
NullCascade
Posts: 121
Joined: 16 Jan 2012, 07:58

Re: Windows Pre-build Script Documentation Updated

Post by NullCascade »

davidcernat wrote: 09 Jun 2018, 06:33 If we had a client-server model (even for singleplayer, like in Neverwinter Nights) where only the serverside scripting system could interface with online services, that would require servers to make sure they don't use scripts harmful to themselves, but the clients would be at no risk.

That seems way better to me than dealing with client mods providing strange and risky DLL files. Instead, server owners would mostly just have to contend with cross-platform vetted libraries and human-readable scripts.

Everything else could be done as an engine feature (or optional engine plugin).
NWN is a good source to learn from for this. Another place to look to find mistakes/inspiration is the Legends of Aria project. It's a paid game, sort of a spiritual successor to Ultima Online and Neverwinter Nights. It emphasizes modders, and everything from skill systems to area interaction to class mechanics to GUIs are driven by server-side scripting. NWN2 is slightly less clean for this. It does mostly server-side scripting but GUI scripts fire on the client. Either way, the client-server model would be fantastic and still works just fine for single player too.

That's going to require a whole lot of rewriting though. OpenMW wasn't written with this in mind.
davidcernat
Posts: 256
Joined: 19 Jul 2016, 01:02

Re: Windows Pre-build Script Documentation Updated

Post by davidcernat »

NullCascade wrote: 09 Jun 2018, 07:17 That's going to require a whole lot of rewriting though. OpenMW wasn't written with this in mind.
I think it's gone from the realm of hypotheticals because we already have a client-server model in TES3MP. Yes, our server is quite limited and unusually minimalist because we don't want to break compatibility with OpenMW, but if we moved some of the world logic over to the server or to the shared components where both the client and server can use it, you'd have a done deal, and then we'd just have bugs to fix to make sure everything works the same as before.
User avatar
Jemolk
Posts: 237
Joined: 20 Mar 2017, 02:12
Location: Sadrith Mora

Re: Windows Pre-build Script Documentation Updated

Post by Jemolk »

I just want to say, as a user who doesn't have the requisite coding skill to really help speed things up, the lack of MWSE support combined with the insistence that it can only be meaningfully discussed -- not implemented, discussed -- post-1.0 is incredibly frustrating to watch. Beyond frustrating, in fact. Can we just get this question answered -- if LUA scripting a la MWSE 2.1 were introduced, with sandboxing and without the ability to execute native code, would that be mergeable? Would it require changes to esp or ess file formats and thus require OpenMW to be post-1.0 to really implement (but allow us to prepare for it beforehand)? And would it be possible to have the script interact with MWSE and OpenMW such as to output the same result, even if the method of execution is different? Honestly, what gets me is the insistence that we shouldn't even ask the questions and plan that out in advance. Because Zini, while we truly could not be more in your debt for the work you've already put into OpenMW, and while we are truly grateful for it, I for one would really like to have some idea of where the future leads. That's the stressful part of it for me, and I wouldn't be surprised if NullCascade feels largely the same way.
Post Reply