Lua scripting in OpenMW

Everything about development and the OpenMW source code.
User avatar
silentthief
Posts: 456
Joined: 18 Apr 2013, 01:20
Location: Currently traversing the Ascadian Isles

Re: Lua scripting in OpenMW

Post by silentthief »

ptmikheev wrote: 02 May 2021, 00:26 What to expect in the nearest months
I think that at first we should focus on adding features, that are not possible via MWScript. UI, camera control, custom AI.
Looking forward to what can be done with these two: Camera control, custom AI

I could see a modder making a mod that changes the ending of the mad god questline of tribunal, so that instead of just freezing you when you meet Almalexia in the tower of Sotha Sil, so she can talk to you, instead the camera could do some dramatic close up of her, or some other trick with the camera

(youtube vid of that meeting -> https://www.youtube.com/watch?v=5Lv4nyAEG7M)
(questline for that meeting -> https://en.uesp.net/wiki/Tribunal:The_M ... _Sotha_Sil)

---

As for AI, the concept of Morrowind Comes Alive as a way to give NPC schedules (as in, they come out during the day or go inside at night, maybe head to the tavern around dinnertime) is a start to making NPCs have actual behaviors other than the AI wander thing which makes then just walk around in the general location. AI would give the ability for plugging in certain behaviors which would make things WAY more interesting, such as proper fleeing (instead of an NPC who just runs around in the same interior, they go through doors and run to the nearest guard if attacked) or a modder could have a dark brotherhood attack where the NPC attacks, then runs away after sustaining wounds, only to lead the player into a trap with many more dark brotherhood assassins.

Just floating a few thoughts, and looking forward to the future of OpenMW

ST
ps - edited for clarification
kuyondo
Posts: 243
Joined: 29 Mar 2016, 17:45

Re: Lua scripting in OpenMW

Post by kuyondo »

ptmikheev wrote: 02 May 2021, 00:26 What to expect in the nearest months
I think that at first we should focus on adding features, that are not possible via MWScript. UI, camera control, custom AI.
I hope, some simple functions such as addItem or removeItem, even though possible with MWscript, could also be given some implementation.

Or maybe provide a way for OpenMW Lua to interact with in-game MWscripts, such as getting a local variable or global variable.
Cammera
Posts: 19
Joined: 08 Oct 2017, 22:35

Re: Lua scripting in OpenMW

Post by Cammera »

Someone just brought this up in the discord- stuff for enhanced music and sound mods shouldn't be too hard (Mostly making scripts able to check on mroe stuff, wouldn't it?) and it'd be a good early boon to rattle interest, we've had better audio mods since forever but they've all been old and lua MWSE stuff so far. Maybe a way to disable vanilla music behavior altogether too?
Just a stray thought really.
User avatar
FiftyTifty
Posts: 63
Joined: 15 Oct 2014, 21:02

Re: Lua scripting in OpenMW

Post by FiftyTifty »

When Lua does get added to OpenMW, and we have access to all sorts of properties and such, AI should be relatively easy enough to add. We already have AI packages, it's just Morrowind has such barebones and horribly limited conditions, targets, etc.

Now, if we can do the logic in Lua? We can fake a bunch. Move to a door, then teleport the NPC to the house's cell if they can unlock the door. Close enough, and it does the trick.
ptmikheev
Posts: 69
Joined: 01 Jun 2020, 21:05
Gitlab profile: https://gitlab.com/ptmikheev

Re: Lua scripting in OpenMW

Post by ptmikheev »

OpenMW Lua roadmap

Initial Lua MR was merged some time ago and basic (still very limited) OpenMW Lua is available in latest unreleased builds (can be found here).
Here are plans for further development.

Dehardcoding

Dehardcoding means reimplementing game mechanics (which are currently implemented in C++) as built-in Lua mods.
Now it is one of the main priorities for development.

Why is it so important?
Without dehardcoding Lua scripts are not fully functional and can not alter game mechanics. For example without dehardcoding we can not write a script that prevents player death, because when health goes to zero, C++ code immediately starts death animation before scripts can do anything. OpenMW Lua doesn't have an analogue of TES3MP validators that allow to cancel something that is done by C++ code.
Instead we plan to use script interfaces and event system. But this approach works only if game mechanics are already implemented in Lua.

Public API and temporary internal API

Suppose that there is an important C++ function actorDoSomething(ptr) and we want this functionality to be available from Lua as soon as possible. But in future we plan to dehardcode this function and need a way to do it without breaking mods.

Then the steps are:
  1. Design an interface that will still be relevant after dehardcoding.
    In our example we may decide that after dehardcoding "actorDoSomething" will be implemented in a builtin script 'basic_ai.lua' and will be available in local scripts via a script interface as interfaces.BasicAI.doSomething().
  2. Add an internal function self:_doSomething() to the Lua package "openmw.self". This function doesn't need documentation and shouldn't be used by modders.
  3. Create 'basic_ai.lua' and implement interfaces.BasicAI.doSomething() as a simple wrapper around self:_doSomething(). It is public interface that can be used in mods.
  4. Later during dehardcoding we can remove self:_doSomething() and replace the wrapper with a real implementation without changing the public interface.
All functions and engine handlers with prefix "_" are internal API that shouldn't be used outside of built-in scripts. But note that it is a temporary thing. At the end there will be no internal API at all and built-in scripts will not have any "special powers" comparing to normal mods.

Principles of extending Lua API
  1. It is easy to add a new command, but it is hard to remove if it was designed badly because somebody may already depend on it. So we should be careful. At the current stage we should focus on major stuff (like designing an approach how to work with records from Lua). There is no sense in adding a lot of minor commands (unless they are needed for major ones), because too lot of things are yet unclear and something can change.
  2. New commands should be conceptually compatible with multiplayer:
    A local script can see only some area of the game world (cells that are active on a specific client). Any data from inactive cells can’t be used, as they are not synchronized and could be already changed on another client.
    A local script can only modify the object it is attached to. Other objects can theoretically be processed by another client. To prevent synchronization problems the access to them is read-only.
    Global scripts can access and modify the whole game world including unloaded areas, but the global scripts API is different from the local scripts API and in some aspects are limited, because it is not always possible to have all game assets in memory at the same time.
    Though the scripting system doesn’t actually work with multiplayer yet, the API assumes that there can be several players. That's why any command related to UI, camera, and everything else that is player-specific can be used only by player scripts.
  3. We need to remember that scripts and rendering work in separate threads. It is possible to run some callbacks from the main thread if it is really needed, but let’s try to minimize the number of such cases.
Adding new commands to Lua API
  1. Create an issue on gitlab with a proposal on what should be added to Lua API.
  2. Discuss the proposal and check that it meets the principles above. We will add a special gitlab tag "Approved feature request" (or something like this) for issues that have passed this step and are ready to work on.
  3. Create an MR with the change. It should also update documentation and increment core.API_REVISION. Once a framework for integration tests is ready, there will be a requirement that every new Lua command should be used in at least one integration test.
What must be done before 0.48 release

For now we are free to change Lua API in any way without thinking about compatibility. But after 0.48 (the first stable release with Lua) we will not be able to remove commands. So the main thing that must be done before the release is to change in advance everything that we may want to change in the future.

In particular:
  1. Design a good interface for AI packages that will not be changed after dehardcoding (i.e the built-in script "basic_ai.lua" as described above). Remove from public API (i.e. add prefix "_") all functions that work with AI directly:

    Code: Select all

    self:enableAI(v), self:getCombatTarget(), self:setDirectControl(control),
    self:startCombat(target), self:stopCombat()
    
    They should be available via script interfaces instead.
  2. Review the whole API one more time.
  3. Decide whether we continue to use lua-scripts=something.omwscripts in openmw.cfg, or somehow store it inside omwaddons.
Of course before the release we also want to add to OpenMW Lua as much new functionality as possible, but it is less critical than the points above.

Work in progress
  • Package “openmw.input” – access user input and input bindings (merge request 1073)
  • New testing framework (merge request 1005)
  • Camera dehardcoding (merge request 1030)
  • Lua UI (Uramer works on it)
  • Safe way to pass Lua callbacks to C++ code, needed at least for UI and async raycasting (I am going to work on it, but not yet started)
How to contribute

It is recommended to contact me first ;)

There are some tasks in queue:
  • Mod settings and a persistent storage for settings.
  • Design an interface for AI packages.
  • Create/remove/pick/drop objects in Lua (see some ideas here).
  • Issues on gitlab with tag "Lua".
And there is a lot of major not yet designed stuff.
Each item in this list requires a forum thread, discussion, and probably a design doc. One of the questions for each item is "how is it going to work with multiplayer". Another question is how it will work after dehardcoding.
  • Work with records from Lua. Currently Lua scripts can get recordId (string) from any object, but can not get any info from the record itself. Records should also support queries.
  • Call MWScript from Lua. Maybe access variables from MWScript. Also need to think how to get rid of (i.e. replace with a converter) MWScript in future.
  • How to work with a player’s journal? Need to take into account both multiplayer (each player has their own journal, but some notes and quests can be common) and later games like Skyrim. Requires significant generalization of current journal implementation.
  • ...
mikeprichard
Posts: 113
Joined: 16 Dec 2018, 19:42

Re: Lua scripting in OpenMW

Post by mikeprichard »

ptmikheev (and others contributing) - as a non-coder/programmer, I don't understand a word you just wrote. ;) However, what I do understand is this is a massive task that has been very (very) long in the planning stages, and I hugely appreciate the work you're doing to make this a reality for all OpenMW users. Kudos!
User avatar
psi29a
Posts: 5355
Joined: 29 Sep 2011, 10:13
Location: Belgium
Gitlab profile: https://gitlab.com/psi29a/
Contact:

Re: Lua scripting in OpenMW

Post by psi29a »

For now we are free to change Lua API in any way without thinking about compatibility. But after 0.48 (the first stable release with Lua) we will not be able to remove commands. So the main thing that must be done before the release is to change in advance everything that we may want to change in the future.
Agreed that once we make a decision, we have to carry it for as long as we can. However we do need the ability to deprecate a command. So let's not design ourselves into a box either and say we have to carry a function perpetually, however we can offer a reasonable deprecation period/policy if that is necessary.

The safe assumption is yes, once we make it, we need to commit to it. There needs to be a very special reason for deprecation.

Also important, the eventual replacement of our mwscript interpreter with transpiler. That Lua shouldn't bother with touching or using mwscript at all.
User avatar
Thunderforge
Posts: 503
Joined: 06 Jun 2017, 05:57

Re: Lua scripting in OpenMW

Post by Thunderforge »

ptmikheev wrote: 02 Aug 2021, 18:15But after 0.48 (the first stable release with Lua) we will not be able to remove commands.
I agree with psi29a, we need to allow for the possibility of deprecating and removing problematic commands (which would only be exercised on very rare occasions). Otherwise, we run the risk of major issues appearing and us being stuck with them forever, which is not good for anyone.

Say that we create a command in 0.49 and it becomes clear for whatever reason that it is problematic and needs to be changed or removed entirely. We could make so that in 0.50 it is marked as "deprecated for removal in 0.52".

To make sure that it gets noticed and fixed, perhaps on startup a dialog box could appear that says "Mod XYZ uses deprecated functions and will not work on OpenMW 0.52 or later. Please contact the author to get this fixed." And then the console logs have more details.

Again, this would be a last resort and hopefully quite rare, but it would allow us to correct any major issues that accidentally get introduced.
dmbaturin
Posts: 6
Joined: 27 Aug 2017, 19:25

Re: Lua scripting in OpenMW

Post by dmbaturin »

Decide whether we continue to use lua-scripts=something.omwscripts in openmw.cfg, or somehow store it inside omwaddons.
I believe it's better to store scripts inside .omwaddon files, for multiple reasons:
  • Storing scripts separately will make mods that include both Lua scripts and other content (dialog, cells...) much harder to install.
  • Having a global directory for scripts will make such mods especially hard to install and keep updated, such a procedure would be a real repellent for people who want to play the game rather than dedicated and enthusiastic mod testers.
  • MWSE uses external *.lua files just because there's no way to modify the ESP format and the old engine is limited to 255 of them anyway. Neither is an insurmountable problem for OpenMW.
  • If OpenMW is to evolve to support later TES games, the file format will need to support different script types in any case.
ptmikheev
Posts: 69
Joined: 01 Jun 2020, 21:05
Gitlab profile: https://gitlab.com/ptmikheev

Re: Lua scripting in OpenMW

Post by ptmikheev »

dmbaturin wrote: 03 Aug 2021, 08:06
Decide whether we continue to use lua-scripts=something.omwscripts in openmw.cfg, or somehow store it inside omwaddons.
I believe it's better to store scripts inside .omwaddon files, for multiple reasons: ...
I meant only "*.omwscripts" files that contain list of scripts to run.
I am quite confident that scripts themselves should be in separate files the same as all game assets. Omwaddons can have links to scripts the same way as it has links to models and other assets, and it doesn't make mod installation harder.

If "*.lua" files would be packed in omwaddon, we wouldn't be able to use external IDEs or use such features as console command "reloadlua" (it is essential for debugging).
Post Reply