Lua API for GUI

Everything about development and the OpenMW source code.
User avatar
urm
Posts: 69
Joined: 02 Jun 2017, 16:05
Contact:

Lua API for GUI

Post by urm » 23 Feb 2021, 00:05

With the Lua MR taking form, it's probably time to consider how to implement various parts of the Lua API. This thread is meant to be a discussion on how to best approach it for the in-game GUI.

Current state of UI in OpenMW:

OpenMW uses MyGUI to render in-game UI. The implementation consits of XML layouts, with some parts generated with C++ code, which also defines any logic or event handling. We have a few custom Widgets, such as elements which automatically resize to fit their content.

Overall most of the UI code is quite old, as it changed little after reaching Morrowind.exe parity, and has some quirks which are not acceptable for a modding API (e. g. some widgets are hard coded to ignore a part of their texture to work around vanilla assets).

Existing UI APIs for Morrowind:
1. MWScript - simple popups and status messages, with no real customization, outside of button text.
2. MWSE Lua - defined directly and entirely through Lua code. AFAIK, has limited customization, but it's easy to make the UI fit the vanilla style. https://mwse.readthedocs.io/en/latest/l ... ght=tes3ui
3. tes3mp Custom UI PR - not a part of any official tes3mp release (yet?), implemented by me. It is essentially an extension of every Widget both from MyGUI and OpenMW, which allows for some simple UI logic to be described right in the XML layout, and to bind some properties and events to network messages. Due to tes3mp only having server-side Lua API, all of the significant logic is handled there, which causes some limitations. https://github.com/uramer/openmw/tree/0.7.1-custom-ui
4. Editing MyGUI layout and resources files included with OpenMW - technically, we already allow some interface customization, which could be expanded by reducing the portion of the layouts that's entirely generated by C++ code. The look of the UI can also be altered by providing replacement textures, which a few mods already do.

The simple approach to the new API:

Expose relevant MyGUI classes through Lua bindings, and allow mods to provide their own resource and layout MyGUI files (probably registered in the script, rather than in openmw.cfg).

There are some issues with this one. Firstly, MyGUI has some questionable or buggy behaviour. Secondly, we would be bound to the current MyGUI API. Both make it clunky to expose to modders, as updating MyGUI would be difficult due to backwards compatibility. Also, MyGUI layout syntax is relatively complex, and is overkill for most mods.

An API abstracted from MyGUI:

A much more flexible approach would be to design an API from scratch, and "implement" it by using MyGUI. It would solve the issues related to MyGUI, and would keep the possiblity of replacing MyGUI with a different UI library (which we might have to do if we realize we can't use MyGUI for later Bethesda games, for instance).

The problems are rather obvious: it's a lot more work, and we could introduce our own bugs.

Specifics to discuss:
1. Should we provide a layout declaration, or construct UI entirely in code? If we do have a layout, which features should it have? (e.g. only widget relations and style, like in MyGUI, or maybe registering events and property binds, like in my PR)
2. Most mods would prefer to edit as little of the UI as possible (e.g. add new info to item popups), how can we offer that option?
3. Multiple mods editing the same UI should be as compatible as possible. An entirely different inventory UI (e. g. Skyrim-like lists with 3d item previews) should not necessarily appear any differently to other scripts interacting with it (e.g. ones that handle right click interactions on items).
4. We should probably provide simple functins similar to the MWScript ones, as many modders would probably still have a use for them.
5. Currently styling MyGUI layouts in vanilla style requires a decent amount of work and boilerplate. No matter which approach we pick, we should provide tools which make it easy.

Both approaches imply we would have to "dehardcode" existing UI into the new API (probably easier with the first option), to allow mods to override parts of them.

Please comment if you have any thoughts, ideas or objections.
Last edited by urm on 23 Feb 2021, 17:10, edited 4 times in total.

User avatar
wazabear
Posts: 32
Joined: 13 May 2020, 19:31
Gitlab profile: https://gitlab.com/glassmancody.info

Re: Lua API for GUI

Post by wazabear » 23 Feb 2021, 06:41

urm wrote:
23 Feb 2021, 00:05
2. Most mods would prefer to edit as little of the UI as possible (e.g. add new into to item popups), how can we offer that option?
Note, what I'm about to say bleeds into (3). This is something really important to get right! I think a good solution to this is to give every widget its own mini API and not differentiate between core components and a modded component. Every component must then be registered with some UID such that other mods can utilize them. You'd also want every component to have some shared dynamics, ideally controlled via events. For example, onHide(UID, handler).

A possible API for the tooltip could be something like this.

Code: Select all

Tooltip (id=UID)
    setFilter(pred(focus_object)) -> what makes this popup show? 
    setWidth()
    setHeight()
    setFadeDelay()
    hide()
    show()
    blockCount() -> number of blocks including the default ones
    getBlock(index) -> returns component, can be anything!
    insertBlock(index)
    removeBlock(index)
    popBlock()
Ideally, builtin components shouldn't be handled any differently then anything else. The problem is that it is extremely dangerous, OpenMW can crash if you mess around with the tiniest UI element from my experience with MorroUI. Things will need to change on the mwgui side, it's mostly just OpenMW unconditionally expecting widgets to be there. You hid that textbox, too bad you're done for.

Another advantage to this is that any mod can override any other mod by simply handling the appropriate event. If events can cascade (events overrides can pass on execution as normal) this becomes even more powerful.
Fade away and radiate

User avatar
urm
Posts: 69
Joined: 02 Jun 2017, 16:05
Contact:

Re: Lua API for GUI

Post by urm » 23 Feb 2021, 20:01

We already have a concept of script interfaces in the Lua API. Essentially, it allows a script to assign a table to a string, which is then available to all other scripts attached to the same object. In that case, we could have every UI element be a script attached to the player, and smaller UI elements (e. g. item tooltips) are used by larger UI elements (e. g. inventory). Other scripts can override the interface through assigning an interface with the same name while being later in the load order. Since the later script can access the older interface (unlike the others), it can pass any unchanged calls up to the standard UI (or a mod above it).

And yeah, I think the idea of splitting the UI elements into pieces that are as small as possible, and have each of them provide an API is good. It might be a decent compromise between the two suggestions I've given: if every UI script is expected to generate a MyGUI widget (potentially with some children), we can combine them as if in a component system.

We will need to rewrite/refactor the existing UI in any case. Making it robust to changes will not be trivial though, I agree.

I'm not sure that exact API is good for tooltips, but some kind of way to attach blocks of information makes a lot sense.

User avatar
psi29a
Posts: 5030
Joined: 29 Sep 2011, 10:13
Location: Belgium
Gitlab profile: https://gitlab.com/psi29a/
Contact:

Re: Lua API for GUI

Post by psi29a » 23 Feb 2021, 23:23

There is already a refactor MR to get things usable for VR... not sure if that will cause conflicts.

User avatar
urm
Posts: 69
Joined: 02 Jun 2017, 16:05
Contact:

Re: Lua API for GUI

Post by urm » 24 Feb 2021, 18:59

psi29a wrote:
23 Feb 2021, 23:23
There is already a refactor MR to get things usable for VR... not sure if that will cause conflicts.
From what I know about that one, it doesn't really touch the same things I'm talking about. The main change there is the ability to render parts of the UI into separate buffers. It would, of course, be nice to merge that first, before we do another big refactor of UI code.

User avatar
wazabear
Posts: 32
Joined: 13 May 2020, 19:31
Gitlab profile: https://gitlab.com/glassmancody.info

Re: Lua API for GUI

Post by wazabear » 13 Mar 2021, 04:38

urm wrote:
23 Feb 2021, 00:05
5. Currently styling MyGUI layouts in vanilla style requires a decent amount of work and boilerplate. No matter which approach we pick, we should provide tools which make it easy.
I've put some more thought into this. It's definitely true that making skins for MyGUI is not user friendly, especially for modders. Layouts on the other hand are pretty standard, there's not anything really special going on, you'll see the same structure in virtually every UI API out there for desktop applications which provide structured layouts in XML-like formats. Skins are the odd man out, and they are certainly the most confusing aspect out of the layout system. Usually, at least with desktop apps, the "skin" is simply implemented with some CSS-like format. I'm not sure if such a styling method is out of the question, it's certainly user-friendly though. Consequently, I don't believe how you choose to implement layouts is a big deal, you'll end up with basically the same thing we have now. Sure you can slim things down and add fancy features like variables and the likes, but at its core it will still be defining type, name, position, alignment, properties, and children.

Regardless of the format that is chosen, the important thing is that layout and style are clearly separable, which is not the current state of MyGUI. From the user-end I think we can all agree modders should not be worrying about anything like resources or image groups or skins or basis skins. One possibility is to take the CSS path and abstract the styling and just expose layouts along with a stylesheet. Every widget needs a background image, borders, font face, etc... I think this approach plays very nicely with a components based system.
Fade away and radiate

ptmikheev
Posts: 55
Joined: 01 Jun 2020, 21:05
Gitlab profile: https://gitlab.com/ptmikheev

Re: Lua API for GUI

Post by ptmikheev » 04 Apr 2021, 01:14

Let's add some details to this topic :)

For performance reasons Lua scripts work in parallel with OSG Cull.
In a simplified form the game loop looks this way:

Code: Select all

while (true) {
  updateAll(); // mechanics, physics, input, etc
  
  DO_IN_PARALLEL(
  	mViewer->renderingTraversals(),  // OSG Cull
        mLuaManager->update());  // Run Lua scripts and put actions that conflicts with OSG Cull in a queue
  
  mLuaManager->applyQueuedChanges();  // Apply actions from the queue
}
We can not do anything with MyGUI from "mLuaManager->update" because it would interfere with OSG Cull.
It means that Lua scripts can not use MyGUI commands directly. So we need an intermediate layer.
On Lua side we can have any UI model we want, and in "applyQueuedChanges" we will copy data from this UI model to MyGUI.

The simplest solution I see is that UI model is just a Lua table.
Something like this:

Code: Select all

local ui = require('openmw.ui')
local async = require('openmw.async')

local function myHandler(text)
    ui['myMod_myDialog'] = nil       -- hide dialog
    ui.update()
    print('User input:', text)
end

local myDialog = {
    size = {200, 150},
    color = {0.7, 0.7, 0.7},
    layout = 'vertical',
    content = {
        {
            padding = {30, 30},
            text = "Enter something to the text field and press enter",
        },
        {
            type = 'TextField',
            padding = {30, 30},
            -- `async:callback` is needed to automatically remove the handler
            -- when the script where it was create is terminated.
            onEnterPressed = async:callback(myHandler),
        }
    },
}

local function onKeyPress(code)
    if code == string.byte('m') and not ui['myMod_myDialog'] then
        ui['myMod_myDialog'] = myDialog     -- show dialog
        ui.update()    -- it only signals to LuaManager that ui requires update,
                       -- real update will be done later in `applyQueuedChanges`.
    end
end
Implementation steps can be:
  • Implement a small prototype that supports only a few widget types and only the simplest settings. It shouldn't take a lot of time.
  • Basing on the prototype, discuss and approve the full list of widgets we need.
  • Finally implement it.
What do you think?

User avatar
AnyOldName3
Posts: 2096
Joined: 26 Nov 2015, 03:25

Re: Lua API for GUI

Post by AnyOldName3 » 04 Apr 2021, 01:41

I don't really see why it's fine to run MWScript before the cull traversal, but Lua needs to be done in parallel and then applied to the frame after. Lua should be faster than MWScript. Also, as the rendering guy, that's when I want the entire CPU available to me and no one else so I don't get other threads booting me off my core, so even if we did have to have a frame of latency wherever we put it, I see in parallel with the cull traversal as the single worst place.
AnyOldName3, Master of Shadows

ptmikheev
Posts: 55
Joined: 01 Jun 2020, 21:05
Gitlab profile: https://gitlab.com/ptmikheev

Re: Lua API for GUI

Post by ptmikheev » 04 Apr 2021, 02:17

AnyOldName3 wrote:
04 Apr 2021, 01:41
I don't really see why it's fine to run MWScript before the cull traversal, but Lua needs to be done in parallel and then applied to the frame after. Lua should be faster than MWScript. Also, as the rendering guy, that's when I want the entire CPU available to me and no one else so I don't get other threads booting me off my core, so even if we did have to have a frame of latency wherever we put it, I see in parallel with the cull traversal as the single worst place.
It is not directly related to GUI, so let's discuss it in the Lua scripting topic. I've answered there.

User avatar
urm
Posts: 69
Joined: 02 Jun 2017, 16:05
Contact:

Re: Lua API for GUI

Post by urm » 04 Apr 2021, 08:59

ptmikheev wrote:
04 Apr 2021, 01:14
Let's add some details to this topic :)

For performance reasons Lua scripts work in parallel with OSG Cull.

We can not do anything with MyGUI from "mLuaManager->update" because it would interfere with OSG Cull.
It means that Lua scripts can not use MyGUI commands directly. So we need an intermediate layer.
On Lua side we can have any UI model we want, and in "applyQueuedChanges" we will copy data from this UI model to MyGUI.
In this case, we have more or less the same limitations as tes3mp server-driven UI. That means that the parts of my tes3mp PR one could call hacky for OpenMW are not hacky at all :) https://github.com/TES3MP/openmw-tes3mp/pull/566

The choice between my PR (or something similar) - extending MyGUI and keeping XML layouts - and designing a completely new API is the same choice I've highlighted in the original post.

Post Reply