Lua API for VR
Posted: 18 Mar 2023, 15:50
The main obstacle to merging VR is the sheer complexity of its MR, with a *lot* of changes scattered pseudo-randomly about the code and making duplicates of many UI files for small changes. One step to reducing this complexity is to re-write as much as possible of the MR as a Lua mod. This will also allow modders tools to improve or build on the VR experience. And may on the way provide modding tools for flatscreen openmw as well.
Engine requirements:
Evidently, not every part of VR can be moved to a Lua script. The engine must implement a few things.
Requirements:
A summary of what the Lua API needs to achieve.
A summary of intended APIs and their motivation
Engine requirements:
Evidently, not every part of VR can be moved to a Lua script. The engine must implement a few things.
- Stereo Rendering
A code layer implementing all we need for stereo rendering. This is already merged: https://gitlab.com/OpenMW/openmw/-/merge_requests/1757 - VR runtime
A code layer implementing all we need from OpenXR. Such as managing frame synchronization, swapchains, and tracking data. This currently is part of the draft VR MR: https://gitlab.com/OpenMW/openmw/-/merge_requests/948
Requirements:
A summary of what the Lua API needs to achieve.
- Tracker to actor bone mapping.
Motivation: Self-evident - Tracker to camera mapping.
Motivation: Self-evident - Modifying visibility of actor parts.
Motivation: Many people will find disembodied limbs to be a more comfortable VR experience. - Arbitrary control of actor bones.
Motivation: Implementing visual gestures such as finger pointing. - Read world space or model space position of any part/bone
Motivation: Implementing realistic combat and archery - Input subactions
Motivation: VR inputs differ from other inputs in that they may differentiate between what limb triggered the input.
- Lua API for GUI - For modifying existing UI items and providing new UI items, rather than having to modify existing UI layout files.
A summary of intended APIs and their motivation
- Actor Part Manipulation API
Needed to map tracking data to actor bones. This API shall allow arbitrary manipulation of the position, orientation, and visibility of any and all actor parts. - Input Actions and sub-actions
Needed to extend the input API with VR specific controls. Sub-actions are needed to differentiate control source, such as left vs right hand controllers, if this is relevant for an action. - VR API
An API providing access to VR specific information and methods. - 3D GUI API
Needed to render and interact with GUI as 3D geometry placed within the scene.