Config/Launcher discussion (or: One guy wants some things)

Everything about development and the OpenMW source code.
User avatar
AnyOldName3
Posts: 1454
Joined: 26 Nov 2015, 03:25

Config/Launcher discussion (or: One guy wants some things)

Post by AnyOldName3 » 06 Sep 2016, 03:29

There are a few things that I think need discussing about the launcher and config files before 1.0 happens and we have to worry about 'breaking changes'. Some are some worries I have, while others are probably legitimate bugs, and some more things are feature requests. Some or all of this might already be on the issue tracker (if not as an issue, instead as a couple of comments on a tangentially related issue), but things here will interact with each other, so should probably be discussed in the same place.
  • Escape characters
    A couple of months ago, I did some investigation related to two issues both marked as bugs. I learned some things, and made some changes meaning OpenMW now behaves as follows:
    • Boost, when loading directory paths, uses '&' as an escape character. This allows paths to contain '"' (the double quotes character, in case that hasn't come out clearly). As an example, to add the directory

      Code: Select all

      ~/morrowind/mods/The "To be or not to be" mod/DataFiles
      , one would add

      Code: Select all

      data="~/morrowind/mods/The &"To be or not to be&" mod/DataFiles"
      with ampersands preceeding the double quotes, and, sticking with the shakesperian theme,

      Code: Select all

      ~/morrowind/mods/Romeo&Juliet/DataFiles
      would require

      Code: Select all

      data="~/morrowind/mods/Romeo&&Juliet/DataFiles"
      , with ampersands preceeding any ampersands. Currently, this is somewhat weird, (especially for Windows users, as the double quotes charactr is illegal in NTFS, FAT, and anything else Windows properly supports), and isn't documented anywhere.
    • Initially, the launcher only counted '#', the hash character, as starting a comment if it was the first non-whitespace character on the line, whereas Boost's Program Options module allowed it to start a comment anywhere, causing issues when loading anything with a '#' in it, such as esp files, data directories, or potentially any string. I 'fixed' this by sneaking and '#' characters which didn't start a line through Boost as an escape sequence. This works fine, but may not serve us perfectly in the future. Also I just realised I never checked if I needed to apply this change to the CS, so that's something I should look at.
    I think, however, that there are a few limitations with the 'fix'. Firstly, I think someone might want to add a comment part way through a line at some point, and there's now no way of doing that. Additionally, there's bound to be something else that something trips up on at some point, and as it stands, it will need a new solution when it rears its head. An example I can think of right now is if some string, such as a level up message, needs to contain a new line.

    Ideally, I think we should decide upon an escape character that can be used anywhere in our config files to trigger the next character to be interpreted as plain text (or the numbers after the escape character to be interpreted as a unicode code point, but I'm not the greatest fan of this, as it makes a lot of things a pain). This should really be decided on before 1.0 hits, as changing the config files to potentially be not backwards or forwards compatible sounds like asking for trouble. We could then use it for file paths so no one has to be confused by an ampersand, to allow a hash to be put in anywhere it's needed, and for any other character which comes along (or that some cautious user decides might break stuff, so they escape it just to be on the safe side). This actually wouldn't be too hard or tedious to get working using the code I added as a base. However, we'd need a little bit of (probably, but not definitely, minor) tweaking to the launcher.

    However, it's also possible that I've missed something obvious and everything is fine with the current system after all. I just can't convince myself that this is the case on my own, though.
  • Adding data directories and BSA archives
    Currently, for a user to add these to their game, they have to edit the config files, but to enable or disable plugins, there's a nice GUI that handles it for them. This isn't the most convenient thing in the world, and is worsened by caveats like having to remember to escape ampersands (or potentially do something else if my proposed escape character system is implemented). In my opinion, it would add a lot to the usability of OpenMW (especially while there's no feature-complete mod manager) if a GUI for this was implemented as part of the launcher. As user-facing jobs go, this could be pretty simple and could be attempted by someone who wanted a simple task instead of tackling a long-standing bug or engine feature - they'd essentially be copying the plugin tab as a new data directory tab.

    Another possibility to making things simpler for modders would be if someone were to write a plugin for Mod Organizer to make it work with OpenMW. MO already has fancy features like a virtual file system, and has the advantage of allowing traditional tools, like Wrye Mash to be used within the VFS. If it just wrote its data file directory, plugin and archive selection to OpenMW's config file, then everything could 'just work'.
  • Order of deactivated mods
    Quite a number of mod managers I've used (well really just NMM, Wrye *ash and MO) keep track of the position of deativated mods in the mod list, which is really helpful when trying to group optional esps with the non-optional ones from the same mod, or when testing different mods which do the same thing. It probably also has other uses which I can't think of.

    This could be added by adding "data-deactivated", "content-deactivated" and "archive-deativated" keys to the config, which the engine would safely ignore, but which the launcher could use to keep track of relative positions. This might be a thing only I want, though.
  • Probably an actual bug
    Finally, I've also noticed that sometimes, when activating esps or adding data directories, comments which I've added in between "data=" and/or "content=" are removed when the launcher loads and resaves the config file. This should probably go on the issue tracker, and I'll try to get around to that once I've remembered exactly what I was doing, and therefore exactly how to reproduce it, but I'm putting it here just on the off-chance that someone will be inspired by this post to do everything it says, as they'll be working on that part of the code anyway.
In an ideal world, I'd do all this myself and make a bunch of pull requests, as nothing is all that hard, but I almost certainly won't have time due to the man in charge of my degree being a big fan of setting more work than it's humanly possible to do, and then making it compulsory. As things are, this is mostly being written to invite discussion, and to consolidate some of my thoughts in a place where people who care and know what they're talking about can shoot them down safely.
AnyOldName3, Master of Shadows

Chris
Posts: 1558
Joined: 04 Sep 2011, 08:33

Re: Config/Launcher discussion (or: One guy wants some thing

Post by Chris » 06 Sep 2016, 07:27

AnyOldName3 wrote:
  • Ideally, I think we should decide upon an escape character that can be used anywhere in our config files to trigger the next character to be interpreted as plain text (or the numbers after the escape character to be interpreted as a unicode code point, but I'm not the greatest fan of this, as it makes a lot of things a pain).
My personal preference would be to use %num code points, since that's how other things already deal with files and paths containing otherwise ambiguous characters (e.g. a website URL that contains a ? that isn't starting a parameter list).

Alternatively, in the case of using an escape character, I'd prefer the standard backslash. It's what you use in most programming languages to read the next non-alpha-numeric character verbatim as part of the string. I don't really like how different parsers all have their own escape character (&, %, $, \, etc), made worse by them using escape characters inconsistently (e.g. ampersand is used in HTML to produce a named character, like & or " for & and " respectively, unlike Boost). A backslash is almost universally used to say 'read the next non-alpha-numeric character as part of the string, even if it would otherwise terminate it'.

Relatedly, I notice that the config file parser seems to be inconsistent in how it treats paths and filenames. For instance, content filenames (esm, esp, omwaddon, etc) must not be quoted, even if they contain a space. Whereas data paths must be quoted. It would be nice to have consistency in string parsing, where unescaped spaces and such are either part of the string, or a string delimiter (and as such, can be fixed by either putting the whole name in quotes or escaping the spaces).
  • Adding data directories and BSA archives
    Currently, for a user to add these to their game, they have to edit the config files, but to enable or disable plugins, there's a nice GUI that handles it for them. This isn't the most convenient thing in the world, and is worsened by caveats like having to remember to escape ampersands (or potentially do something else if my proposed escape character system is implemented). In my opinion, it would add a lot to the usability of OpenMW (especially while there's no feature-complete mod manager) if a GUI for this was implemented as part of the launcher. As user-facing jobs go, this could be pretty simple and could be attempted by someone who wanted a simple task instead of tackling a long-standing bug or engine feature - they'd essentially be copying the plugin tab as a new data directory tab.
+1
  • Order of deactivated mods
    Quite a number of mod managers I've used (well really just NMM, Wrye *ash and MO) keep track of the position of deativated mods in the mod list, which is really helpful when trying to group optional esps with the non-optional ones from the same mod, or when testing different mods which do the same thing. It probably also has other uses which I can't think of.

    This could be added by adding "data-deactivated", "content-deactivated" and "archive-deativated" keys to the config, which the engine would safely ignore, but which the launcher could use to keep track of relative positions. This might be a thing only I want, though.
I had the idea of simply prepending the filename with "0:". So for instance

Code: Select all

content=0:somemod.esp
Would be understood by the launcher and OpenMW to be a deactivated content file, to not load it, but leave it ordered correctly. Same for data paths

Code: Select all

data="0:/some path/to/mod"
would be understood as a deactivated data path, to not include it, but still leave it ordered correctly (this shouldn't be a problem on Windows, since drives are represented by letters and can't confuse 0: as a drive). Similarly, prepending "1:" could indicate an activated mod/data path, though that can be omitted.
  • Probably an actual bug
    Finally, I've also noticed that sometimes, when activating esps or adding data directories, comments which I've added in between "data=" and/or "content=" are removed when the launcher loads and resaves the config file. This should probably go on the issue tracker, and I'll try to get around to that once I've remembered exactly what I was doing, and therefore exactly how to reproduce it, but I'm putting it here just on the off-chance that someone will be inspired by this post to do everything it says, as they'll be working on that part of the code anyway.
I honestly don't see how you can properly preserve config file comments when you have programs altering the contents of the file. Especially if order is important and said programs will reorder lines. For example, if you have

Code: Select all

content=foo.esp
# entries below are something something...
content=bar.esp
Then you use the launcher to move bar.esp up before foo.esp. Where does the comment end up? Does it stay below foo.esp, or above bar.esp? There's no way to know what it should do. IMO, any file that is automatically generated shouldn't guarantee that user comments will be preserved.

User avatar
Zini
Posts: 5536
Joined: 06 Aug 2011, 15:16

Re: Config/Launcher discussion (or: One guy wants some thing

Post by Zini » 06 Sep 2016, 15:08

Regarding the escape character: I agree that any changes need to be made before 1.0.

Adding data directories and BSA archives: The idea is to move away from the whole concept. The user really isn't expected to fiddle around with data directories and at least for post-1.0 content resources archives will be picked up automatically (we may introduce some workaround for older content). I really see no point in improving stuff now that we most likely will deprecate some time after 1.0.

Order of deactivated mods: Again, that looks like a workaround for a system we are trying to get away from. I don't have a solution at hand, but surely we can find a better way to address the issues you mentioned once the resources and content file handling changes are in.

User avatar
AnyOldName3
Posts: 1454
Joined: 26 Nov 2015, 03:25

Re: Config/Launcher discussion (or: One guy wants some thing

Post by AnyOldName3 » 06 Sep 2016, 22:30

The biggest problem with using backslash as an escape character is that it's the default path separator on Windows. It would remove the ability for Windows users to copy and paste data directory paths straight out of explorer and into the config file. This will remain an issue until we have a nice GUI that can automatically escape things, so we probably need either a GUI or a different escape character.

As for why I'm not a fan of numerical code points, they rely on users knowing the code points of any character they want to type, so if they want the path "C:\Games\Morrowind\Mods\Lysol's Guar Skin Banners Normal Mapped 1.0 for OpenMW", they need to know a backslash is 5C, and change it to "C:\5CGames\5CMorrowind\5CMods\5CLysol's Guar Skin Banners Normal Mapped 1.0 for OpenMW", whereas it's easier just to put a backslash instead of anything worrying, to give "C:\\Games\\Morrowind\\Mods\\Lysol's Guar Skin Banners Normal Mapped 1.0 for OpenMW". This is perhaps not the best example, as everything could just be swapped for forward slashes, but if a hash, or a newline, or a chinese character, or an emoji ever needs replacing, it would rapidly get ugly and unreadable. In terms of implementation, little differs, though. In escape.xpp, there's an input stream filter for stuff going into Boost and another input stream filter for stuff coming out of it (although some objects can be constructed without using the latter filter as there's a faster option). If the system was extended to support arbitrary characters, the input filter would need to convert to the escape character followed by code points if that wasn't how the file was presented (it could just 'do nothing' if the config file required numerical code points), and with either option, the other filter would need to convert back.

The reason that file names and file paths are treated inconsistently is because of how Boost's Paths module works. If you construct a file path from a string, it takes it as-is, but if it's from a stream, it stops at the first whitespace character that isn't within quotes. When loading from the config file, it works from a stream, but everything OpenMW does internally with paths is achieved with strings. If we were messing around with what escape characters did what in what context, we could build a Boost path factory kind of thing, and make quotes unnecessary, but it's probably quite a bit of work for little gain if no one's fiddling with that code anyway.

I've just tested, and, at least on Windows, both 'data-deactivated=...' and 'data="0:...' already work as-is. However, while 'content-deactivated=...' works, 'content=0:...' causes the engine to crash on startup, so the engine would definitely need tweaking to allow that proposal. Additionally, it's completely valid for file names, folder names and relative paths to start with '0:' in unix, so it's a poor choice of thing to use as a magic value. Finally, '0:' feels really dirty to me, as it's putting data about how to interpret a value in the value itself, rather than the key. If we had to put both inactive and active mod files under the same key, I feel it would be better practice to redefine the fields as having to hold a pair of the original value and whether or not it's active.

As for comment preservation, I'm not sure how feasible this is in the general case, but for my second year group project which I did last year, I treated ini values as a three-tuple of comments, key and value. The comments held anything (except whitespace-only lines) between the previous value and the key for the current one, and the key and value had their obvious meanings. When writing things back, it could reorder things arbitrarily and still put comments back in sensible places. This would need a fourth field for any comments which were on the same line as the key and value, but while potentially ugly, would preserve everything.



Now onto Zini's stuff...

I'm struggling to see a way of maintaining the same flexibility as the current system while also keeping things any kind of automatic. Can you clarify what you mean, as I'm just picturing everything on fire and can't figure out how to avoid it.


Either way, it looks like some of this stuff might be better implemented as a Mod Organizer plugin if it's to be abandoned eventually in favour of something better.
AnyOldName3, Master of Shadows

aesylwinn
Posts: 243
Joined: 14 Dec 2015, 20:30

Re: Config/Launcher discussion (or: One guy wants some thing

Post by aesylwinn » 06 Sep 2016, 23:15

Maybe it's just me, but I don't think ordering the mods for organization will mix well with setting the load order. How is the GUI going to work for that?

User avatar
AnyOldName3
Posts: 1454
Joined: 26 Nov 2015, 03:25

Re: Config/Launcher discussion (or: One guy wants some thing

Post by AnyOldName3 » 07 Sep 2016, 01:28

Much like that of Mod Organizer or Nexus Mod Manager:
Capture.PNG
MO plugins pane
Untitled.png
MO mods pane (effectively the same as OpenMW's data= stuff, as it's a list of folders for use in the VFS)
Note how both have lists of things with a defined priority, but some are checked and some are unchecked. Also, the esp list has two priority columns, one with numbers which MO uses, and one with hexadecimal values which the game sees.

MO keeps track of mods (VFS folders) in modlist.txt, with the section screenshotted coming out as follows:

Code: Select all

+Climates Of Tamriel - Dragonborn Patch
+Climates Of Tamriel - Dawnguard Patch
+Build Your Own Home
+Blocksteal Redux - Prevents accidentally pick up
+Better Shaped Creatures
+Be a Milk Drinker
+SkyUI
+Automatic Follower Loot System
+Auto Outfit Change
+Animated Dwemer Lift Load Doors
-Coverkhajiits
+Digitigrade Khajiit and Argonian Raptor Combined
-High Quality Eyes
-Improved NPC Clothing
-Brows
-Beards
For the esps, it keeps two files, one called plugins.txt, which has the active plugins in-order, which is fed to the game running in the VFS, and another called loadorder.txt, which holds all the plugins in the VFS in the order it will display them in, regardless of whether or not they're active. I assume it doesn't hold all this in the same file as the game's expecting plugins.txt to be in a certain format, and also could edit it.

It's definitely feasible, and is already done for the premier mod manager for Bethesda's Gamebryo games (excluding Morrowind as no one's written a plugin for it yet. If we're including mod manager features, like plugin activation and deactivation and a VFS in OpenMW, it probably makes sense to make them useful.
AnyOldName3, Master of Shadows

User avatar
Zini
Posts: 5536
Joined: 06 Aug 2011, 15:16

Re: Config/Launcher discussion (or: One guy wants some thing

Post by Zini » 08 Sep 2016, 08:40

I'm struggling to see a way of maintaining the same flexibility as the current system while also keeping things any kind of automatic. Can you clarify what you mean, as I'm just picturing everything on fire and can't figure out how to avoid it.
The general idea is to avoid the loose file collection we currently have and automatically deal with resources. That means for all future content all resources files should be packeged into an archive (with support within OpenMW-CS to simplify the process) and then associate each content file with an archive, meaning that the archive is pulled in automatically as soon as the content file is selected. This can either be done via matching the name or via a new record in the content file (we haven't decided yet).

Chris
Posts: 1558
Joined: 04 Sep 2011, 08:33

Re: Config/Launcher discussion (or: One guy wants some thing

Post by Chris » 08 Sep 2016, 11:45

Sounds like that could be an issue for a mod where you just want the resources, but not the esp/omwaddon file.

It also might be a bit cumbersome, since downloaded mods will first come as a single archive package, which contains both the resources and any content files. With that setup, you'd require users to manually extract that archive somewhere to get the resource archive (an archive in an archive..) and content file(s), so you can then enable the content files to get the resources.

Mod managers these days work the other way around. They'll list the downloaded archive as a mod package, and 'enabling' the mod will make its resources available to the game (as well as execute scripts for mods that are broken up into modular pieces, so you can selectively install different parts or variations of parts of the mod), which adds the esps/omwaddons to the list of available content files that can be enabled. This is much more flexible as you can simply get the resources without requiring a content file, it allows mods to be more modular (e.g. a monster mod that has a "lore friendly" base of a relative handful of new creatures, and an extended set that includes those lore friendly ones plus a bunch more that may not fit the setting), allows mods to provide variations in one package (e.g. a sky replacer that offers multiple different sky textures at different resolutions), and allows prioritizing resources separately from content files. It also makes it more convenient and user friendly, since all users will have to do is download the mod from their favorite mod site into their mod folder, and it's ready to be enabled.

Considering OpenMW already allows each mod to be installed into a separate folder, it doesn't have the problems loose files caused for Bethesda's games. There won't be any on-disk conflicts where multiple mods are overwriting the same file, nor be any confusion about what file should be on-disk when a mod is removed or re-prioritized. Of course, these loose files can be handle automatically by the launcher... for example, when a mod archive is enabled in the launcher, it automatically extracts the archive to its own folder and adds a new data= line for openmw.cfg that points to it, also allowing the content files to be added to the list of available content files. The user doesn't have to see or go around messing with those folders of loose files, as its all handled for them.

User avatar
AnyOldName3
Posts: 1454
Joined: 26 Nov 2015, 03:25

Re: Config/Launcher discussion (or: One guy wants some thing

Post by AnyOldName3 » 08 Sep 2016, 12:30

It's also a pain if you want to mod a mod - I've got several Skyrim mods which I've not liked the implementation of, but have liked the idea, so have assembled bits and pieces of other mods and my own work to make the mod I wanted to use. if everything used archives instead of loose files, it would take forever. Quite a lot of Skyrim and Fallout mods offer BSA and loose-file-only versions, as lots of people prefer loose files and the fine-grained control they get, especially when using something like MO.
AnyOldName3, Master of Shadows

User avatar
Ace (SWE)
Posts: 823
Joined: 15 Aug 2011, 14:56

Re: Config/Launcher discussion (or: One guy wants some thing

Post by Ace (SWE) » 08 Sep 2016, 13:35

Just to throw my own five cents in;

Been using Mod Organizer since forever, with their built-in BSA unpacker. Mainly because that actually means you are able to pick and choose models and meshes without having to deal with bulky archives.
Of course, I don't have anything against mods distributing as archives as long as there's some built-in way to unpack them.

Plus, when you have a VFS implementation then archives feel a bit redundant for what they actually provide.

Post Reply