Feature Discussion: Reload Data Files

Involved development of the OpenMW construction set.
aesylwinn
Posts: 243
Joined: 14 Dec 2015, 20:30

Re: Feature Discussion: Reload Data Files

Post by aesylwinn »

Thanks for testing. I never noticed the long reload time since I'm on an SSD and it only takes about 1 second to reload.
Lamoot wrote: 26 Aug 2017, 15:21 I wonder though, is there a way to further speed up reloading?
There's a good chance there is. I'd have to profile the different parts though to see what's taking it so long though. Unfortunately, I don't have an HDD attached to my development machine at the moment. Certain portions of the load code can easily be sped up, but others cannot without EXTREMELY significant changes.

Edit: I tried running it off a flash drive, and it still only took a second because of filesystem caching. Looks like I'll need some more details on your setup. Are you using your own build, or one of the nighties? What OS is this on? How much RAM do you have? Approximately how many mods do you have installed within the data directories specified in openmw.cfg? Approximately how many .bsa's? Are you working with multiple cells open (if so how many) or just a preview?
Lamoot
Posts: 176
Joined: 22 Apr 2016, 12:03

Re: Feature Discussion: Reload Data Files

Post by Lamoot »

aesylwinn wrote: 26 Aug 2017, 16:14 Are you using your own build, or one of the nighties? What OS is this on? How much RAM do you have? Approximately how many mods do you have installed within the data directories specified in openmw.cfg? Approximately how many .bsa's? Are you working with multiple cells open (if so how many) or just a preview?
I'm using the nightly build
Windows 10
4GB of RAM
3 .bsa (Morrowind, Tribunal, Bloodmoon)
1 mod installed (Vality's Bitter Coast Addon)
I only have the preview window open

It takes the same amount of time to reload in a completelly new game project (created in CS). The lists of meshes, textures, sounds, videos etc. are still populated with Morrowind's assets though, even in a new .omwgame.
aesylwinn
Posts: 243
Joined: 14 Dec 2015, 20:30

Re: Feature Discussion: Reload Data Files

Post by aesylwinn »

I spent some time setting up a windows dev environment with a HDD, and added some basic profiling. With an unoptimized debug build I get the following:

Code: Select all

Adding BSA archive D:\SteamLibrary\steamapps\common\Morrowind\Data Files\Morrowind.bsa
Adding BSA archive D:\SteamLibrary\steamapps\common\Morrowind\Data Files\Tribunal.bsa
Adding BSA archive D:\SteamLibrary\steamapps\common\Morrowind\Data Files\Bloodmoon.bsa
Adding data directory D:\SteamLibrary\steamapps\common\Morrowind\Data Files
Adding data directory C:\Users\kylea\Downloads\YAPHM (Yet Another Portable House Mod) V 1.3-43480-1-4\Data Files
Adding data directory C:\Users\kylea\OneDrive\Documents\My Games\OpenMW/data
Time spent recreating file index: 4340 ms
Time spent recreating resource tables: 1182 ms
Time spent clearing out cache: 1 ms
Time spent recreating paged scene: 55 ms
With an optimized release build I get the following:

Code: Select all

Adding BSA archive D:\SteamLibrary\steamapps\common\Morrowind\Data Files\Morrowind.bsa
Adding BSA archive D:\SteamLibrary\steamapps\common\Morrowind\Data Files\Tribunal.bsa
Adding BSA archive D:\SteamLibrary\steamapps\common\Morrowind\Data Files\Bloodmoon.bsa
Adding data directory D:\SteamLibrary\steamapps\common\Morrowind\Data Files
Adding data directory C:\Users\kylea\Downloads\YAPHM (Yet Another Portable House Mod) V 1.3-43480-1-4\Data Files
Adding data directory C:\Users\kylea\OneDrive\Documents\My Games\OpenMW/data
Time spent recreating file index: 209 ms
Time spent recreating resource tables: 45 ms
Time spent clearing out cache: 2 ms
Warning: Unable to find land texture index 15 in plugin 1, using default texture instead
Warning: Unable to find land texture index 102 in plugin 1, using default texture instead
Time spent recreating paged scene: 60 ms
In both cases, file indexing takes the longest. I wonder if the nightlies are debug builds? I guess I'll download one and check.

Edit: Nightlies are release builds. I'm not sure why it takes so much longer on your computer. Based on what you mentioned, it shouldn't be taking 6 seconds. I'm assuming that the issue is file indexing, because that fits both with what you mentioned and my own testing. This should only be an issue if you have tons of files in the data folders being scanned, or many added bsa's. Can you check the terminal to make sure those are the only 3 bsa's being added?
Lamoot
Posts: 176
Joined: 22 Apr 2016, 12:03

Re: Feature Discussion: Reload Data Files

Post by Lamoot »

Here's the output I get when running CS.

Code: Select all

Loading settings file: ./openmw-cs.cfg
Loading config file: C:\Users\Lamoot\Documents\My Games\OpenMW/openmw.cfg... done.
Loading config file: ./openmw.cfg... done.
Adding BSA archive C:\Program Files (x86)\GalaxyClient\Games\Morrowind\Data Files\Morrowind.bsa
Adding BSA archive C:\Program Files (x86)\GalaxyClient\Games\Morrowind\Data Files\Tribunal.bsa
Adding BSA archive C:\Program Files (x86)\GalaxyClient\Games\Morrowind\Data Files\Bloodmoon.bsa
Adding data directory C:\Program Files (x86)\GalaxyClient\Games\Morrowind
Adding data directory C:\Program Files (x86)\GalaxyClient\Games\Morrowind\Data Files
Adding data directory C:\Users\Lamoot\Documents\My Games\OpenMW/data
Not sure if related, but starting CS anew takes 13 seconds from the Open dialogue to getting the editor window ready. I also removed OpenMW and Morrowing along with data folders and installed anew, but with no changes.
aesylwinn
Posts: 243
Joined: 14 Dec 2015, 20:30

Re: Feature Discussion: Reload Data Files

Post by aesylwinn »

Lamoot wrote: 27 Aug 2017, 11:59 starting CS anew takes 13 seconds from the Open dialogue to getting the editor window ready.
Yes that's related. That's when the index is initially built.

If your willing, it would be nice of you could run a build with the reload profiling. The branch is here if you want to build it yourself. Otherwise, I've built the binaries for windows 10 available here. It should be as simple as unzipping the folder and running openmw-cs. You may need to install the visual studio redistributable. The SHA256 hash is 7db9728f1da917957ba0ba1274f25056b0205d479004ced72831111d3de7c7bb.
Lamoot
Posts: 176
Joined: 22 Apr 2016, 12:03

Re: Feature Discussion: Reload Data Files

Post by Lamoot »

With your build, when opening the editor I get:

Code: Select all

Loading settings file: ./openmw-cs.cfg
Loading config file: C:\Users\Lamoot\Documents\My Games\OpenMW/openmw.cfg... done.
Loading config file: ./openmw.cfg... done.
Adding BSA archive C:\Program Files (x86)\GalaxyClient\Games\Morrowind\Data Files\Morrowind.bsa
Time spent indexing C:\Program Files (x86)\GalaxyClient\Games\Morrowind\Data Files\Morrowind.bsa (11090 'files'): 90 ms
Adding BSA archive C:\Program Files (x86)\GalaxyClient\Games\Morrowind\Data Files\Tribunal.bsa
Time spent indexing C:\Program Files (x86)\GalaxyClient\Games\Morrowind\Data Files\Tribunal.bsa (1174 'files'): 7 ms
Adding BSA archive C:\Program Files (x86)\GalaxyClient\Games\Morrowind\Data Files\Bloodmoon.bsa
Time spent indexing C:\Program Files (x86)\GalaxyClient\Games\Morrowind\Data Files\Bloodmoon.bsa (1545 'files'): 10 ms
Adding data directory C:\Program Files (x86)\GalaxyClient\Games\Morrowind
Adding data directory C:\Program Files (x86)\GalaxyClient\Games\Morrowind\Data Files
Adding data directory C:\Users\Lamoot\Documents\My Games\OpenMW/data
Time spent indexing C:\Program Files (x86)\GalaxyClient\Games\Morrowind (21092 files): 2613 ms
Time spent indexing C:\Program Files (x86)\GalaxyClient\Games\Morrowind\Data Files (21065 files): 2652 ms
Time spent indexing C:\Users\Lamoot\Documents\My Games\OpenMW/data (2 files): 0 ms
When reloading I then get this

Code: Select all

Adding BSA archive C:\Program Files (x86)\GalaxyClient\Games\Morrowind\Data Files\Morrowind.bsa
Time spent indexing C:\Program Files (x86)\GalaxyClient\Games\Morrowind\Data Files\Morrowind.bsa (11090 'files'): 167 ms
Adding BSA archive C:\Program Files (x86)\GalaxyClient\Games\Morrowind\Data Files\Tribunal.bsa
Time spent indexing C:\Program Files (x86)\GalaxyClient\Games\Morrowind\Data Files\Tribunal.bsa (1174 'files'): 8 ms
Adding BSA archive C:\Program Files (x86)\GalaxyClient\Games\Morrowind\Data Files\Bloodmoon.bsa
Time spent indexing C:\Program Files (x86)\GalaxyClient\Games\Morrowind\Data Files\Bloodmoon.bsa (1545 'files'): 9 ms
Adding data directory C:\Program Files (x86)\GalaxyClient\Games\Morrowind
Adding data directory C:\Program Files (x86)\GalaxyClient\Games\Morrowind\Data Files
Adding data directory C:\Users\Lamoot\Documents\My Games\OpenMW/data
Time spent indexing C:\Program Files (x86)\GalaxyClient\Games\Morrowind (21092 files): 2499 ms
Time spent indexing C:\Program Files (x86)\GalaxyClient\Games\Morrowind\Data Files (21065 files): 2472 ms
Time spent indexing C:\Users\Lamoot\Documents\My Games\OpenMW/data (2 files): 0 ms
Time spent recreating file index: 5282 ms
Time spent recreating resource tables: 228 ms
Time spent clearing out cache: 4 ms
User avatar
scrawl
Posts: 2152
Joined: 18 Feb 2012, 11:51

Re: Feature Discussion: Reload Data Files

Post by scrawl »

20000 loose files? Looks like you got your answer there.

Anyway something's wrong with your openmw.cfg. The data path is added twice, so the scan takes twice as long again:

Time spent indexing C:\Program Files (x86)\GalaxyClient\Games\Morrowind (21092 files): 2499 ms <------ This data folder is unneeded, go to openmw.cfg and remove it
Time spent indexing C:\Program Files (x86)\GalaxyClient\Games\Morrowind\Data Files (21065 files): 2472 ms
aesylwinn
Posts: 243
Joined: 14 Dec 2015, 20:30

Re: Feature Discussion: Reload Data Files

Post by aesylwinn »

Just for reference, here's mine, albeit with an SSD in this case.

Code: Select all

Loading settings file: ./openmw-cs.cfg
Loading settings file: /home/aesylwinn/.config/openmw/openmw-cs.cfg
Loading config file: /home/aesylwinn/.config/openmw/openmw.cfg... done.
Loading config file: ./openmw.cfg... done.
Adding BSA archive /home/aesylwinn/.openmw/Data Files/Morrowind.bsa
Time spent indexing /home/aesylwinn/.openmw/Data Files/Morrowind.bsa (11090 'files'): 64 ms
Adding BSA archive /home/aesylwinn/.openmw/Data Files/Tribunal.bsa
Time spent indexing /home/aesylwinn/.openmw/Data Files/Tribunal.bsa (1174 'files'): 4 ms
Adding BSA archive /home/aesylwinn/.openmw/Data Files/Bloodmoon.bsa
Time spent indexing /home/aesylwinn/.openmw/Data Files/Bloodmoon.bsa (1545 'files'): 6 ms
Adding data directory /home/aesylwinn/.openmw/Data Files
Adding data directory /home/aesylwinn/.openmw/NOM
Adding data directory /home/aesylwinn/.local/share/openmw/data
Time spent indexing /home/aesylwinn/.openmw/Data Files (7218 files): 116 ms
Time spent indexing /home/aesylwinn/.openmw/NOM (824 files): 21 ms
Time spent indexing /home/aesylwinn/.local/share/openmw/data (7 files): 0 ms
So first of all, it looks as though you are indexing your data folder twice (missed that at first). Do you need to do so? I would suggest removing either "C:\Program Files (x86)\GalaxyClient\Games\Morrowind\Data Files" or "C:\Program Files (x86)\GalaxyClient\Games\Morrowind\" from the list of data folders in your openmw.cfg. That should halve your reload times.

I'm afraid though, I cannot optimize the other issue, which is a large amount of files. Really, the only way to significantly reduce the amount of time spent doing that would be to remove it. That strategy doesn't fit so well with how the editor/game which share this same bit of code work.

Edit: Looks like scrawl beat me.
Edit 2: Remove "C:\Program Files (x86)\GalaxyClient\Games\Morrowind\". I wasn't thinking when I gave the option of either.
Lamoot
Posts: 176
Joined: 22 Apr 2016, 12:03

Re: Feature Discussion: Reload Data Files

Post by Lamoot »

Having removed one data path and halving the reload time, it's become a lot more usable. It's not everyday you get a 100% speedup! It's a funny thing with the data paths though. As mentioned, I did a clean install of OpenMW and Morrowind:

C:\Program Files (x86)\GalaxyClient\Games\Morrowind is what OpenMW wizard offered me as the default data path, but neither OpenMW, the launcher, nor OpenMW-CS could run the game (morrowind.bsa missing).
C:\Program Files (x86)\GalaxyClient\Games\Morrowind\Data Files is what I added and it worked, but I didn't notice it caused double loading of assets, having wrongly asumed that the other path didn't do anything, as it didn't work. Is this a bug then?

Anyway, thank you for your help on the issue, glad to have it working rather decently now.
aesylwinn
Posts: 243
Joined: 14 Dec 2015, 20:30

Re: Feature Discussion: Reload Data Files

Post by aesylwinn »

Lamoot wrote: 28 Aug 2017, 19:16 C:\Program Files (x86)\GalaxyClient\Games\Morrowind is what OpenMW wizard offered me as the default data path
It could be a bug. I don't think though that the openmw.cfg file is removed when you uninstall though, so it could also be something left over from the past. Someone would have to look into it.
Post Reply