OpenMW’s mb/s data load rate?

Feedback on past, current, and future development.
User avatar
psi29a
Posts: 5356
Joined: 29 Sep 2011, 10:13
Location: Belgium
Gitlab profile: https://gitlab.com/psi29a/
Contact:

Re: OpenMW’s mb/s data load rate?

Post by psi29a »

cc9cii wrote: 19 Nov 2018, 04:54
AnyOldName3 wrote: 19 Nov 2018, 04:02 Hopefully, we'll have a fast BA2-style archive format by then that Morrowind can be repackaged in, and if that's the case, the only limit will be the power of your machine.
I believe BSA archives already support compressed files? If my BSA changes are merged in OpenMW can already support it. Over the holiday period I'll see if I can find some time for BA2 archive support as well.
I haven't worked on this much, so perhaps someone can put more time into it.
https://gitlab.com/psi29a/openmw/tree/e ... der_cc9cii

It is an attempt to get some of cc9cii's work into master by replacing Ogre3D bits with something that uses std::*

As for the BSA fixes, that has to be refactored as to not use Ogre3D bits as well.
User avatar
Ace (SWE)
Posts: 887
Joined: 15 Aug 2011, 14:56

Re: OpenMW’s mb/s data load rate?

Post by Ace (SWE) »

AnyOldName3 wrote: 19 Nov 2018, 04:02 @Ace how on earth are you finding NVMe SSDs cheaper than SATA ones, or, in fact, any flash storage for less than 5 cents a gigabyte? The cheapest I've ever seen is £0.12/GB, and that's already a steal.
It's all a matter of scale I think, seeing as the NVMe drives I'm buying are larger than the SSDs. Though it seems like I managed to miscalculate the $/GB price slightly due to the conversions, so it should be $0.144 and $0.145 per GB respectively.
Tes96 wrote: 19 Nov 2018, 05:55 Ace, is it possible to transfer all data content from cdrom to storage drive so that data transfer rates are exponentially faster?
That's called installing something, which you are required to do as loading things from CD/DVD directly is insanely slow even compared to a low-performance laptop harddrive.
User avatar
AnyOldName3
Posts: 2668
Joined: 26 Nov 2015, 03:25

Re: OpenMW’s mb/s data load rate?

Post by AnyOldName3 »

The installation process for Morrowind does that already. If you're using an older CD with CD-based DRM, it only checks the CD is there to check you actually own a CD and doesn't load the game off it. If you're using a newer release of Morrowind like the Steam or GOG version, you don't need a CD at all.
User avatar
AnyOldName3
Posts: 2668
Joined: 26 Nov 2015, 03:25

Re: OpenMW’s mb/s data load rate?

Post by AnyOldName3 »

The big performance advantage of BA2s over BSAs (both of which come in compressed and uncompressed formats to optimise for different situations) is that texture BA2s hold DDS images based on their content, not as individual files. In the table of file names, there's a pointer to the DDS header (which can be shared between multiple textures so there's only the need to store each unique one once) and then a set of pointers to each mip level, so textures can be easily streamed in one mip level at a time.
User avatar
Tes96
Posts: 232
Joined: 29 Feb 2012, 03:45
Location: Off-grid

Re: OpenMW’s mb/s data load rate?

Post by Tes96 »

Someone a long time ago came up with the idea of preloading all exterior/interior cells during initial startup. It would probably take hours, depending on how many mods were installed.

What about utilizing RAM for preloading anticipated local exterior cells during gameplay? If I’m in exterior cell [1,2], the RAM could load all the objects in all cells surrounding it. Most gamers have at least 8gb, and some elite users have over a terabyte.

I think one of the biggest bottlenecks is the fact that OpenMW still loads all objects within the player’s field of view. Even when standing up against the side of a house, the engine still crunches those numbers to render the backside of all objects & landscapes even though they can’t be seen.
User avatar
psi29a
Posts: 5356
Joined: 29 Sep 2011, 10:13
Location: Belgium
Gitlab profile: https://gitlab.com/psi29a/
Contact:

Re: OpenMW’s mb/s data load rate?

Post by psi29a »

This is already done via scrawl's cell pre-loading, including doors into interiors and other activators. You shouldn't ever see a cell loading screen... at least I've haven't seen one in a very long time and my hardware is 4+ years old.

And from what I remember, we do scene culling... no point in rendering something that the player/camera doesn't see.
User avatar
cc9cii
Posts: 523
Joined: 28 Mar 2013, 04:01

Re: OpenMW’s mb/s data load rate?

Post by cc9cii »

AnyOldName3 wrote: 19 Nov 2018, 13:50 The big performance advantage of BA2s over BSAs (both of which come in compressed and uncompressed formats to optimise for different situations) is that texture BA2s hold DDS images based on their content, not as individual files. In the table of file names, there's a pointer to the DDS header (which can be shared between multiple textures so there's only the need to store each unique one once) and then a set of pointers to each mip level, so textures can be easily streamed in one mip level at a time.
Looking at this site BA2 seems to be optimised for space by sharing common chunks/parts of a texture. Some profiling needs to be done to be sure, but my gut feel says if anything the processing time will in fact increase unless disk I/O is the what mainly slows things down.
User avatar
Tes96
Posts: 232
Joined: 29 Feb 2012, 03:45
Location: Off-grid

Re: OpenMW’s mb/s data load rate?

Post by Tes96 »

psi29a wrote: 19 Nov 2018, 16:21 This is already done via scrawl's cell pre-loading, including doors into interiors and other activators. You shouldn't ever see a cell loading screen.
Wait a minute, you’re saying OpenMW’s “cell pre-loading” has already eradicated cell loading times to virtually nil when switching between exterior & interior cells and between adjacent exterior cells?



Then what was Ace talking about on the previous page?
Ace (SWE) wrote: 18 Nov 2018, 21:44 All data OpenMW uses needs to be converted during loading, as the format used during runtime is optimized for CPU and RAM efficiency, while the formats on disk are optimized more towards out-of-band modification and storage size.
What this means is that even if you're able to load the entirety of every file in a single CPU cycle, you still wouldn't be rid of loading screens as the majority of the time spent loading is actually taken to convert on-disk formats to in-memory formats.

While it is possible to store data in a homogeneous format on both memory and disk - indeed being the format of choise for many DOS-era game engines - such a thing today with the size of games tends to cause large hits on performance, memory usage, and disk space.
User avatar
AnyOldName3
Posts: 2668
Joined: 26 Nov 2015, 03:25

Re: OpenMW’s mb/s data load rate?

Post by AnyOldName3 »

cc9cii wrote: 19 Nov 2018, 22:38
AnyOldName3 wrote: 19 Nov 2018, 13:50 The big performance advantage of BA2s over BSAs (both of which come in compressed and uncompressed formats to optimise for different situations) is that texture BA2s hold DDS images based on their content, not as individual files. In the table of file names, there's a pointer to the DDS header (which can be shared between multiple textures so there's only the need to store each unique one once) and then a set of pointers to each mip level, so textures can be easily streamed in one mip level at a time.
Looking at this site BA2 seems to be optimised for space by sharing common chunks/parts of a texture. Some profiling needs to be done to be sure, but my gut feel says if anything the processing time will in fact increase unless disk I/O is the what mainly slows things down.
I occasionally work on Mod Organizer 2 as well as OpenMW, and the team there and the people they interact with have done a lot of investigation into the relative performance of various options. Even without a VFS, an uncompressed BA2 or BSA is always faster than loose files simply because there are far fewer system calls if you can just seek around one file handle instead of having one for each asset, and system calls are relatively slow. No games support both BSAs and BA2s, so we can't compare directly, but sharing DDS headers is unlikely to save a significant amount of space - a DDS header is only 124 bytes, so it seems like a bit of a micro-optimisation unless it has multiple benefits, and I'm told it makes texture streaming easier. Depending on the ratio of CPU power to drive speed, uncompressed BSAs/BA2s may or may not be faster than compressed ones.
User avatar
Ace (SWE)
Posts: 887
Joined: 15 Aug 2011, 14:56

Re: OpenMW’s mb/s data load rate?

Post by Ace (SWE) »

Tes96 wrote: 19 Nov 2018, 23:13 Then what was Ace talking about on the previous page?
Pre-loading is still loading, the only difference is that it happens in the background during gameplay instead of in its own paused screen.
I should've perhaps emphasised the loading part of loading screen more, as it doesn't have to be it's own screen, that's my bad.

You'll still have at least two loading screens either way, one when the engine loads all your base data, and one when it builds the initial game state from that data. Those two could theoretically also be pushed to the background - and indeed many games do such loading during the intro videos, though it doesn't make much sense for OpenMW to dedicate developer work to try and do the same.

But either way, it is required to spend CPU cycles converting on-disk formats to in-memory, and that's most definitely going to be the larger part of the loading time - as long as you're not on an ancient low-power hdd or a slow network filesystem.
Post Reply