OpenMW’s mb/s data load rate?

Feedback on past, current, and future development.
User avatar
Tes96
Posts: 215
Joined: 29 Feb 2012, 03:45
Location: Morrowind

OpenMW’s mb/s data load rate?

Post by Tes96 » 18 Nov 2018, 04:59

I read games are limited by their mb/s data load rate. Is anybody familiar with this term? Does this mean no matter how fast your SSD or M.2 drive is, you’ll be limited by the game’s data load rate?
Project-Tamriel

"If one is to understand the arcane arts, one must study all its aspects, not just the dogmatic narrow view of the Mages Guild."

User avatar
AnyOldName3
Posts: 1456
Joined: 26 Nov 2015, 03:25

Re: OpenMW’s mb/s data load rate?

Post by AnyOldName3 » 18 Nov 2018, 14:18

Whoever said that either had only a vague idea of how things work or was oversimplifying things to the point of meaninglessness.

When loading stuff in a game (either at a load screen or when stuff is streamed in while the game is running) you obviously have to get it off the storage drive. This involves working out where the data is, asking the OS for the data, doing any processing that may be needed and then actually using the data. Some parts of this have a fixed cost per file, some have a fixed cost per amount of data and some depend on both. There's not much point trying to quantify speed in mb/s alone as that ignores all of the bits which have a fixed cost per file, and they're usually pretty significant. One of the two main reasons games package all their assets into a few large archives is that it reduces the number of things with a per-file cost as (as far as the OS can see) everything's actually one big file. The only useful way to quantify loading speed is to actually time real situations and compare them directly.
AnyOldName3, Master of Shadows

User avatar
Tes96
Posts: 215
Joined: 29 Feb 2012, 03:45
Location: Morrowind

Re: OpenMW’s mb/s data load rate?

Post by Tes96 » 18 Nov 2018, 17:55

I don’t know what a fixed cost per file is. It sounds like you’re saying even with the fastest hardware on the market, there is a limit as to how quickly the data files can be retrieved, because of a limitation on the actual game data? :confused:
Project-Tamriel

"If one is to understand the arcane arts, one must study all its aspects, not just the dogmatic narrow view of the Mages Guild."

User avatar
Ace (SWE)
Posts: 825
Joined: 15 Aug 2011, 14:56

Re: OpenMW’s mb/s data load rate?

Post by Ace (SWE) » 18 Nov 2018, 21:44

All data OpenMW uses needs to be converted during loading, as the format used during runtime is optimized for CPU and RAM efficiency, while the formats on disk are optimized more towards out-of-band modification and storage size.
What this means is that even if you're able to load the entirety of every file in a single CPU cycle, you still wouldn't be rid of loading screens as the majority of the time spent loading is actually taken to convert on-disk formats to in-memory formats.

While it is possible to store data in a homogeneous format on both memory and disk - indeed being the format of choise for many DOS-era game engines - such a thing today with the size of games tends to cause large hits on performance, memory usage, and disk space.

User avatar
AnyOldName3
Posts: 1456
Joined: 26 Nov 2015, 03:25

Re: OpenMW’s mb/s data load rate?

Post by AnyOldName3 » 18 Nov 2018, 23:12

In a modern game, I'd expect most of the time loading loose files to be spent waiting for the OS to open and close file handles. That's a big part of why BSA and BA2 archives make Bethesda games faster - if you've only got one file, you only need one file handle, with the rest being that these days, it's often faster to load a large compressed file and decompress it on the CPU than load the same file uncompressed and skip the decompression step.
AnyOldName3, Master of Shadows

User avatar
Tes96
Posts: 215
Joined: 29 Feb 2012, 03:45
Location: Morrowind

Re: OpenMW’s mb/s data load rate?

Post by Tes96 » 19 Nov 2018, 02:53

Ace (SWE) wrote:
18 Nov 2018, 21:44
...convert on-disk formats to in-memory formats.

While it is possible to store data in a homogeneous format on both memory and disk - such a thing today...cause large hits on performance, memory usage, and disk space.
My reason for posting this topic is I will be building my computer in about 6-7 years, with a budget of about $20,000. So even though this is a very premature question and OpenMW will likely be more optimized by 2025, I still want to gain some foresight in what type of hardware I should be looking into. I understand that current generation game loading times are mostly dependent on the read-speed of the storage drive (SSD or M.2)

But your description is still a little too technical for me. Are you saying that reducing all cell loading times to milliseconds could be done best on a system that stored memory and disk data in a single file location? Disk data being the actual CD-rom that Morrowind was released on?

Thank you for taking the time to respond.
Project-Tamriel

"If one is to understand the arcane arts, one must study all its aspects, not just the dogmatic narrow view of the Mages Guild."

User avatar
Ace (SWE)
Posts: 825
Joined: 15 Aug 2011, 14:56

Re: OpenMW’s mb/s data load rate?

Post by Ace (SWE) » 19 Nov 2018, 03:55

I don't think you'll see much load-time difference between SSD or NVMe drives on OpenMW. Be it now, or 6 years in the future.
While it'll certainly be faster than loading from an old rotating drive, even the cheapest of SSDs today will most likely be more than fast enough to hit other limits when it comes to data loading times.

What I am trying to say is that there's a limit to how low cell loading times can be brought, and that limit will have barely anything to do with the speed of the underlying storage. The data as it is stored on disk needs to be processed during loading, as the data format on disk is designed for other purposes than direct use in a physics- or rendering-engine. And while we can probably improve asynchronicity and optimize what gets loaded and when, you'll most likely still see loading screens whenever the engine needs to process larger amounts of data.

Though I'd also say that in 6 years this won't be a relevant question either way. From how things look at the moment, you'll most definitely have an NVMe drive in your computer.

We recently installed 12TB of NVMe into a storage cluster at work, because it turned out to be better than SSDs in regards to pricing and delivery times.
I personally put down an order yesterday as well for two SSDs and an NVMe drive as replacement hardware, and as it turns out the NVMe was actually slightly cheaper per GB. ($0.044/GB for the NVMe, $0.045/GB for the SSDs)

User avatar
AnyOldName3
Posts: 1456
Joined: 26 Nov 2015, 03:25

Re: OpenMW’s mb/s data load rate?

Post by AnyOldName3 » 19 Nov 2018, 04:02

By 2025 (provided the memory cartel has been stopped) you should be easily able to fit all of Morrowind's data onto a RAM drive even on a standard desktop, and there's never going to be anything faster than that hardware-wise. Hopefully, we'll have a fast BA2-style archive format by then that Morrowind can be repackaged in, and if that's the case, the only limit will be the power of your machine.

@Ace how on earth are you finding NVMe SSDs cheaper than SATA ones, or, in fact, any flash storage for less than 5 cents a gigabyte? The cheapest I've ever seen is £0.12/GB, and that's already a steal.
AnyOldName3, Master of Shadows

User avatar
cc9cii
Posts: 471
Joined: 28 Mar 2013, 04:01

Re: OpenMW’s mb/s data load rate?

Post by cc9cii » 19 Nov 2018, 04:54

AnyOldName3 wrote:
19 Nov 2018, 04:02
Hopefully, we'll have a fast BA2-style archive format by then that Morrowind can be repackaged in, and if that's the case, the only limit will be the power of your machine.
I believe BSA archives already support compressed files? If my BSA changes are merged in OpenMW can already support it. Over the holiday period I'll see if I can find some time for BA2 archive support as well.

User avatar
Tes96
Posts: 215
Joined: 29 Feb 2012, 03:45
Location: Morrowind

Re: OpenMW’s mb/s data load rate?

Post by Tes96 » 19 Nov 2018, 05:55

Ace, is it possible to transfer all data content from cdrom to storage drive so that data transfer rates are exponentially faster?
Project-Tamriel

"If one is to understand the arcane arts, one must study all its aspects, not just the dogmatic narrow view of the Mages Guild."

Post Reply