Markup Language vs. Binary format for ESX files

[Greendogo]

I was wondering if the reason Bethesda used a binary format, versus, say, a plain text markup language - that you could open with a text editor - was for speed or if it was to save space on the CD; or was it to prevent people from messing around with their ESM/ESPs?

I always wanted to be able to open an ESM/ESP file in a text editor and be able to rewrite it on the spot, without having to mess with using a hex editor and worrying about spacing and whatnot (if I didn't want to directly go through the CS for some reason).

So would it be possible to have one of the changes going on to the ESX format be that it is written in some kind of human readable markup language, maybe some custom format based on the rules of XML? I can imagine a lot of circumstances that being able to open it up and copy something out of a section of the plugin file could save someone a lot of time.

On top of those benefits, a human readable markup language could make the OpenMW a lot more friendly to developers.

[psi29a]

Depends on the use of the ESM, if it is constantly being looked at by Morrowind... then binary format makes sense: think sqlite, db2, tokyocabinet or similiar database and key/value store.

If it is just a tar ball/zip of assets with key data being read in once... well, that is something else

[Greendogo]

What do you mean by assets?

As a far as I know, the ESMs are mostly a spreadsheet of entries. Things like meshes and textures are completely separate from ESMs, having at most their names and their file paths. I think you know this already though

My point though, is that a "tar ball/zip" is unnecessary. Compression is unnecessary. If it's important for speed's sake, you could either load what you need into memory at start up, or you could have the program convert it into a more condense format upon program startup.

[garvek]

Bethesda, like other companies use binary format for several reasons, including (but not exhaustive):
- binary is easier to decode by programs
- binary takes less space
- binary can't be easily tweaked by users, they have to use the editor
- binary can't be reused directly by competitors

Master ESM can be more than 100 MB, yet being binary, compact, compiled format. Equivalent in text file can be like ~600 MB or more, you can forget editing with a simple editor Even "smaller" files of several MB are way too big to handle in text format.

[Greendogo]

The Morrowind ESM is only 76mb. However, I get your point. So I was wrong and Brother Brick was right, tarball/zipping is required if it's not binary.

[sirherrbatka]

vim can eat huge text files just fine

But nothing else can I think.

[ezzetabi]

I have a section in my dissertation about this!

Textual (e.g., XML) advantages
- human readable
- easy to mix with text (think HTML)
- easily modifiable

Binary advantages (e.g., protobuf)
- faster to parse (20 to 100 times)
- smaller

The real question is, why Bethesda did not included an easy binary to text and text to binary converter? As you said their format is just a list it would easy and make many operations faster...

But Bethesda is... suboptimal in many design decisions.

[sirherrbatka]

I'm wonder if they created such converter for internal usage.

[ezzetabi]

I think not. Morrowind seems created with the ``we need-we add'' approach. No central checking or design decisions. This idea allows much faster development, but worse final result. If you used some of the scripting system the effects of this approach are clear. There is no consistency in anything.

The infamous example is ``detect the sex of the main character,'' they needed it for some dialogues they added it there. No way in the scripts.

[garvek]

I think not. Morrowind seems created with the ``we need-we add'' approach. No central checking or design decisions. This idea allows much faster development, but worse final result.

They actually got a great game, so it's not so bad