Export/Import XML

Feedback on past, current, and future development.
User avatar
EmperorArthur
Posts: 31
Joined: 17 May 2014, 07:52

Export/Import XML

Post by EmperorArthur » 03 Aug 2015, 16:29

This would be a nice feature for larger projects like Tamriel Rebuilt.

These projects are sharing around binary blobs. Even if they are using version control, it can't do difference tracking or merging.
Converting esps and esms to one or more XML files would certainly make things like tracking, reviewing, and merging changes much easier.

dteviot
Posts: 144
Joined: 17 Dec 2014, 20:29

Re: Export/Import XML

Post by dteviot » 06 Aug 2015, 09:41

You don't actually need an import/export capability.
A standalone tool that did XML to binary and back conversion would probably do the job.
That said, the fact that we don't fully understand all the bits in the file format makes it a little difficult.

DocClox
Posts: 101
Joined: 10 May 2015, 13:26

Re: Export/Import XML

Post by DocClox » 06 Aug 2015, 11:13

I always wished (and this is a general TES grumble rather than a OpenMW request) but I always wished there was some sort of assembler or compiler that would let me create game items with a text editor. Probably not dungeon layouts or other highly visual aspects, but things like spells, dialogue, journal entries, it always seems to me that a lot of that would be easier to write outside of the CS/CK.

Admittedly, XML wouldn't be mu first choice for a language syntax. On the other hand I could imagine maybe writing a DSL in python and compiling into XML and then importing the result.

All totally blue sky, of course.

Penultima
Posts: 11
Joined: 20 Oct 2014, 01:08

Re: Export/Import XML

Post by Penultima » 08 Aug 2015, 19:54

dteviot wrote:You don't actually need an import/export capability.
A standalone tool that did XML to binary and back conversion would probably do the job.
That said, the fact that we don't fully understand all the bits in the file format makes it a little difficult.
I'll take a crack at it.
DocClox wrote:I always wished (and this is a general TES grumble rather than a OpenMW request) but I always wished there was some sort of assembler or compiler that would let me create game items with a text editor. Probably not dungeon layouts or other highly visual aspects, but things like spells, dialogue, journal entries, it always seems to me that a lot of that would be easier to write outside of the CS/CK.

Admittedly, XML wouldn't be mu first choice for a language syntax. On the other hand I could imagine maybe writing a DSL in python and compiling into XML and then importing the result.

All totally blue sky, of course.
The format ZDoom uses for Decorate is my personal favorite data serialization format. I have my own implementation in C# that generalizes it for more flexibility.

Penultima
Posts: 11
Joined: 20 Oct 2014, 01:08

Re: Export/Import XML

Post by Penultima » 10 Aug 2015, 06:37

Making progress so far with parsing the .ESMs used by Morrowind, Bloodmoon, Tribunal, and Morrowind GOTY for Xbox (which doesn't add any new records or subrecords compared to PC's Morrowind GOTY). However, the OpenMW wiki page lacks documentation on several record types and their subrecords:

Code: Select all

LTEX
	NAME
	INTV
	DATA
CLOT
	NAME
	MODL
	FNAM
	CTDT
	ITEX
	INDX
	BNAM
	CNAM
	ENAM
	SCRI
REPA
	NAME
	MODL
	FNAM
	RIDT
	ITEX
ACTI
	NAME
	MODL
	FNAM
	SCRI
APPA
	NAME
	MODL
	FNAM
	AADT
	ITEX
LOCK
	NAME
	MODL
	FNAM
	LKDT
	ITEX
PROB
	NAME
	MODL
	FNAM
	PBDT
	ITEX
INGR
	NAME
	MODL
	FNAM
	IRDT
	ITEX
	SCRI
BOOK
	NAME
	MODL
	FNAM
	BKDT
	ITEX
	TEXT
	SCRI
	ENAM
ALCH
	NAME
	MODL
	TEXT
	FNAM
	ALDT
	ENAM
LEVI
	NAME
	DATA
	NNAM
	INDX
	INAM
	INTV
LEVC
	NAME
	DATA
	NNAM
	INDX
	CNAM
	INTV
LAND
	INTV
	DATA
	VNML
	VHGT
	WNAM
	VTEX
	VCLR
PGRD
	DATA
	NAME
	PGRP
	PGRC
SNDG
	NAME
	DATA
	SNAM
	CNAM
DIAL
	NAME
	DATA
INFO
	INAM
	PNAM
	NNAM
	DATA
	NAME
	FNAM
	SCVR
	INTV
	BNAM
	ONAM
	ANAM
	DNAM
	CNAM
	RNAM
	FLTV
	SNAM
	QSTN
	QSTF
	QSTR
SSCR
	DATA
	NAME
I haven't even finished implementing all the records and subrecords documented there yet, so I'll cross that bridge when I get there. I can likely reverse-engineer most of these, but if anyone has any pointers for me, I'd appreciate it.

dteviot
Posts: 144
Joined: 17 Dec 2014, 20:29

Re: Export/Import XML

Post by dteviot » 10 Aug 2015, 10:34

Sometimes, I'm an idiot.
It looks like importing/exporting the files as XML (or JSON, or whatever) might be fairly simple.
Here's the basic plan.
An ESM file is made up of a list of objects, of various types. (e.g. Cell, NPC, Creature, etc.)
The OpenMW code base has a collection of structs that represent each of these objects.
(The structs can be found in https://github.com/OpenMW/openmw/tree/m ... onents/esm)
Each struct has a function called load() that reads the structure from the ESM file and save() which writes the structure to an ESM file.
Here's an example

Code: Select all

void Creature::save(ESMWriter &esm) const
{
    esm.writeHNCString("MODL", mModel);
    esm.writeHNOCString("CNAM", mOriginal);
    esm.writeHNOCString("FNAM", mName);
    esm.writeHNOCString("SCRI", mScript);
    esm.writeHNT("NPDT", mData, 96);
    esm.writeHNT("FLAG", mFlags);
    if (mScale != 1.0) {
        esm.writeHNT("XSCL", mScale);
    }

    mInventory.save(esm);
    mSpells.save(esm);
    if (mHasAI) {
        esm.writeHNT("AIDT", mAiData, sizeof(mAiData));
    }
    mTransport.save(esm);
    mAiPackage.save(esm);
}
Note: the save() function doesn't do the writing of the file itself.
It just tells the passed in ESMWriter what data needs to be saved, and the ESMWriter takes care of writing the data to the file in the correct format.
(The load() function uses a similar ESMReader.)
So, in principle to read/write as XML, you'd just need to modify the ESMWriter and ESMReader to emit XML.
And the ESMReader and ESMWriter are actually quite small.

Of cource, the better solution so that you could read and write both the ESM Native format and XML goes as follows.
Make the ESMReader and ESMWriter classes abstract base classes.
Create new classes ESMReaderNative and ESMWriterNative that derive from the base classes and have the exiting functionality.
Create two new classes ESMReaderXML and ESMWriterXML that also derive from the base class, but they read and write to XML.
Then in the code when you pick export, if you want to read a native ESM, you create a ESMReaderNative and pass it to the structs.
If you want to read an XML ESM, then you create a ESMReaderXML and pass it to the structs instead.
The load() and save() functions in the structs don't need to change.

And yes, to the programmers out there, I realize I've just described polymorphism.
Like I said. Sometimes I'm and idiot.

dteviot
Posts: 144
Joined: 17 Dec 2014, 20:29

Re: Export/Import XML

Post by dteviot » 10 Aug 2015, 10:40

Penultima wrote:I haven't even finished implementing all the records and subrecords documented there yet, so I'll cross that bridge when I get there. I can likely reverse-engineer most of these, but if anyone has any pointers for me, I'd appreciate it.
Try this: http://www.uesp.net/morrow/tech/mw_esm.txt
Also, here's a good collection of links to follow: http://www.uesp.net/wiki/Tes3Mod:File_Format

User avatar
EmperorArthur
Posts: 31
Joined: 17 May 2014, 07:52

Re: Export/Import XML

Post by EmperorArthur » 20 Sep 2015, 21:07

dteviot: That would be the easy way ;)

Did you do any work on this? If you did then I could copy it off of your github. If not, thanks for the starting place.

I don't have much time right now for openmw, but thanks for pointing me in the right direction.

dteviot
Posts: 144
Joined: 17 Dec 2014, 20:29

Re: Export/Import XML

Post by dteviot » 22 Sep 2015, 21:31

@EmperorArthur
EmperorArthur wrote: Did you do any work on this?
Sorry, no. Currently my primary focus is Feature #2229 https://bugs.openmw.org/issues/2229. (Although I keep getting distracted trying to clean up some of the more ugly parts of the codebase.)
And I have some open source projects of my own that need some attention.
And I've discovered "endless sky"
And I've got some other projects I've been wanting to start work on for some time.
And I still have to START playing Skyrim.
EmperorArthur wrote: but thanks for pointing me in the right direction.
You are most welcome. If you need a hand, please let me know. I'm suspecting that XML import/export might be the most useful thing I could do.

User avatar
Zini
Posts: 5536
Joined: 06 Aug 2011, 15:16

Re: Export/Import XML

Post by Zini » 23 Sep 2015, 08:57

dteviot PM'ed me and asked for a statement regarding this feature. Okay then. First, of course everyone is free to experiment with the source and with adding new features, independently of what I think about them. That being said I do not think that adding XML support to OpenMW-CS is a good idea. If at all then I would consider this feature as a plugin, once we have plugin support (post 1.0).


Having two file formats would be a duplication of effort. I certainly do not plan to support two file formats once I start extending the existing format. Dealing with one will be bothersome enough.

I also do not see a lot of value in a XML format or a text format in general in our specific situation. The only real advantage it has is the ability to use a version control tool. That certainly would be useful, but it is not worth the additional effort of supporting two separate formats.

Regarding larger projects, you are probably aware that I come from one of these (Redemption), which gives me a pretty good understanding of what is required. My verdict is that we can support all the needs of such a project with the binary formats (with the exception of version control). We already have a working merge tool and we have a limited ability to review plugins. Post 1.0 we will add enhancements on top of that.

Post Reply