This would be a nice feature for larger projects like Tamriel Rebuilt.
These projects are sharing around binary blobs. Even if they are using version control, it can't do difference tracking or merging.
Converting esps and esms to one or more XML files would certainly make things like tracking, reviewing, and merging changes much easier.
Export/Import XML
Re: Export/Import XML
You don't actually need an import/export capability.
A standalone tool that did XML to binary and back conversion would probably do the job.
That said, the fact that we don't fully understand all the bits in the file format makes it a little difficult.
A standalone tool that did XML to binary and back conversion would probably do the job.
That said, the fact that we don't fully understand all the bits in the file format makes it a little difficult.
Re: Export/Import XML
I always wished (and this is a general TES grumble rather than a OpenMW request) but I always wished there was some sort of assembler or compiler that would let me create game items with a text editor. Probably not dungeon layouts or other highly visual aspects, but things like spells, dialogue, journal entries, it always seems to me that a lot of that would be easier to write outside of the CS/CK.
Admittedly, XML wouldn't be mu first choice for a language syntax. On the other hand I could imagine maybe writing a DSL in python and compiling into XML and then importing the result.
All totally blue sky, of course.
Admittedly, XML wouldn't be mu first choice for a language syntax. On the other hand I could imagine maybe writing a DSL in python and compiling into XML and then importing the result.
All totally blue sky, of course.
Re: Export/Import XML
I'll take a crack at it.dteviot wrote:You don't actually need an import/export capability.
A standalone tool that did XML to binary and back conversion would probably do the job.
That said, the fact that we don't fully understand all the bits in the file format makes it a little difficult.
The format ZDoom uses for Decorate is my personal favorite data serialization format. I have my own implementation in C# that generalizes it for more flexibility.DocClox wrote:I always wished (and this is a general TES grumble rather than a OpenMW request) but I always wished there was some sort of assembler or compiler that would let me create game items with a text editor. Probably not dungeon layouts or other highly visual aspects, but things like spells, dialogue, journal entries, it always seems to me that a lot of that would be easier to write outside of the CS/CK.
Admittedly, XML wouldn't be mu first choice for a language syntax. On the other hand I could imagine maybe writing a DSL in python and compiling into XML and then importing the result.
All totally blue sky, of course.
Re: Export/Import XML
Making progress so far with parsing the .ESMs used by Morrowind, Bloodmoon, Tribunal, and Morrowind GOTY for Xbox (which doesn't add any new records or subrecords compared to PC's Morrowind GOTY). However, the OpenMW wiki page lacks documentation on several record types and their subrecords:I haven't even finished implementing all the records and subrecords documented there yet, so I'll cross that bridge when I get there. I can likely reverse-engineer most of these, but if anyone has any pointers for me, I'd appreciate it.
Code: Select all
LTEX
NAME
INTV
DATA
CLOT
NAME
MODL
FNAM
CTDT
ITEX
INDX
BNAM
CNAM
ENAM
SCRI
REPA
NAME
MODL
FNAM
RIDT
ITEX
ACTI
NAME
MODL
FNAM
SCRI
APPA
NAME
MODL
FNAM
AADT
ITEX
LOCK
NAME
MODL
FNAM
LKDT
ITEX
PROB
NAME
MODL
FNAM
PBDT
ITEX
INGR
NAME
MODL
FNAM
IRDT
ITEX
SCRI
BOOK
NAME
MODL
FNAM
BKDT
ITEX
TEXT
SCRI
ENAM
ALCH
NAME
MODL
TEXT
FNAM
ALDT
ENAM
LEVI
NAME
DATA
NNAM
INDX
INAM
INTV
LEVC
NAME
DATA
NNAM
INDX
CNAM
INTV
LAND
INTV
DATA
VNML
VHGT
WNAM
VTEX
VCLR
PGRD
DATA
NAME
PGRP
PGRC
SNDG
NAME
DATA
SNAM
CNAM
DIAL
NAME
DATA
INFO
INAM
PNAM
NNAM
DATA
NAME
FNAM
SCVR
INTV
BNAM
ONAM
ANAM
DNAM
CNAM
RNAM
FLTV
SNAM
QSTN
QSTF
QSTR
SSCR
DATA
NAME
Re: Export/Import XML
Sometimes, I'm an idiot.
It looks like importing/exporting the files as XML (or JSON, or whatever) might be fairly simple.
Here's the basic plan.
An ESM file is made up of a list of objects, of various types. (e.g. Cell, NPC, Creature, etc.)
The OpenMW code base has a collection of structs that represent each of these objects.
(The structs can be found in https://github.com/OpenMW/openmw/tree/m ... onents/esm)
Each struct has a function called load() that reads the structure from the ESM file and save() which writes the structure to an ESM file.
Here's an example
Note: the save() function doesn't do the writing of the file itself.
It just tells the passed in ESMWriter what data needs to be saved, and the ESMWriter takes care of writing the data to the file in the correct format.
(The load() function uses a similar ESMReader.)
So, in principle to read/write as XML, you'd just need to modify the ESMWriter and ESMReader to emit XML.
And the ESMReader and ESMWriter are actually quite small.
Of cource, the better solution so that you could read and write both the ESM Native format and XML goes as follows.
Make the ESMReader and ESMWriter classes abstract base classes.
Create new classes ESMReaderNative and ESMWriterNative that derive from the base classes and have the exiting functionality.
Create two new classes ESMReaderXML and ESMWriterXML that also derive from the base class, but they read and write to XML.
Then in the code when you pick export, if you want to read a native ESM, you create a ESMReaderNative and pass it to the structs.
If you want to read an XML ESM, then you create a ESMReaderXML and pass it to the structs instead.
The load() and save() functions in the structs don't need to change.
And yes, to the programmers out there, I realize I've just described polymorphism.
Like I said. Sometimes I'm and idiot.
It looks like importing/exporting the files as XML (or JSON, or whatever) might be fairly simple.
Here's the basic plan.
An ESM file is made up of a list of objects, of various types. (e.g. Cell, NPC, Creature, etc.)
The OpenMW code base has a collection of structs that represent each of these objects.
(The structs can be found in https://github.com/OpenMW/openmw/tree/m ... onents/esm)
Each struct has a function called load() that reads the structure from the ESM file and save() which writes the structure to an ESM file.
Here's an example
Code: Select all
void Creature::save(ESMWriter &esm) const
{
esm.writeHNCString("MODL", mModel);
esm.writeHNOCString("CNAM", mOriginal);
esm.writeHNOCString("FNAM", mName);
esm.writeHNOCString("SCRI", mScript);
esm.writeHNT("NPDT", mData, 96);
esm.writeHNT("FLAG", mFlags);
if (mScale != 1.0) {
esm.writeHNT("XSCL", mScale);
}
mInventory.save(esm);
mSpells.save(esm);
if (mHasAI) {
esm.writeHNT("AIDT", mAiData, sizeof(mAiData));
}
mTransport.save(esm);
mAiPackage.save(esm);
}
It just tells the passed in ESMWriter what data needs to be saved, and the ESMWriter takes care of writing the data to the file in the correct format.
(The load() function uses a similar ESMReader.)
So, in principle to read/write as XML, you'd just need to modify the ESMWriter and ESMReader to emit XML.
And the ESMReader and ESMWriter are actually quite small.
Of cource, the better solution so that you could read and write both the ESM Native format and XML goes as follows.
Make the ESMReader and ESMWriter classes abstract base classes.
Create new classes ESMReaderNative and ESMWriterNative that derive from the base classes and have the exiting functionality.
Create two new classes ESMReaderXML and ESMWriterXML that also derive from the base class, but they read and write to XML.
Then in the code when you pick export, if you want to read a native ESM, you create a ESMReaderNative and pass it to the structs.
If you want to read an XML ESM, then you create a ESMReaderXML and pass it to the structs instead.
The load() and save() functions in the structs don't need to change.
And yes, to the programmers out there, I realize I've just described polymorphism.
Like I said. Sometimes I'm and idiot.
Re: Export/Import XML
Try this: http://www.uesp.net/morrow/tech/mw_esm.txtPenultima wrote:I haven't even finished implementing all the records and subrecords documented there yet, so I'll cross that bridge when I get there. I can likely reverse-engineer most of these, but if anyone has any pointers for me, I'd appreciate it.
Also, here's a good collection of links to follow: http://www.uesp.net/wiki/Tes3Mod:File_Format
- EmperorArthur
- Posts: 33
- Joined: 17 May 2014, 07:52
Re: Export/Import XML
dteviot: That would be the easy way
Did you do any work on this? If you did then I could copy it off of your github. If not, thanks for the starting place.
I don't have much time right now for openmw, but thanks for pointing me in the right direction.
Did you do any work on this? If you did then I could copy it off of your github. If not, thanks for the starting place.
I don't have much time right now for openmw, but thanks for pointing me in the right direction.
Re: Export/Import XML
@EmperorArthur
And I have some open source projects of my own that need some attention.
And I've discovered "endless sky"
And I've got some other projects I've been wanting to start work on for some time.
And I still have to START playing Skyrim.
Sorry, no. Currently my primary focus is Feature #2229 https://bugs.openmw.org/issues/2229. (Although I keep getting distracted trying to clean up some of the more ugly parts of the codebase.)EmperorArthur wrote: Did you do any work on this?
And I have some open source projects of my own that need some attention.
And I've discovered "endless sky"
And I've got some other projects I've been wanting to start work on for some time.
And I still have to START playing Skyrim.
You are most welcome. If you need a hand, please let me know. I'm suspecting that XML import/export might be the most useful thing I could do.EmperorArthur wrote: but thanks for pointing me in the right direction.
Re: Export/Import XML
dteviot PM'ed me and asked for a statement regarding this feature. Okay then. First, of course everyone is free to experiment with the source and with adding new features, independently of what I think about them. That being said I do not think that adding XML support to OpenMW-CS is a good idea. If at all then I would consider this feature as a plugin, once we have plugin support (post 1.0).
Having two file formats would be a duplication of effort. I certainly do not plan to support two file formats once I start extending the existing format. Dealing with one will be bothersome enough.
I also do not see a lot of value in a XML format or a text format in general in our specific situation. The only real advantage it has is the ability to use a version control tool. That certainly would be useful, but it is not worth the additional effort of supporting two separate formats.
Regarding larger projects, you are probably aware that I come from one of these (Redemption), which gives me a pretty good understanding of what is required. My verdict is that we can support all the needs of such a project with the binary formats (with the exception of version control). We already have a working merge tool and we have a limited ability to review plugins. Post 1.0 we will add enhancements on top of that.
Having two file formats would be a duplication of effort. I certainly do not plan to support two file formats once I start extending the existing format. Dealing with one will be bothersome enough.
I also do not see a lot of value in a XML format or a text format in general in our specific situation. The only real advantage it has is the ability to use a version control tool. That certainly would be useful, but it is not worth the additional effort of supporting two separate formats.
Regarding larger projects, you are probably aware that I come from one of these (Redemption), which gives me a pretty good understanding of what is required. My verdict is that we can support all the needs of such a project with the binary formats (with the exception of version control). We already have a working merge tool and we have a limited ability to review plugins. Post 1.0 we will add enhancements on top of that.