Records saving

Everything about development and the OpenMW source code.
User avatar
lgromanowski
Site Admin
Posts: 1193
Joined: 05 Aug 2011, 22:21
Location: Wroclaw, Poland
Contact:

Records saving

Post by lgromanowski » 18 Aug 2011, 19:33

Hi,
I'm considering to take Record saving task.
I have following (rough) plan, but is there anything else which should I take for consideration?
  1. Rename components/esm/load*.[hpp|cpp] into components/esm/classname.[hpp|cpp] (it's not mentioned in coding policies, but when I add store() method then file name become misleading)
  2. Rename esm_reader* to esmreader* (according to Naming conventions)
  3. Write esmwriter class (based on esm_reader)
  4. For each load*.[hpp|cpp] write a store() method
  5. Prepare a tests (maybe we could use CppUnit or Google C++ testing framework?)
best regards,
Lukasz

User avatar
Zini
Posts: 5536
Joined: 06 Aug 2011, 15:16

Re: Records saving

Post by Zini » 18 Aug 2011, 19:57

Okay on #1, #2 and #4.

#3 looks like overkill, since we don't write esms from the data structures we have in components. We need to write in two cases:

1. Save game (dynamically generated records only): This will probably use a new file format; maybe similar to esm/p. But we haven't designed it yet and we won't write the data from the default record store anyway.

2. Save esm/p from the editor: The editor will use a different kind of data storage (components/esm yes, components/esm_store no).

We may need something like an esmwriter for the testing part, but I suspect that we can get away with writing out each record immediately after we have read it in.

About #5: I am considering revamping the esm tool: Clean up the command line interface and add commands similar in style to git. The current function could be handled by a "dump" command. And the testing by a "test" command, which would read in an esm, write it to a separate location (on disc or in memeory), read it in again and compare this with what was read originally.
This approach might be more useful in the long run. But if you are more comfortable with using a testing framework, that would be okay too.

P.S.: I think Corristo has some code for handling some terrain-related record types that are not available in master yet. Maybe you should pull them in (cherry pick?) .

Edit:

P.P.S.: Also, I think jhooks has a fix for one animation related record in his branch. Cherry pick too?

User avatar
lgromanowski
Site Admin
Posts: 1193
Joined: 05 Aug 2011, 22:21
Location: Wroclaw, Poland
Contact:

Re: Records saving

Post by lgromanowski » 18 Aug 2011, 20:20

Zini wrote:Okay on #1, #2 and #4.

#3 looks like overkill, since we don't write esms from the data structures we have in components. We need to write in two cases:

1. Save game (dynamically generated records only): This will probably use a new file format; maybe similar to esm/p. But we haven't designed it yet and we won't write the data from the default record store anyway.

2. Save esm/p from the editor: The editor will use a different kind of data storage (components/esm yes, components/esm_store no).

We may need something like an esmwriter for the testing part, but I suspect that we can get away with writing out each record immediately after we have read it in.
OK.
Zini wrote: About #5: I am considering revamping the esm tool: Clean up the command line interface and add commands similar in style to git. The current function could be handled by a "dump" command. And the testing by a "test" command, which would read in an esm, write it to a separate location (on disc or in memeory), read it in again and compare this with what was read originally.
This approach might be more useful in the long run. But if you are more comfortable with using a testing framework, that would be okay too.
OK, I will do both - revamp the esm tool and prepare a unit tests.
Zini wrote: P.S.: I think Corristo has some code for handling some terrain-related record types that are not available in master yet. Maybe you should pull them in (cherry pick?) .
OK, I will cherypick them.
best regards,
Lukasz

User avatar
Zini
Posts: 5536
Joined: 06 Aug 2011, 15:16

Re: Records saving

Post by Zini » 18 Aug 2011, 20:34

I hope you saw my PPS in the edit.

jhooks1
Posts: 780
Joined: 06 Aug 2011, 21:34

Re: Records saving

Post by jhooks1 » 18 Aug 2011, 21:05

Zini wrote:
P.P.S.: Also, I think jhooks has a fix for one animation related record in his branch. Cherry pick too?
Yes, check the commit "Fixed part list adding"

User avatar
lgromanowski
Site Admin
Posts: 1193
Joined: 05 Aug 2011, 22:21
Location: Wroclaw, Poland
Contact:

Re: Records saving

Post by lgromanowski » 18 Aug 2011, 21:17

OK, thanks, I will check jhooks commit too.
best regards,
Lukasz

User avatar
Zini
Posts: 5536
Joined: 06 Aug 2011, 15:16

Re: Records saving

Post by Zini » 24 Sep 2011, 09:12

@lgro: Since you were planning to revamp the esmtool anyway, could you have a look at this bug I found:
esmtool: /home/marc/OpenMW/openmw/components/to_utf8/to_utf8.cpp:152: std::string ToUTF8::getUtf8(ToUTF8::FromType): Assertion `0' failed.
Happens when running the esmtool with a non-English Morrowind.esm. I assume the encoding fixes in OpenMW broke something in the tool (probably just requires an option to specify encoding or something like that).

User avatar
lgromanowski
Site Admin
Posts: 1193
Joined: 05 Aug 2011, 22:21
Location: Wroclaw, Poland
Contact:

Re: Records saving

Post by lgromanowski » 01 Oct 2011, 12:39

Zini wrote:@lgro: Since you were planning to revamp the esmtool anyway, could you have a look at this bug I found:
esmtool: /home/marc/OpenMW/openmw/components/to_utf8/to_utf8.cpp:152: std::string ToUTF8::getUtf8(ToUTF8::FromType): Assertion `0' failed.
Happens when running the esmtool with a non-English Morrowind.esm. I assume the encoding fixes in OpenMW broke something in the tool (probably just requires an option to specify encoding or something like that).
Sorry for late reply. Yes I will look into it.
best regards,
Lukasz

User avatar
werdanith
Posts: 295
Joined: 26 Aug 2011, 16:18

Re: Records saving

Post by werdanith » 01 Mar 2012, 09:08

Resurrecting this thread since I was planning to use the esmtool, and then found out that it doesn't print the records of weapons. First though a couple of issues I wanted to tackle.

issue #1:
Zini wrote:@lgro: Since you were planning to revamp the esmtool anyway, could you have a look at this bug I found:
esmtool: /home/marc/OpenMW/openmw/components/to_utf8/to_utf8.cpp:152: std::string ToUTF8::getUtf8(ToUTF8::FromType): Assertion `0' failed.
Happens when running the esmtool with a non-English Morrowind.esm. I assume the encoding fixes in OpenMW broke something in the tool (probably just requires an option to specify encoding or something like that).
I had that issue too, and fixed it up temporarily to get the tool working, I will have to look into a more permanent solution now.


issue #2:
When you compile the game in release mode it throws this warning a million times (due to the optimizer):

Code: Select all

components/esm/esm_reader.hpp: In member function ‘float ESM::ESMReader::getFVer()’:
components/esm/esm_reader.hpp:156:58: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
which comes from this line:

Code: Select all

float getFVer() { return *((float*)&mCtx.header.version); }
It turns out that we have a struct ESM_Context whose Head sub-member has a integer member named version, which is actually storing a float and we are using this hacky cast to convert it back to float when we request it.

Not sure how to proceed with this one.
There are only two possible values for the version:

Code: Select all

enum Version
  {
    VER_12 = 0x3f99999a,
    VER_13 = 0x3fa66666
  };
so one solution would be to convert the function into something like this:

Code: Select all

float getFVer() {  if(mCtx.header.version == VER_12) return 1.2; else return 1.3;  }
ideas?

User avatar
werdanith
Posts: 295
Joined: 26 Aug 2011, 16:18

Re: Records saving

Post by werdanith » 01 Mar 2012, 09:19

In retrospect, starting a new thread and adding a link for the quote I wanted was probably a better idea.

Post Reply