I want to join

Join the team. This area is for people who want to participate in OpenMW's development in one way or another.
mark76
Posts: 17
Joined: 20 Feb 2012, 10:33

Re: I want to join

Post by mark76 »

I have to apologize for the new delay, but several things got into my way at once. First, Real Life stuff (had to go to a conference among other things), and second, the poor excuse for that Microsoft calls a C++ compiler. After trying to sync my local branch with some more recent additions to the main branch, the compiler choked on some new lines, after fixing this, the code (not mine!) crashed with almost irreproducible stuff. At this point (i.e. after wasting even more time trying to fix it), I finally decided to switch to Linux. Not even two hours later, packages were installed, code was built (three times faster than the Windows compiler), no build errors, no runtile errors, and launching 8 times faster. (Well, *some*of this might be related to my 64 bit Quad Core CPU being supported better by gcc, any maybe some debug/release mode stuff.) Goodbye, Microsoft Visual C++, I'l miss *some* of your features!

If I understood the procedure correct, I'll still need write access at github before I can push my own branch (they call it the "list of collaborators" or something), so here's my github account name: mark76 (same as here). If this should not be required, I must have missed it.

Anyway, back to the main issues. My last post (30 minitues before leaving to said conference) was not as detailled as it could have been, so let's try it again: Multiple esm/esp files touch pretty much everything, without actually introducing a new subsystem; it's more like a boundary condition/design issue that requires specific hacks throughout the engine. I have added basic support for loading multiple esX files (as selected by the launcher, which works out-of-the-box), and verified that *most* stuff works as expected:
- Newly created objects introduced by separate plugins work as expected.
- Modified objects mostly work as expected (but see below for exceptions).
- Deleting entries from a parent file seems to work as expected (but it required a hack in recstore.hpp, to delete entries from the individual lists).
- Undeleting previously deleted entries also seems to work as expected, without having to hack 20-30 individual "loadxxxx.cpp" files or refactoring something else.

In the testing phase, some issues emerged. Textures on Solstheim were hopelessly wrong. I knew from my own project that each plugin introduces a separate list of terrain textures, so I tried if I could quickly port this (which works now, but required some more hacks between the esm parser, the recstore, and the texture loader). Then I got more assertions on Solstheim due to some "i > size"-type conflict within the landscape loader. I had a deeper look at the code and found that (in a nutshell) the terrain importer is doing something involving neigboring cells, and does not take into account yet that the neighboring cell may use a different terrain palette. And this is where I stopped for now, as it got complicated. It looks nice, but I'll need help by the terrain gurus to finish this aspect.

In fact, I know of (at least) six existing subsystems that may need to be adapted for one reason or another to support multiple plugins (dialogue, NIF renderer, landscape textures, GMSTs, references, scripting), and this list might not even be complete. The basic code is in place, it seems to be sufficiently stable to push it on the repository for other people to toy with (and maybe pull it into the main branch, so that we can address a bigger list of beta testers). Then we can collaborate to get the additional work required done. And that's what I meant in my last post - it is almost possible to start working on the individual subsystems mentioned above. It may still become somewhat messy, though, with many different hacks required throughout the engine. The way I see it, I mostly know what needs to be done in these subsystems, and other people can help in finding out how to actually implement this.

This also explains why I haven't tried to push something earlier - I knew that *some* things had to be addressed/checked to prevent semi-random crashes, and I'm not the type of person who pushes something that is known not to work.

One final point that will need to be addressed at some point is the ability to read and write (and select) save files, which are essentially just plain old plugin files. I did not find anything specific to this in the code (except for some save-related hacks in ESMReader::open), so I was wondering what kind of plans for this feature do already exist. And this is what I meant with my question whether anything related to this does already exist - moved references is just the most complicated aspect of this. Right now, my plan is to tackle this feature (i.e. reading/writing saves) next, as it is the point that should require the most work.

I'll push my own branch to the repository when I get write access (and after integrating the most recent changes from the main branch). I'll post some more details on what needs to be done in the individual subsystems in the issue tracker in a few minutes. Feel free to have a look at it, and if you see something you can resolve/help resolving quickls, feel free to say so.

And that's it, essentially. Feel free to correct me if you have a different opinion on any of these points :) .
User avatar
Zini
Posts: 5538
Joined: 06 Aug 2011, 15:16

Re: I want to join

Post by Zini »

You don't need write access. Make your own fork of OpenMW on github and then push to it. See here for additional documentation.

We will not be using the original save file format. Maybe some parts of it, but that has not been decided yet. We will look into it much later. A MW save file importer is still planned though.

These subsystem modifications you propose don't sound good. Except for a few edge cases (like terrain textures) this feature should not require changing anything not ESM file related. If it does, something went wrong (either with your implementation or my thinking).
That is, unless you are also implementing changing plugins between saves and loads, which is definitely not part of the task. For this feature we need to do some more hard thinking, but that should definitely be handled separately.

Well, I can't say anything more about it until I have more details (and seen the code).

Edit: And welcome to the Linux world. Always good to see someone leave the sinking Windows ships.
User avatar
Zini
Posts: 5538
Joined: 06 Aug 2011, 15:16

Re: I want to join

Post by Zini »

Okay, your comments on the tracker cleared that up somewhat. You were trying to pull some other issues into this one. They look like they belong to the ESX task at first, but if you think about it a bit more they certainly don't.
garvek
Posts: 34
Joined: 03 Jun 2012, 21:20

Re: I want to join

Post by garvek »

It may be way out of topic, but I couldn't help, I wanted to react on these posts, I'm sorry if it is not the good place, feel free to move it:
After trying to sync my local branch with some more recent additions to the main branch, the compiler choked on some new lines, after fixing this, the code (not mine!) crashed with almost irreproducible stuff.
Enable -pedantic on GCC and you will start regretting ... :D GCC is way too permissive on some declarations and functions constructions, moreover some code generation in Linux is memory-leak free, where in windows it is not, because of the way the stack and return calls are handled (subtile and very frustratic at debug time).

Anyway the crashes are a big part of window's builts (encountered same issues with another engines which uses same dependencies), I guess that some components (Ogre ? Bullet ?) are not very stable under Redmond OS either.
Always good to see someone leave the sinking Windows ships.
Not so good, since less and less people support it will make harder to fix issues in Windows (my own experience with another project). Anyway, I must be some kind of code masochist. :P
User avatar
scrawl
Posts: 2152
Joined: 18 Feb 2012, 11:51

Re: I want to join

Post by scrawl »

Maybe we should PM yacoby about the terrain?
User avatar
sirherrbatka
Posts: 2159
Joined: 07 Aug 2011, 17:21

Re: I want to join

Post by sirherrbatka »

Or chris. Or both.
User avatar
Zini
Posts: 5538
Joined: 06 Aug 2011, 15:16

Re: I want to join

Post by Zini »

First let's get the code into github. As long as we can't see it, it is rather pointless to discuss about it.
User avatar
sirherrbatka
Posts: 2159
Joined: 07 Aug 2011, 17:21

Re: I want to join

Post by sirherrbatka »

Very valid point.
User avatar
Greendogo
Posts: 1467
Joined: 26 Aug 2011, 02:04

Re: I want to join

Post by Greendogo »

I'm glad this vital component is getting taken care of! Way to go Mark!

Also, could we move this discussion about multiple ESP/ESM support to the proper thread HERE, in the Development thread? Having it in the "New Developers Applications" subforum may prevent it from getting the proper attention it deserves and cause confusion.

This subforum is for new developers who want to work on the project and you are already deep into the code! :D
mark76
Posts: 17
Joined: 20 Feb 2012, 10:33

Re: I want to join

Post by mark76 »

No write access required, just make my own fork? I am loving git(hub) more and more :)

Anyway, my branch is now sitting at mark76/multiple_esm_esp, tested, rebased against the most recent master branch, test build worked as expected (had to pin boost libraries at 1.42). And with this, I fully agree we move any further discussion on the specific code and subsystems that may need to be updated to the development forums.

Maybe just some closing remarks. It it very good to see that most of the points raised earlier are already covered by other topics. I had taken the freedom to enumerate everything that is touched by multiple plugin support in one way or another, and which does not (or only in a limited form) appear in a single plugin config. Well, that and this is essentialy my list in my own project. I am aware that most of this can be isolated as a separate issue - but some of these issues (e.g. the dialogue one) only appear in a multiple plugin configuration and, strictly speaking, are required for full multiple esX support.

It is interesting to hear that you intend not to reuse the original Morrowind save file format. This might help in overcoming some existing limits inherited from Morrowind (and in fact eliminates the need for writing custom plugin files). Anyway, I'll post what I had prepared on this more complex topic in the new thread, maybe it'll help.

And, somehow, I am relieved that Windows issues are not exclusive to the MS compiler - I was starting to doubt my ability to set up an IDE for a specific project. On the other hand, I do not want to transform into a "C++ standard police" of sorts, which is why I have given up on the Windows platform. Maybe somebody is interested in some details, so here's a small (likely incomplete) list of new build errors, warnings etc. that I got:

- Somewhere in a piece of code that was recently added, there was a line similar to "static const float sOmething = 30;" in a class body. VC++ didn't like this, as it is against the standard. From what my Google-Fu found out, "static const int" is allowed, and the same thing for float is about to be added in C++0x (again)
- Somewhere deep in the Windows SDK headers, something seems to define the macro "_DEBUG", as undefining it in the call to the compiler does not help. This enables a small block of debug code in "pmove.cpp", which refers a field in the parent class that is commented out in the header. Bam, instant "undefined symbol" error.
- Some warnings related to a "non-virtual destructor", and that this might result in memory coccuption.
- At one point, an error within Ogre due to a bounding box possessing illegal values (wtf?)
- Beta versions of Visual Studio 2012 already exist, they are supposed to have full support for C++0x, *but* the free Express editions will *only* support Win 8/Win Mobile and that ugly "Metro" UI!

Grerat job, MS, if you intend to chase even more customers away. Roughly at this point, I sait "s**** it, I'll switch to Linux". Unfortunately, I am well aware of issues that do emerge on a regular base when the compiler is sufficiently forgiving...

But anyway, enough about this, see you all over in the development forums.
Post Reply