Resource loading outside of bsa files

[lgromanowski]

So I am looking for somewhere to start with OpenMW development I figure I will tackle: http://github.com/korslund/openmw/issues#issue/5.

So the question is where should I start?

I have been looking around the code a bit but don't have much of an idea, hopefully this question isn't too general or somesuch.

Welcome!

Welcome!

Thanks.

Also I have been trying to build OpenMW(on Ubuntu Linux 10.04) and I have all of the dependencies and cmake generates a Makefile fine but I get errors about files in extern/caelum trying to include a "Ogre.h" and I look in these files and they are trying to include a "Ogre.h", I don't know if I am missing a file or if these should be including "<OGRE/Ogre.h>".

First a few comments regarding github:

- You better comment on the issues in Nico's repository. If you add issue comments in your own fork they most likely will get lost.

- It seems something went wrong with your commit. git considers your cpp file a binary.



Regarding the build problem. The code is correct. If it doesn't build you have a setup or configuration problem.

Regarding the issue you want to tackle: I don't know much about the whole resources handling. Nico wrote all the code. I remember though that he recently added a file_finder component, that successfully sorted out most of the problems for sound files.

Welcome!

That is correct about file_finder. The problem with path finding is mostly that Unix file systems are case sensitive, while the dataset uses all kinds of cases on filenames it references (and also uses \ dir separators.) The filefinder I wrote browses a given dir tree at startup, and lets you convert file names at runtime. So it converts eg "Fx\item\LongBlad.wav" to "/your/data/dir/Sound/Fx/item/longblad.wav" (if that was an actual file found at startup.) It doesn't do any file reading or anything like that, it's just a pure filename lookup.

It's currently only used in apps/openmw/mwsound/soundmanager.cpp. Another place it would be great to use it is for meshes and textures in components/nifogre/

- It seems something went wrong with your commit. git considers your cpp file a binary.

Actually that file is still empty, dunno why github lists that as binary.

It's currently only used in apps/openmw/mwsound/soundmanager.cpp. Another place it would be great to use it is for meshes and textures in components/nifogre/

I guess then that having it used for meshes and textures would be the logical first step here then.

Heres some build outputs:

Build output from: make caelum

Code: Select all

zachary@zachary-crunch:~/documents/openmw_build/build$ make caelum 
Scanning dependencies of target caelum
[  0%] Building CXX object extern/caelum/CMakeFiles/caelum.dir/src/UniversalClock.cpp.o
In file included from /home/zachary/documents/openmw_build/extern/caelum/src/UniversalClock.cpp:21:
/home/zachary/documents/openmw_build/extern/caelum/include/CaelumPrecompiled.h:24:18: error: Ogre.h: No such file or directory
In file included from /home/zachary/documents/openmw_build/extern/caelum/include/UniversalClock.h:24,
                 from /home/zachary/documents/openmw_build/extern/caelum/src/UniversalClock.cpp:22:
/home/zachary/documents/openmw_build/extern/caelum/include/CaelumPrerequisites.h:114: error: â??Ogreâ?? has not been declared
/home/zachary/documents/openmw_build/extern/caelum/include/CaelumPrerequisites.h:115: error: â??Ogreâ?? has not been declared
/home/zachary/documents/openmw_build/extern/caelum/include/CaelumPrerequisites.h:116: error: â??Ogreâ?? has not been declared
/home/zachary/documents/openmw_build/extern/caelum/include/CaelumPrerequisites.h:117: error: â??Ogreâ?? has not been declared
/home/zachary/documents/openmw_build/extern/caelum/include/CaelumPrerequisites.h:118: error: â??Ogreâ?? has not been declared
/home/zachary/documents/openmw_build/extern/caelum/include/CaelumPrerequisites.h:120: error: â??Ogreâ?? has not been declared
/home/zachary/documents/openmw_build/extern/caelum/include/CaelumPrerequisites.h:121: error: â??Ogreâ?? has not been declared
/home/zachary/documents/openmw_build/extern/caelum/include/CaelumPrerequisites.h:124: error: â??Stringâ?? does not name a type
/home/zachary/documents/openmw_build/extern/caelum/include/CaelumPrerequisites.h:130: error: â??Ogreâ?? has not been declared
/home/zachary/documents/openmw_build/extern/caelum/include/CaelumPrerequisites.h:131: error: â??Ogreâ?? has not been declared
/home/zachary/documents/openmw_build/extern/caelum/include/CaelumPrerequisites.h:132: error: â??Ogreâ?? has not been declared
/home/zachary/documents/openmw_build/extern/caelum/include/CaelumPrerequisites.h:133: error: â??Ogreâ?? has not been declared
/home/zachary/documents/openmw_build/extern/caelum/include/CaelumPrerequisites.h:134: error: â??Ogreâ?? has not been declared
/home/zachary/documents/openmw_build/extern/caelum/include/CaelumPrerequisites.h:135: error: â??Ogreâ?? has not been declared
/home/zachary/documents/openmw_build/extern/caelum/include/CaelumPrerequisites.h:136: error: â??Ogreâ?? has not been declared
/home/zachary/documents/openmw_build/extern/caelum/include/CaelumPrerequisites.h:171: error: expected initializer before â??&â?? token
In file included from /home/zachary/documents/openmw_build/extern/caelum/src/UniversalClock.cpp:22:
/home/zachary/documents/openmw_build/extern/caelum/include/UniversalClock.h:37: error: expected initializer before â??UniversalClockâ??
In file included from /home/zachary/documents/openmw_build/extern/caelum/src/UniversalClock.cpp:23:
/home/zachary/documents/openmw_build/extern/caelum/include/Astronomy.h:44: error: expected initializer before â??Astronomyâ??
/home/zachary/documents/openmw_build/extern/caelum/src/UniversalClock.cpp:83: error: expected â??}â?? at end of input
make[3]: *** [extern/caelum/CMakeFiles/caelum.dir/src/UniversalClock.cpp.o] Error 1
make[2]: *** [extern/caelum/CMakeFiles/caelum.dir/all] Error 2
make[1]: *** [extern/caelum/CMakeFiles/caelum.dir/rule] Error 2
make: *** [caelum] Error 2

Build output from: make MyGUIOgrePlatform

Code: Select all

zachary@zachary-crunch:~/documents/openmw_build/build$ make MyGUIOgrePlatform 
[  0%] Building CXX object extern/mygui_3.0.1/OgrePlatform/CMakeFiles/MyGUIOgrePlatform.dir/src/MyGUI_OgreDataManager.cpp.o
In file included from /home/zachary/documents/openmw_build/extern/mygui_3.0.1/OgrePlatform/src/MyGUI_OgreDataManager.cpp:10:
/home/zachary/documents/openmw_build/extern/mygui_3.0.1/OgrePlatform/include/MyGUI_OgreDataStream.h:14:28: error: OgreDataStream.h: No such file or directory
/home/zachary/documents/openmw_build/extern/mygui_3.0.1/OgrePlatform/src/MyGUI_OgreDataManager.cpp:12:18: error: Ogre.h: No such file or directory
In file included from /home/zachary/documents/openmw_build/extern/mygui_3.0.1/OgrePlatform/src/MyGUI_OgreDataManager.cpp:10:
/home/zachary/documents/openmw_build/extern/mygui_3.0.1/OgrePlatform/include/MyGUI_OgreDataStream.h:24: error: â??Ogreâ?? has not been declared
/home/zachary/documents/openmw_build/extern/mygui_3.0.1/OgrePlatform/include/MyGUI_OgreDataStream.h:24: error: expected â??)â?? before â??_streamâ??
/home/zachary/documents/openmw_build/extern/mygui_3.0.1/OgrePlatform/include/MyGUI_OgreDataStream.h:33: error: â??Ogreâ?? has not been declared
/home/zachary/documents/openmw_build/extern/mygui_3.0.1/OgrePlatform/include/MyGUI_OgreDataStream.h:33: error: ISO C++ forbids declaration of â??DataStreamPtrâ?? with no type
/home/zachary/documents/openmw_build/extern/mygui_3.0.1/OgrePlatform/include/MyGUI_OgreDataStream.h:33: error: expected â??;â?? before â??mStreamâ??
/home/zachary/documents/openmw_build/extern/mygui_3.0.1/OgrePlatform/src/MyGUI_OgreDataManager.cpp: In member function â??virtual MyGUI::IDataStream* MyGUI::OgreDataManager::getData(const std::string&)â??:
/home/zachary/documents/openmw_build/extern/mygui_3.0.1/OgrePlatform/src/MyGUI_OgreDataManager.cpp:66: error: â??Ogreâ?? has not been declared
/home/zachary/documents/openmw_build/extern/mygui_3.0.1/OgrePlatform/src/MyGUI_OgreDataManager.cpp:66: error: expected â??;â?? before â??streamâ??
/home/zachary/documents/openmw_build/extern/mygui_3.0.1/OgrePlatform/src/MyGUI_OgreDataManager.cpp:67: error: â??streamâ?? was not declared in this scope
/home/zachary/documents/openmw_build/extern/mygui_3.0.1/OgrePlatform/src/MyGUI_OgreDataManager.cpp:71: error: expected type-specifier before â??Ogreâ??
/home/zachary/documents/openmw_build/extern/mygui_3.0.1/OgrePlatform/src/MyGUI_OgreDataManager.cpp:71: error: expected â??)â?? before â??::â?? token
/home/zachary/documents/openmw_build/extern/mygui_3.0.1/OgrePlatform/src/MyGUI_OgreDataManager.cpp:71: error: expected â??{â?? before â??::â?? token
/home/zachary/documents/openmw_build/extern/mygui_3.0.1/OgrePlatform/src/MyGUI_OgreDataManager.cpp:71: error: â??::FileNotFoundExceptionâ?? has not been declared
/home/zachary/documents/openmw_build/extern/mygui_3.0.1/OgrePlatform/src/MyGUI_OgreDataManager.cpp:71: error: expected â??;â?? before â??)â?? token
/home/zachary/documents/openmw_build/extern/mygui_3.0.1/OgrePlatform/src/MyGUI_OgreDataManager.cpp: In member function â??const MyGUI::VectorString& MyGUI::OgreDataManager::getDataListNames(const std::string&, bool)â??:
/home/zachary/documents/openmw_build/extern/mygui_3.0.1/OgrePlatform/src/MyGUI_OgreDataManager.cpp:94: error: â??Ogreâ?? has not been declared
/home/zachary/documents/openmw_build/extern/mygui_3.0.1/OgrePlatform/src/MyGUI_OgreDataManager.cpp:94: error: expected â??;â?? before â??pFileInfoâ??
/home/zachary/documents/openmw_build/extern/mygui_3.0.1/OgrePlatform/src/MyGUI_OgreDataManager.cpp:96: error: â??pFileInfoâ?? was not declared in this scope
/home/zachary/documents/openmw_build/extern/mygui_3.0.1/OgrePlatform/src/MyGUI_OgreDataManager.cpp:98: error: â??Ogreâ?? has not been declared
/home/zachary/documents/openmw_build/extern/mygui_3.0.1/OgrePlatform/src/MyGUI_OgreDataManager.cpp:98: error: expected â??;â?? before â??fiâ??
/home/zachary/documents/openmw_build/extern/mygui_3.0.1/OgrePlatform/src/MyGUI_OgreDataManager.cpp:98: error: â??fiâ?? was not declared in this scope
make[3]: *** [extern/mygui_3.0.1/OgrePlatform/CMakeFiles/MyGUIOgrePlatform.dir/src/MyGUI_OgreDataManager.cpp.o] Error 1
make[2]: *** [extern/mygui_3.0.1/OgrePlatform/CMakeFiles/MyGUIOgrePlatform.dir/all] Error 2
make[1]: *** [extern/mygui_3.0.1/OgrePlatform/CMakeFiles/MyGUIOgrePlatform.dir/rule] Error 2
make: *** [MyGUIOgrePlatform] Error 2

Build output from 'make openmw' is the same as 'make MyGUIOgrePlatform' because it has to build 'MyGUIOgrePlatform' anyway.

'make MyGUIOgre' and 'make esmtool' both compile fine.

All these builds were attempted against the latest revision here:
http://github.com/korslund/openmw

Fixed my build problem by setting the CPLUS_INCLUDE_PATH environment variable to /usr/include/OGRE/, yay!

An update, I have resource overides loading ingame now, and being used instead of the .bsa contents, I think some of my code is bad at the moment, and an feature that I have not implemented is external resources would only be loaded if they had a newer timestamp than the bsa content.

Git branch: http://github.com/Mach1723/openmw/tree/ ... sitive_bsa

Screenshots of some of the problems\the resource loading: http://picasaweb.google.com/Mach1723/Openmw#

Sorry, does not work here. I am still getting

NIFLoader: Warn:File not found.

I assume you are going through the file finder and the path problems (case and path element separator) should have been addressed. But unfortunately they are not.

Also, the performance is bad. We will have to do something about that, but the correctness is more important now.

This is only guesswork, but some re-enabled and extended diagnostics indicate, that the problem only shows up with resources, that do not exist in the bsa at all.

This is only guesswork, but some re-enabled and extended diagnostics indicate, that the problem only shows up with resources, that do not exist in the bsa at all.

If you have the time it would be helpful if you could uncomment the cout's at line 157-159 of components/bsa/bsa_file.cpp and paste the output somewhere, it would be useful to know what FileFinder is returning to be used as a path.

That's what I meant with

some re-enabled and extended diagnostics

Little point in posting that though, because this part is working fine. All those files are handled correctly and I don't get any errors for them.

The resources, that are not found, are not listed at all by those couts.

Little point in posting that though, because this part is working fine. All those files are handled correctly and I don't get any errors for them.

The resources, that are not found, are not listed at all by those couts.

I am not quite understanding the issue at this point.. do the resources have an equivalent in the bsa?

No, they don't. That exactly is the point.

No, they don't. That exactly is the point.

I was planning on adding the loading of non-bsa equivalent resources when we have multiple esp/esm support, since I don't see the point of loading a resource that isn't referenced by an esp/esm/bsa.

Okay, now I understand. Well, what you have implemented is more like an edge case (but thanks for bringing it up; might have missed it otherwise), while what you are still planning to do is the main issue.

Regarding no point in finishing it up now: Take any total conversion (Redemption for example, but IIRC there is at least another one out there). These usually only use one esm, which might not use a bsa at all.