Project Directory Structure

Everything about development and the OpenMW source code.
Locked
User avatar
lgromanowski
Site Admin
Posts: 1193
Joined: 05 Aug 2011, 22:21
Location: Wroclaw, Poland
Contact:

Project Directory Structure

Post by lgromanowski »

athile wrote: Since the C++ version of OpenMW is still at an early stage, I was wondering if it might be a good time to create a consistent, flexible directory structure that will allow OpenMW to expand over time. This might be overkill at this point in the project, but I thought I would share the idea to let others consider it - even if it is decided that it's not yet necessary.

Below is what I had in mind, with an explanation following.

Code: Select all

openmw\
    apps\               
        openmw\
    components\
        engine\
            input\
            renderer\
            nif\
            ...
        game\
    plugins\
    libs\
        platform\
        mangle\
        monster\
    resources\
    extern\
        caelum\
    benchmarks\
    unittest\
    samples\
    tools\
Here's an explanation of some of the directories:
  • apps - this would be the executable that basically just loads and drives the "engine" and "game" components. If someone writes a TES editor replacement, this would be a sub-directory under "apps"
  • components - this would essentially be the DLLs or linux shared libraries. The two major components would be the "engine" which is the general game code (basically everything currently in the OpenMW source would fall under this) and the "game" which is any Morrowind-specific game logic, scripts, etc. for specific entities, places, etc. This would be similar to how id software has traditionally split the Quake games into a engine and a "game" DLL which they let modders modify.
  • plugins - for any smaller, pluggable parts of the engine. Perhaps the music player, the weather rendering engine, etc.
  • libs - for static libraries reusable throughout the project. For example, moving all the platform-independent wrapper code into a lib, any math helper libraries, etc.
  • resources - textures, OGRE materials, etc.
  • extern - any third-party libraries that OpenMW uses
  • benchmarks - automated tests to measure OpenMW's performance as well as the performance of the various components, plug-ins, libs, etc.
  • unittest - automated tests to ensure the components are all working correctly
  • samples - a kind of "developer's sandbox" where uses of the engine, experimental code, work-in-progress features, modding examples, etc. could go
  • tools - any standalone executables used internally
nicolay wrote: Great work, this is a good idea. I've been thinking about this myself lately, and I think your structure is basically sound. I have a few suggestions:
  • The engine/ dir should IMHO only contain things that are generic and not tied to OpenMW. The code that is in there now have no direct relation to Morrowind and could be reused in other games (in fact I plan to do so myself.) I would put nif/ bsa/ nifogre/ and similar directories directly in the components/ dir.
  • As it stands now, I think everything that is currently in the game/ directory would fit nicely into apps/openmw/, although that might change. Or did you intend the components/game/ directory for some other use as well?
  • I would drop the plugins/ dir along with any other dir that is currently empty, until we know we need it. There might not be that clear a distinction between a plugin and a component IMHO.
With that in mind, here's my suggestion, with comments:

Code: Select all

openmw\
    apps\               
        openmw\                - contains all the game code, uses components
    components\
        engine\                - reusable and game-independent components
            input\
            renderer\
            ...                - will add sound, physics, etc
         nif\
         bsa\
         caelum\               - our code for caelum (might fit in apps/openmw)
         misc\                 - what is now called tools/
    libs\
        platform\
        mangle\
    extern\
        caelum\                - for the actual library itself I assume?
I particularly like your apps/ structure. Not only does it suggest an editor, but also that if someone eg. wants to redo all the game logic in a new and radically different way (like, everything in Python) then they could make openmw_myversion and write an independent implementation there without conflicts.
Zini wrote: If we are going to have this change, could we have it ASAP, so that it is out of the way? Working on a fork with the upstream codebase suddenly receiving a radical directory structure change sounds ungood (even with a tool as advanced as git).
Zini wrote: Well, unless athile wants to have a go at it, I can do it myself as well. I need some clarifications about the desired directory structure though.

Following your proposal I would transform the current directory structure into this:

Code: Select all

openmw\
    apps\               
        openmw\   (the old game directory)
            mwinput\
            mwrender\
    components\
        engine\
            input\
            renderer\
                ogre\
                nifogre\
         nif\
         bsa\
         esm\
         esm_store\
         misc\ (old tools directory)
    libs\
        platform\
        mangle\
Does this look acceptable to you?
nicolay wrote: Looks great, except I'd put nifogre/ out in the components dir besided nif, bsa etc. The logic being that nifs are mw-specific and something that's sort of added to the engine, not part of the game engine itself.
athile wrote: Nicolay, your changes to the originally posted structure sound good to me!

Zini, it would be great if you could take the task :) I'm new enough to git and CMake that I'm assuming you could handle this task better than I could.
Zini wrote: Working on it now. Adjusting the directories in the CMake files is a pain though. I know you don't like GLOBing source files. But can we at least glob the header? They aren't compiled directly anyway. It only affects the file listing in IDEs.
nicolay wrote: Guess that's not a problem with headers. But what happened to search and replace? ;-)

I agree the paths in the CMake files are a pain though. Maybe there is some way to shorten it? Eg. in Makefiles you can do:

Code: Select all

PARTS=file1 file2 file3
FILES=$(PARTS:%=dir1/dir2/%.cpp)
# Equivalent to FILES=dir1/dir2/file1.cpp dir1/dir2/file2.cpp dir1/dir2/file3.cpp
Zini wrote:
But what happened to search and replace?
Ended up as search and destroy in one place, so I decided the spare me the headache and do it manually.
I agree the paths in the CMake files are a pain though. Maybe there is some way to shorten it?
If there is, then I don't know about it. Anyway, it doesn't matter. I am almost done with the CMake files.
Zini wrote: Done. Kinda ... :(

I haven't found a way to cleanly move the mangle directory to a different location. The CMake files and the source files have been properly adjusted though.
I hope you have an idea about what to do with the sub-module stuff, because I am out of ideas.

Edit: If everything fails, we can leave mangle in the old location. I would need to change the CMake file and some of the source files back though.
nicolay wrote:
Zini wrote:Done. Kinda ... :(

I haven't found a way to cleanly move the mangle directory to a different location. The CMake files and the source files have been properly adjusted though.
I hope you have an idea about what to do with the sub-module stuff, because I am out of ideas.

Edit: If everything fails, we can leave mangle in the old location. I would need to change the CMake file and some of the source files back though.
Great work!

I fixed the submodule stuff by simply removing the whole thing and adding it anew in libs/mangle/. This seems to be the standard way of doing it, and it doesn't change much internally as a submodule is just a reference after all. For future reference, here's what I did:

Code: Select all

git rm .gitmodules (since it was the only entry)
edit .git/config and remove mangle entry (probably didn't need to do this)
git rm --cache mangle (note: no / at the end)
git submodule add git://github.com/korslund/mangle.git libs/mangle/
However, for everybody else it SHOULD be enough to do (after updating):

Code: Select all

git submodule update
(and maybe deleting the old mangle/ dir manually)
If this doesn't do it, let me know.

I've marked the Roadmap item as done!
athile wrote: I'm a bit confused about how the new directory structure is being used.

I have originally thought:
  • every top-level directory under apps would build to its own executable
  • every top-level directory under components would build to its own shared library / DLL
  • every top-level directory under libs would build to its own static library
For example, I was imagining a "openmw.exe" being built under apps and a "omwengine.dll" being built under components. For example, mwcompiler & mwinterpreter would likely have been built into static libraries: with the test executables linking against those libraries as well as openmw.exe linking against them.

Also, CMake is really designed for nested CMakeLists.txt files. Currently everything seems be put into the top-level CMakeLists.txt. I believe (to keep things organized), if something is building a separate executable, it should have its own CMakeLists.txt.

I don't mean to impose anything on how others do their work, but those were my thoughts on how the directory structure might work.
Zini wrote: I see little point in making component libraries dynamic. But apart from that, I have no objections.
athile wrote: The point of making components dynamic was the eventual possibility of OpenMW being both an core, reusable engine and a specific executable. This would be in case others ever wanted to write a new game using OpenMW as the underlying engine. At this early stage though, I see little need for it.
Zini wrote:
The point of making components dynamic was the eventual possibility of OpenMW being both an core, reusable engine and a specific executable. This would be in case others ever wanted to write a new game using OpenMW as the underlying engine.
Yes, but you can do that as well with static libraries.
athile wrote: I gave mwcompiler and mwinterpreter their own CMakeList files. (And thanks for the github link in the other thread, I may be closer to having fixed the chmod/line-ending problems from Windows). It's in commit http://github.com/athile/openmw/commit/ ... d7e228b838.

I don't want to touch the code you're working on, but should the files in openmw/components/interpreter and open/components/compiler actually go into directories openmw/libs/interpreter and openmw/libs/compiler. That way the cpp files don't need to be recompiled for both openmw.exe and the mwcompiler and mwinterpreter exectuables - they'd both be linking against the same static lib. Obviously this isn't necessary as the current project is compiler just fine, but it does seem like this would be a cleaner setup.

I'd offer to make this change myself but I don't want to interrupt you by moving the code you're currently working on.
Zini wrote: If you make components into (static) libraries, they don't need to be compiled twice either. I don't really agree with moving them from components to libs, because the compiler and the interpreter are really components. Being used in more than one executable isn't really an argument, because sooner or later that might become true for most components.
athile wrote: If "components" is used to build static libs, then what's the defining difference between the "components" and "libs" directories?

There's lots of ways to potentially organize this. I don't really care what's chosen as long as it's consistent and logical.
Zini wrote: Libs are libraries, that can be used for any generic purpose. Components are part of a game and its associated toolset. You can use the components to build a different game. You can use the libs as part of an entirely different kind of program.

At least that is how I see it. The directory layout wasn't my concept after all.

Thinking about it, mangle is probably not well placed in libs. But we had already enough trouble with moving it around, so I propose we leave it were it currently is.
nicolay wrote:
Zini wrote:Libs are libraries, that can be used for any generic purpose. Components are part of a game and its associated toolset. You can use the components to build a different game. You can use the libs as part of an entirely different kind of program.
That's more or less the way I see it too. Components are part of this project. Libraries are really separate projects that we are using.

There's also the dependency way of looking at it: libraries depend on nothing else but them selves (and possibly other libraries). Components can depend on libraries and on other components, but should have as few dependencies as possible. The can NOT depend on anything in the apps/ directory. The apps can depend on components and libraries, but not on each other, and they form stand-alone programs.

All in all I think the current layout is working pretty well. Lets keep it for a while and see how it works out.
ap0 wrote: I think that adding a "doc" folder would be great.
Zini wrote: Well, we have doc folders in the directories of sub-systems, that actually have additional documentation. And at the top-level we have the Doxygen-stuff.
Locked