Opensource TESAnnwyn, a possible plugin for the Editor

[psi29a]

As some of you might know, Hircine has been busy make OpenMW's first demo suite using one of my heightmaps. He used a tool called TESAnnwyn (currently closed source windows only binary). I decided to track down the maker: Lightwave (aka Paul) and ask a few questions.

Hello Lightwave,

at least, I hope this is Lightwave of TESAnnwyn fame. I was wondering
if you ever thought of open-sourcing your projects?

I'm BrotherBrick from OpenMW (google us if you like), and we've used
your tool in the past to convert BMPs (8-bit greyscale) to an usable
esm file in OpenMW. I see that you support 8/16/32-bit RAW files and
8-bit Greyscale, but you do not mention 16-bit Greyscale which can
give us better precision (64K instead of 256 values per pixel). There
is even 48-bit colour BMP that give 64K values per pixel as well. Are
there any plans to support PNG (8-bit and 16-bit) heightmaps? Even
better would be support for a more precise format, something like HDF
(http://en.wikipedia.org/wiki/Hierarchical_Data_Format) which can
store heightmap information with 64-bit float (double) precision per
pixel/element.

I figured if there was no desire to open up the code, then perhaps you
could give me a few pointers on how to go about creating a ready esm
from an existing heightmap.

Thank you for your time.

Cheers,
BrotherBrick

[psi29a]

Hi BrotherBrick,

I had always intended to open up TESAnnwyn as a GNU project (it's in the C source code! ) but never got round to it - largely embarrasment due to the number of evolved hacks as the project developed over time, it could do with a lot of tidying and restructuring. It's fairly tight and efficient and doing the heavy work though and apart from ZLIB (required to compress and decompress certain ESM/ESP records for TES4+ style files) it doesn't use any third party libraries, so is quite platform independent (I always develop on Linux - it runs 10x faster than the C code when compiled on 32-bit Windows 7, though for 32-bit XP it was nearly as quick). It's really not at that stage though unfortunately (and bolting a front-end on, even as a separate exe was always my intention, but other projects and life have stalled that indefinitely).

I really like the OpenMW project idea, that's quite ambitious of you guys but well done getting as far as you have.

All the heightmap formats TESAnnwyn supports are very simple; just a regularly spaced rectangular or square 2D grid, each point of which represents an absolute height value. Each increment of height represents what I've called a 'THU' (TES height unit), which is the smallest increment of height that Morrowind, Oblivion, Fallout3 and Skyrim support. 1 THU is exactly 8 game units, and AFAIR is approx 11cm or 14cm (can't remember exactly it's been a while!) For OpenMW to be compatible it will already be doing the same thing as TESAnnwyn.

For 8-bit and 16-bit BMP and RAW files, each height value is stored as an integer. AFAIR the 32-bit support interprets each height value as floating point, so you can be very very precise with those, though fractions less than 1 would have no meaning and get rounded up or down due to the poor resolution of the TES game engines. A RAW file is perfect simplicity, data stored in rows, two bytes per height (a height range of -32768 to +32767 THU). Hence a RAW heightmap which is 2048x512 RAW file is 4096 bytes west->east by 1024 bytes north->south (in Morrowind that's 32x8 cells). A BMP is just slightly bigger, it contains a small header (AFAIR 54 or 56 bytes) at the front of the file which tells you the bit-depth and image dimensions. The rest of the image data is exactly the same, whether RAW or BMP; there is a tiny difference that BMP files start filling rows of height from the south-west of an image, whereas RAWS start from the north-west, though TESAnnwyn actually ignores this so that the importer code is identical regardless of what format you're using, but results RAW files being vertically flipped when imported (one of the caveats in the Readme is to do a north-south flip on RAW files to correct it).

Btw, a 16-bit RAW file is already what you might call 16-bit greyscale; i.e. if you load it in to Photoshop it is displayed as a 16-bit greyscale file. But obviously the colours shown by an imaging program are just a visual representation of raw pixel data so a human can better perceive its meaning. Photoshop still only shows you 256 colours though as 24-bit graphics cards can only render 256 shades of grey (which is still much better than that bestseller currently storming the book market. ) The data is there in the background, but no PC graphics card will display it to you.

Since a RAW file is raw height data without any header then if you want to open it in an imaging program (e.g. Photoshop) then you obviously need to specify the image dimensions, bit depth and byte order (big endian or little endian) to the imaging program in order for it to look right. The imaging program will represent that to you as a greyscale file. Basically think of most image formats other than RAW as a header (which can optionally contain colour overrides for each pixel) and the image data itself. The BMP header colour overrides is basically a visual filter, changing how you see it, but not changing the raw data. There is actually no difference in the image data between an 8-bit greyscale image and an 8-bit colour image. Likewise for a 16-bit greyscale and a 16-bit colour file, the data stays the same (an exception is if you reprocess each pixel's colour and turn them in to shades of grey, but then you actually end up with is a 8-bit detail, not 16-bit detail). The difference between the colour file and the greyscale file is simply that a greyscale file contains lots of colour replacements in the header to filter what you see, so it looks different to the human eye. The image data value for each pixel itself never changes; hence whether it's colour or greyscale is only of help to the human eye, not when processing heightmaps in programs.

An 8-bit BMP file contains colour overrides so that height 0 has RGB #000000, height 1 has RGB #010101, height 2 has RGB #020202 and so on, up to #FFFFFF. This means you have up to 256 shades of grey.

An 16-bit BMP file contains colour overrides so that heights 0 to 255 have a black RGB #000000, height 256 to 511 has RGB #010101, heights 512 to 767 have RGB #020202 and so on, up to #FFFFFF (covering the last height range up to 65535, which appears as white). With 24-bit RGB you still only have 256 shades of grey across the range and no mainstream graphics cards that I'm aware of improve on that; one also needs a monitor that can support more than 16 million colours to do more shades of grey (mine does, but it's wasted on the 24-bit graphics card.

So all that really matters is that the data in the image is detailed enough. For the TES game engines (everything from Morrowind, through the Fallout3 games and including Skyrim) 16-bit is generally what you would use since the land isn't that refined. A 16-bit file gives you a height range of more than 9km (65536 * 0.14 metres), which is far more than the useful range of the TES game engines; horrible rendering problems start showing up long before this with the game engine Bethesda use. The original Morrowind game actually has a height cap of 32760 game units of height - landscape higher than that overflows and becomes displayed as -32768 height instead. The Construction Set can display more, but the game cannot. When I converted Oblivion to run on the Morrowind game engine, I had to logarithmically rescale the mountain heights so they didn't exceed the Morrowind limit (which meant softening the peaks).

Oblivion and above can render much much higher terrain heights, but in all the game engines (Morrowind to Skyrim) you see horrible animation jitters started to occur, like the player and NPCs have nervous issues, and at greater distances clothes and weapons float alongside the NPC's bodies. Hence 16-bit files more than amply cope with the useful height range of the games and no-one is currently using anything that requires more range.

These Bethesda quirks probably don't affect your game engine though (hopefully) - so you could potentially go much higher, or represent finer height detail in the landscape than the 14cm (ish) jumps that Bethesda use: However all the ESM/ESP files using by the TES games wouldn't be compatible with this greater range so I'd guess your game engine isn't trying to do any of this?

32-bit support really was added as a whim by me (just because it was a simple step to provide 32-bit over 16-bit). However as explained it's well outside the usable range of the TES games so no-one would need it (except potentially importing a heightmap from a 32-bit heightmap generator that can't generate any other format). Likewise anything higher, such as 48-bit is really quite pointless - unless you're designing an entirely new type of game that isn't compatible with Morrowind.

I never considered PNG support (or any compressed support) as there doesn't seem much need or purpose; their only advantage is when transferring them from one person to another due their decreased size (I usually use .7z heightmaps if emailing them). Otherwise it slows the whole process down having to uncompress them to their original size to process them (or vice versa), using more disk space in the process, when they can be turned in to RAW or BMP files using any image editor or convertor.

There are (lots!) of examples in the TESAnnwyn Readme on how to turn a heightmap in to an ESP. You need to understand the format of your source data though (the bit-depth and are values stored as integers or floating points?) Most people develop 16-bit RAWs using Photoshop (kinda tricky) or using other heightmap editors (the buggy Construction Set ones used by Oblivion, Fallout3, Skyrim use 16-bit RAWs). Most greyscale images that people get hold of are just 8-bit, so they have to use the "-s" parameter when importing to scale up the heights (otherwise height 255 only gives a max height of 29 metres), then export it as a 16-bit RAW, blur it a few times in Photoshop and re-import it (the blurring of a 16-bit image gets rid of the steppiness created using.

Obviously heightmaps exported and imported from the TES games require no processing - provided you use 16-bit format (BMP or RAW, it does not matter) then they will always be precisely reproduced (and the landscape rendering format is the same for all the games).

Thanks for the HDF suggestion, though it looks like gross overkill for non-vector heightmap data, which is really a very simple form of data; just a 2D grid with regular spaces points, each of which has a single value (height). Keeping it simple is good. If there was another fairly standard format for highly detailed formats I would consider supporting that, but no-one has (thus far) come across and asked me about one.

I guess you're asking me because you have a heightmap supplied in a certain format, but don't know how to go about importing it (?) What do you know about this format you have?

Good luck and keep up the cool work,

Lightwave.

[psi29a]

Hello Lightwave,

I just finished reading your wall of text!

Thank you for taking the time to respond.

While I agree that 256 (8-bit) of grey is better that 50 (no desire to
read or watch), the point of 16-bit was to get 64k worth of precision
instead of just 256 which makes a world of difference when converting
from a double (0.0f - 1.0f) data set with 64-bit precision. I'm going
to lose precision with the conversion to an image regardless of format
(raw, bmp or png). 48-bit images do exist only to give each channel
16-bit values instead of 8-bit per channel in a 32/24-bit image. So
48-bit colour is the equivalent of 16-bit greyscale. I only see
'images' as storage medium that just happens also allow you to better
visualize the data, sadly with the trade-off of precision.

I have my own tool (python based, multi-platform) that I will
eventually open-source that is tentatively called "WorldGen' which
does exactly what it implies and exports the data as hdf to keep the
64-bit precision heightmap, it also exports 16-bit greyscale PNG
(trivial to work with and uses either lossless or no compression). The
OpenMW team is creating a blank ESM based on a heightmap I made for
them with this tool and used TESAnnwyn to convert it. Great work by
the way. My only disappointment was that it only dealt with the
heightmap (BMP) as 8-bit (256, -128 to 128) when I had height data
from 64-bit (1.7E +/- 308 (15 digits)) of precision.

Anyway, would you be willing to make a linux and macosx binary
available? We are mostly Linux guys anyway. This would be a great
stepping stone before releasing the code itself (when you are ready).
You could also just throw it up on github and let others help you
clean it up.

I will be releasing my WorldGen tool shortly once I've stabilized the
GUI and merged all my branches in the repo. In case you are
wondering, it can generate a heightmap based on either MDA, DSA and
SphereOverlapAlgo. It also handles heatmaps, wind and precipitation
(with rain shadows), river and lake generation, drainage, basic
erosion and biomes. The only things left are is to make it more user
friendly (better configuration dialog via pySide, a QT python lib). I
have plans to eventually do civilization simulations with it. OpenMW
would like it be LGPL so that they can use it conjunction with their
OpenMW Editor (better version of the TES3:CS).

This leads me to you! If you and I can contribute our projects to the
OpenMW Editor (as either GPL or LGPL) then we could have a fully
funtional suite of tools to allow people create their own worlds. How
cool would that be!

RIght, back to work here. Have a good one.

Cheers,
BrotherBrick

[sirherrbatka]

Topic moved...

[psi29a]

Topic moved...

Then move it back, it has to do with the OpenMW Editor which as far as I can tell is not off-topic but general discussion with direct impact on the OpenMW Editor project as a whole.

[sirherrbatka]

ok, you are right

[Greendogo]

Wow, there has been some impressive discussion on this.

I hope he decides to open-source or release linux and mac binaries.

[psi29a]

Emphasis mine...

Heya BrotherBrick!

No not at all, just trying to find time to get back to you (busy in work in the daytime, but managed to sneak a while to reply to you yesterday - but it's been crazily busy today). Linux binary no problem, I'll just find the last stable release I released for Windows (on my home PC ... somewhere!). I should probably just give you the code ... really, I know. I always test and develop on Linux anyway.

I do appreciate the detail drop going from 64-bit/32-bit to 16-bit heights. What I was trying to point out was that for the TES games it's really a moot point because they can't render really tall landscape or really fine landscape. A 16-bit file provides the finest detail the TES games can offer from sea level all the way up to heights higher than Mt Everest (up to 9000km height range). Of course if you really wanted to go higher that then you will need more detail than 16-bits can offer (TESannwyn supports 32-bits, but no-one in their right mind would use it on the existing game engines) - if you use a different game engine that allows finer height jumps, then it is also needed. The highest mountain in Oblivion (which is the tallest mountain in Skyrim - the landscapes along the border are similar in both games) is actually only 750m tall - Bethesda haven't even used 1/10th of the available height in a 16-bit file. I originally based my format to be completely compatible with the TES4 Construction Set RAW files (16-bit integers where each increment corresponds to one height increment in the landscape - which is approx 14cm). I can support 64-bits of course, but it just wouldn't be useful for the TES games - unless there's a plan to make openMW render finer landscape than the Bethesda game engine(?) - to do that you'd need to convert all existing ESPs to store data at the new scale.

AFAIR the landscape records in an ESP are actually 8-bit heights - or rather gradients, so each height is actually the height difference from the point adjacent to the west; there's a 32-bit float at the beginning of each cell that states the starting height for the bottom left (south-west) corner of the cell; every other value is an 8-bit offset. The heightmap is generated by taking the starting height of each cell, then summing the gradients on each row to determine the actual height at each point. Took me a while to decode the landscape format (my original project was when converting TES3 ESPs to TES4), but got there eventually, easy once you know how.

Will get back to you on the Linux binary/source when I find it.

Cheers,
Lightwave

[ElderTroll]

Very exciting news brotherbrick. There's this whole disparate community of artists, programmers, and modders that I think have been waiting for something like OpenMW to come along and unify it all.

[Tarius]

Very exciting news brotherbrick. There's this whole disparate community of artists, programmers, and modders that I think have been waiting for something like OpenMW to come along and unify it all.

Actually an interesting thought here. Everyone knows about Source. That isnt exactly open source though. I would look at the mod community around the Source engine as a comparison for this.