.ess Importer

[AnyOldName3]

We'd love for someone to finish the ESS importer. We've got a few issues on the tracker with the right label, but there are probably a bunch of other things wrong with it that aren't documented. https://gitlab.com/OpenMW/openmw/-/issu ... 20Importer

[Mantar]

It's a volunteer project, the only reason the ess importer is incomplete is nobody's volunteered to fix the importer. If you want to do it, go for it!

[ProgramGamer]

Nice, just wanted to make sure I had yall's blessing before taking over someone else's work is all ^^

Since it seemed like a relatively easy first step, I went ahead and looked at how the journal is structured. Calling its format HTML though is a bit, uh, generous. I did make a very rudimentary parser that skips over tags and lets me read each entry's date and text, and I also made a function that splits the date into numbers (namely the day of the month, current month, and days since the save was started) but it currently only handles english saves since (I assume) saves in different languages have slightly different formats for the entry date, not to mention different month names. I'll take a look at all the save files that have been posted here to see if any are in different languages, otherwise I'll just create a bunch of early game saves in different languages to determine what their format is.

For reference, here's an example journal entry in an english save file:

Code: Select all

<FONT COLOR="9F0000">16 Last Seed (Day 1)</FONT><BR>
I've decided not to help Hrisskar.<P>

Every entry seems to be formatted this way. A date colored in red, a line break tag, a newline, the text of the journal entry, a paragraph tag (with no balancing closing tag), and finally another newline. The only identifying information for what quest an entry belongs to is its text, meaning that reading the .esm files at some point to determine the quest id is inevitable, assuming journal entry text is always unique. In theory it would be possible to extract the quest data and bake it into the converter, but I'd then worry about bethesda deeming that some sort of redistribution of their game assets, so best not go that route imo.

---

If I may go on a tangent for a second, while I appreciate the desire to not require the base .esm files to convert saves into the new format out of principle, I really don't think that's realistic if the converter is to behave as expected and convert saves losslessly in the future. Certain things will almost certainly need to be checked for consistency, and the journal is already an obvious case of the game files needing to be available to convert data at all.

While I agree that asking users to locate the .esm files and pass them into a command line tool themselves involves too much friction for the average player, that friction can be alleviated greatly by the tool being integrated into the launcher via some sort of save file conversion UI, which needs to be aware of the .esm files to start morrowind anyway. Not to mention ess-importer also currently loads the openmw configuration files regardless, which means it's aware of the install location of both the base game and openmw (assuming it's been configured), which gives us an obvious opportunity to provide reasonable defaults.

As an aside, mass conversion of all save files in the base game's save folder to the new format should probably be a feature. Checking character names and classes should be enough to split saves into character folders, and only loading the .esm files once to convert several saves at once seems like a fairly obvious optimization opportunity.

Anyway, that's all I have to report for now. Cheers!

[AnyOldName3]

I don't think it's the case that journal entry text has to be unique, and there's no need to know which quest each journal entry came from. As far as I can tell, the Morrowind journal genuinely just functions as a place timestamped text strings get stored. Entries that came from uninstalled mods persist and there are literally mods that add a textbox that let you write whatever you want into it, so there isn't really such a thing as an invalid string that needs removing even if the quest ID is stored and we've just missed it (e.g. if there's a separate table mapping journal indices to quest IDs). You just need to separate the time header and text, parse the time header, and use that information to fill in the fields in a journal entry in the OpenMW save. I've not found the type definition yet or I'd have sent it through.

[ProgramGamer]

Ahh, that simplifies things a lot then, and most of the work is already done! Just gotta find where the journal entries are stored and write them there I suppose

[psi29a]

This is pretty cool, it is a pretty cool little project so your welcome to it!

Also... while it was initially written in C++; we'd like it to stay that way to that we can all review and maintain it.

However, I wouldn't be opposed to it being re-written in Rust as it is a standalone program.

[EvilEye]

I'd be very surprised if there wasn't a way to get a journal entry's ID and index given the existence of https://gitlab.com/OpenMW/openmw/-/issues/5625

Also, it's not technically possible to match a recorded entry's text to its editor defined text. Not only are they not unique, they can contain variables.

[ProgramGamer]

I think rewriting the converter in rust would be more trouble than it's worth tbh, especially considering how much work the tool reuses from the rest of the codebase that would then have to be reimplemented. Unless good interop tools exist for rust/C++, I think the best course of action is still to stay in C++ land, which I don't hate anyway

---

The big issue with journal entries is they're stored as a text log in legacy saves and not as any kind of data structure, meaning what you see in the journal UI is pretty much the only thing you get. Variables wouldn't be an obstacle to getting the quest IDs since the text is stored before those get replaced, but if journal entries aren't always unique anyway then I guess that idea is dead in the water. Oh well!

---

Anyway, in the meantime I got journal entries to carry over! I had to modify some bits of the engine (namely MWDialogue::Journal::write and MWDialogue::Journal::readRecord) to make it work correctly and make it save legacy journal entries back to save files properly, but it does work, at least for english saves. The only obstacle there was that openmw saves some extra metadata with journal entries and checks for its presence when loading saves, so I had to add a special case to the saving/loading code for when journal entries lack that metadata, but other than that the journal is able to load and display those entries no problem! Glad the first task I decided to tackle turned out to be a relatively easy one, heh

[ProgramGamer]

Thanks to AnyOldName3 for linking me this documentation of the .ess format via discord, which wasn't previously linked in this thread and seems more detailed than the format documentation in this thread's first post, this will probably be a valuable resource to figure out some of the more obscure quirks of the format

[AnyOldName3]

I've told the forum software that we think you're a real human, so your posts don't need manual approval anymore and should show up immediately.