Journal alphabetical index with non-english content files

Everything about development and the OpenMW source code.
User avatar
akortunov
Posts: 146
Joined: 13 Mar 2017, 13:49
Location: Samara, Russian Federation
Github profile: https://github.com/akortunov

Journal alphabetical index with non-english content files

Post by akortunov » 19 Nov 2017, 12:14

This topic was created to discuss the bug #2485.

Basically, we have three issues here:
1. We do not have the "content language" setting (only have the "content encoding" one).
2. Index alphabete is hardcoded.
3. Comparison function is hardcoded.

So what we may to do:
1. Provide the missing setting.
2. Provide the some kind of JournalIndex class which will:
2.1. Build the jounnal index alphabete based on the content language and encoding
2.2. Provide the comparison function based on the content language and encoding
If someone is interesting how these functions may be implemented, take a look at my old code for the CP1251.
How it looks:
Screenshot_20171119_145101.png
Instead of 2.2 we can convert all texts to the UTF-8 on game start and implement the single UTF-8 case-insensitive comparator.

Any suggestions?
Last edited by akortunov on 20 Nov 2017, 14:51, edited 2 times in total.

User avatar
scrawl
Posts: 2085
Joined: 18 Feb 2012, 11:51
Contact:

Re: Journal alphabetical index with non-english content files

Post by scrawl » 19 Nov 2017, 14:29

convert all texts to the UTF-8 on game start
We already do. Note that the journal converts to UTF-32 before rendering, but I don't think that's relevant for any of the code you'd be touching.

Regarding encoding/language, is there any other language of Morrowind that uses the same encoding as the russian one? If there isn't, do we really need the language setting?

User avatar
akortunov
Posts: 146
Joined: 13 Mar 2017, 13:49
Location: Samara, Russian Federation
Github profile: https://github.com/akortunov

Re: Journal alphabetical index with non-english content files

Post by akortunov » 19 Nov 2017, 14:49

scrawl wrote:
19 Nov 2017, 14:29
Regarding encoding/language, is there any other language of Morrowind that uses the same encoding as the russian one? If there isn't, do we really need the language setting?
According to the feature #1161, no. But there are many languages that use windows-1252 encoding.
Or all versions with windows-1252 just use the default English index?

User avatar
scrawl
Posts: 2085
Joined: 18 Feb 2012, 11:51
Contact:

Re: Journal alphabetical index with non-english content files

Post by scrawl » 19 Nov 2017, 14:58

windows-1252 languages might use some accents e.g.

áéèúäüö

But I doubt that the index would list these because there are far too many combinations. I would guess that one of two things happens:
- Topics starting with an accent are not available through the index
- Accents are resolved to their 'base' letter (e.g. é shows under the index for e)

User avatar
akortunov
Posts: 146
Joined: 13 Mar 2017, 13:49
Location: Samara, Russian Federation
Github profile: https://github.com/akortunov

Re: Journal alphabetical index with non-english content files

Post by akortunov » 19 Nov 2017, 15:12

Can any French/German game user test how the Morrowind handles accents then?

As I understood, only Russian content files have serious issues with the journal index because of Cyrillics?

User avatar
akortunov
Posts: 146
Joined: 13 Mar 2017, 13:49
Location: Samara, Russian Federation
Github profile: https://github.com/akortunov

Re: Journal alphabetical index with non-english content files

Post by akortunov » 19 Nov 2017, 15:57

And another question: how can I get access to the "encoding" variable in the openmw.cfg during game session?

Chris
Posts: 1298
Joined: 04 Sep 2011, 08:33

Re: Journal alphabetical index with non-english content files

Post by Chris » 20 Nov 2017, 00:27

akortunov wrote:
19 Nov 2017, 15:57
And another question: how can I get access to the "encoding" variable in the openmw.cfg during game session?
Ideally you wouldn't. The game loads and converts the esm/esp files' text according to the encoding value, then during the game session it's all treated as Unicode (UTF-8, -16, or -32).

User avatar
akortunov
Posts: 146
Joined: 13 Mar 2017, 13:49
Location: Samara, Russian Federation
Github profile: https://github.com/akortunov

Re: Journal alphabetical index with non-english content files

Post by akortunov » 20 Nov 2017, 14:56

Chris wrote:
20 Nov 2017, 00:27
Ideally you wouldn't.
Ideally, yes. But I need to determine somehow if the game is using the Russian content files to build the Russian journal index.

Chris
Posts: 1298
Joined: 04 Sep 2011, 08:33

Re: Journal alphabetical index with non-english content files

Post by Chris » 20 Nov 2017, 15:49

akortunov wrote:
20 Nov 2017, 14:56
Ideally, yes. But I need to determine somehow if the game is using the Russian content files to build the Russian journal index.
Can't it be sorted as normal Unicode text would be, regardless of the content file text encoding? What would happen if a new game is made in Russian with UTF-8 content file encoding?

User avatar
akortunov
Posts: 146
Joined: 13 Mar 2017, 13:49
Location: Samara, Russian Federation
Github profile: https://github.com/akortunov

Re: Journal alphabetical index with non-english content files

Post by akortunov » 20 Nov 2017, 19:40

I created a pull request.

Any feedback would be helpful.

Post Reply

Who is online

Users browsing this forum: No registered users and 4 guests