Bangla Translations is broken in master! about unciv

[touhidurrr]

To be sure, I revived bangla-translations branch, and yes, it is working:

[yairm210]

This actually changed in fb47359
This means that Options -> Advanced -> Generate translation files did this
This could be a problem with the encoding we write the translation files in
In TranslationFileWriter line 236 it says "write the new files as UTF-8"
Does Bangla not work with UTF-8?

[yairm210]

No, I see it should

And we read is using that same charset
Not sure what's going on there, maybe the stringbuilder...?

[yairm210]

Anyway I would put a breakpoint in translationfilewriter and try to dig into where this change begins

[yairm210]

https://stackoverflow.com/questions/17907727/is-there-a-way-to-specify-the-character-encoding-to-the-java-lang-stringbuilder

Well damn
Java chars are always UTF-16 so I guess the back-and-forth encoding is what broke it
Then we should be able to solve this by just changing the read-and-write charset to UTF-16 and copy-pasting the old Bangla translations...?

[yairm210]

Hilariously "copy file" doesn't copy the bangla - not sure what encoding the clipboard uses, and damn am I happy that I have never had to think about that before, what a mess
Okay so that didn't work, the read-to-write flow works but the entire translation mechanism fails
It's midnight here, I'm going to bed

[yairm210]

NOPE reading UTF-8 files as UTF-16 did NOT work, that's almost certainly not the way
Not sure what is the problem
Debug TranslationFileWriter to find out, I guess

[touhidurrr]

It's midnight here, I'm going to bed

It's 3:00AM here actually. But anyways, good night, see you later!

[touhidurrr]

This actually changed in fb47359

Wait.................... This is not it................... I am pretty sure the reason is same as here: fb47359#r143772983

[touhidurrr]

Basically, diacritics support rewrites strings with unassigned codepoints from UTF-16. Then it asks the OS to generate the bitmap fonts for the strings we rewrote so that we can display them.
Now, when you press the Options -> Advanced -> Generate translation files button, original translations strings are already rewritten!! And this is causing the issue!!!

[touhidurrr]

As, for how to fix it... Since, diacritics support is supposed to do this on the fly.. The originals have to stay somewhere locally...
That button is needed to modified such as it uses the originals, not the modified ones!!!

[touhidurrr]

As for now, an easy fix would be copy paste my Bangla.properties from my pr and drop it to master. Unless we wait for this to be fixed. Again take my ping @SomeTroglodyte .

[touhidurrr]

This will let us deploy 4.12.7 but maybe we really should not. As this is also problematic:

Unciv/fastlane/metadata/android/bn/short_description.txt

Line 1 in fb47359

ওন  4X স র 

Unciv/fastlane/metadata/android/bn/full_description.txt

Lines 1 to 13 in fb47359

	সর সব ত স-ণ র এক ওন-  য়ন - ত, ট, পন ও !
	র স গ ন, গষ কন , ত কন শহর এ প কন শর!
	 অধ, গ   লপ  কর ন?  আ https://github.com/yairm210/Unciv/issues
	   ম  ন আর সক https://discord.gg/bjrB4Xw ;)
	অদ কর  কর  https://yairm210.github.io/Unciv/Other/Translating/
	অ কর ! আপর স পণত কন    ক সমর প!
	বরর  উনড এ র সমর জ 'স টও স' অম জয়৷ র  শন য় অ সম  অম  আর API এ অ।  টও বর অম মড , মড উনড, সত উনড এ র ম আপড/উনড কর বত হ। Unciv  অ ন র ইরট গ  ক হ ।

[SomeTroglodyte]

Basically, diacritics support rewrites strings with unassigned codepoints...

Exactly. Sorry I did not think of the effect on TFW. Basically, TFW would just need to load Bangla with that engine off I'd guess. Yea, "modified such as it uses the originals, not the modified ones".

ping

My Linux box still is not fully recovered - at least I proved the jumbled colours were due to the oibaf drivers. But checking Studio and the git state of Unciv I haven't had the courage yet, as it'll likely be frustrating with quite a few changes lost.

[touhidurrr]

My Linux box still is not fully recovered - at least I proved the jumbled colours were due to the oibaf drivers. But checking Studio and the git state of Unciv I haven't had the courage yet, as it'll likely be frustrating with quite a few changes lost.

You don't need to check that history. For correct Bangla.properties, copy paste it from #11755. The one at master has been corrupted.

[SomeTroglodyte]

Maybe:

Index: core/src/com/unciv/models/translations/Translations.kt
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/core/src/com/unciv/models/translations/Translations.kt b/core/src/com/unciv/models/translations/Translations.kt
--- a/core/src/com/unciv/models/translations/Translations.kt	(revision 9c864c34f0d7a65daed1395cce0137a57c108b73)
+++ b/core/src/com/unciv/models/translations/Translations.kt	(date 1719999331512)
@@ -82,7 +82,7 @@
     /** This reads all translations for a specific language, including _all_ installed mods.
      *  Vanilla translations go into `this` instance, mod translations into [modsWithTranslations].
      */
-    private fun tryReadTranslationForLanguage(language: String) {
+    private fun tryReadTranslationForLanguage(language: String, noDiacritics: Boolean = false) {
         val translationStart = System.currentTimeMillis()
 
         val translationFileName = "jsons/translations/$language.properties"
@@ -107,21 +107,22 @@
                     modsWithTranslations[modFolder.name()] = translationsForMod
                 }
                 try {
-                    translationsForMod.createTranslations(language, TranslationFileReader.read(modTranslationFile))
+                    translationsForMod.createTranslations(language, TranslationFileReader.read(modTranslationFile), noDiacritics)
                 } catch (ex: Exception) {
                     Log.error("Exception reading translations for ${modFolder.name()} $language", ex)
                 }
             }
         }
 
-        createTranslations(language, languageTranslations)
+        createTranslations(language, languageTranslations, noDiacritics)
 
         debug("Loading translation file for %s - %sms", language, System.currentTimeMillis() - translationStart)
     }
 
     @VisibleForTesting
-    fun createTranslations(language: String, languageTranslations: HashMap<String, String>) {
-        val diacriticSupport = DiacriticSupport(languageTranslations).takeIf { it.isEnabled() }
+    fun createTranslations(language: String, languageTranslations: HashMap<String, String>, noDiacritics: Boolean = false) {
+        val diacriticSupport = if (noDiacritics) null
+            else DiacriticSupport(languageTranslations).takeIf { it.isEnabled() }
         for ((key, value) in languageTranslations) {
             val hashKey = if (key.contains('[') && !key.contains('<'))
                 key.getPlaceholderText()
@@ -182,7 +183,7 @@
 
         DiacriticSupport.reset()
         for (language in getLanguagesWithTranslationFile()) {
-            tryReadTranslationForLanguage(language)
+            tryReadTranslationForLanguage(language, noDiacritics = true)
         }
         DiacriticSupport.freeTranslationData()
 

.. turns off the fake alphabet mapping from readAllLanguagesTranslation but not from tryReadTranslationForCurrentLanguage - I think the former is TFW-only , the latter gameplay-only. And the TFW one doesn't need cleanup because it runs on a separate instance of Translations() that is discarded after it's done. Likely has some unit-tests interaction though.

... but I still can't test or PR any code.

[SomeTroglodyte]

Hey Tou, this is worth a little chuckle:
https://search.f-droid.org/?q=unciv&lang=bn ...shows your fastlane short desc nicely...
Following the link, however, goes back to english. Which means they have a master selection of supported languages somewhere, and bn is not on it. I suspected as much but didn't really check until now...