Using --keyasareference with duplicate text breaks when converting back to i18next about i18next-gettext-converter

[oasalonen]

Has todo with using -K. Not even sure who and when this got introduced...should work without using it...

Yes that will fix the issue from a technical point of view and will probably be what I end up doing.

However, I'm using the -K (--keysasareference) switch because I want the reference text (English) to be the msgid in the gettext format. Without the -K switch, the resulting gettext file is

msgid "role-a.step.intro"
msgstr "Intro"

msgid "role-b.step.intro"
msgstr "Intro"

I'm trying to get the reference text in the msgid field because the translation tool (Mozilla Pontoon) makes the translation experience much, much better. See for example this snippet from their Firefox translation project. The translator sees the English version as a reference and even gets automated suggestions based on it. Without the '-K' switch, they see the original i18next key ('role-a.step.intro'), and the translation is marked as complete because there is content in the msgstr (the English reference).

[oasalonen]

But just to clarify: even if this is somehow fixed, I will not be using this approach going forward because all the instances of the 'City' text will get the same translation, which may not always be the case. I will use my original keys as msgids

[jamuhl]

Has todo with using -K. Not even sure who and when this got introduced...should work without using it...

[jamuhl]

I'm rather sure whoever implemented that feature did not expect that there are same keys in a file...

#: role-a##step##intro role-b##step##intro
msgid "Intro"
msgstr ""

that would need some extra delimiter separating keys - currently, it looks just like adding to it replacing . with ##

Wondering why Pantoon uses the key as reference text - Pantoon supports fluent (mozilla) which does not use source text in keys but pure technical keys...

If you like you might provide a PR -> https://github.com/i18next/i18next-gettext-converter/blob/master/src/lib/json2gettext.js#L170 should be the location it is used (but honestly no idea how difficult this will be to fix)

[oasalonen]

I've flattened my i18next JSON (no more nested keys) to help debug the issue. I'm seeing something strange going on which I don't yet understand. The flattened version looks now like this (I've added another string, 'City' to demonstrate the behavior):

{
  "role-a.step.intro": "Intro",
  "role-b.step.intro": "Intro",
  "role-a.step.city": "City",
  "role-b.step.city": "City",
  "city.page-title": "City",
  "role-a.profile.contact-details.city.label": "City",
  "role-b.profile.contact-details.city.label": "City",
}

The resulting gettext is this:

#: role-a.step.intro role-b.step.intro
msgid "Intro"
msgstr ""

#: role-a.step.city city.page-title role-b.step.city
#: role-a.profile.contact-details.city.label
#: role-b.profile.contact-details.city.label
msgid "City"
msgstr ""

Note how in the 'City' conversion, the profile.contact-details keys have been separated to their own lines as comments. This looks like the correct behavior if I'm looking at the conversion code

i18next-gettext-converter/src/lib/json2gettext.js

Line 209 in c39567e

].comments.reference.join('\n');

It should be splitting the keys to their own lines as comments, but for some reason some of those keys get merged into the same comment, separated by a space. I don't see where this is happening in the file there. Any idea? Something to do with the JSON flattening already here

i18next-gettext-converter/src/lib/json2gettext.js

Line 30 in c39567e

const bflat = flatten(JSON.parse(options.base), options);

?

[oasalonen]

Nevermind, this was actually caused by a msgcat that I ran as part of the conversion process. It merged some gettext comments which resulted in these problems. Sorry for the confusion!

[oasalonen]

Just a quick update: I decided to dump gettext completely and use Fluent with Pontoon. Unfortunately there doesn't seem to be a stable project to convert i18next JSON to Fluent. The only one I found was i18next-to-fluent and its author discourages its use in production. However, I'm using it successfully by being careful to use a subset of i18next and Fluent features.

I'm hoping to see a proper converter some day so I my automated i18n pipelines work properly. Right now the pipeline is .js code with i18next-react -> i18next-scanner -> i18next-to-fluent -> Pontoon -> .ftl loaded with i18next-fluent.

I also investigated the possibility of dropping i18next in favor of Fluent in the codebase to avoid conversions, but i18next tooling and features seem to be much better at the moment.

[jamuhl]

Ah I see...while there is i18next-fluent - there is no real extraction tooling to get that saved as .ftb files...

Hm...guess that is still a good selling point for our locize.com - which works out of the box using the runtime submission of missing keys.