microsoft / inmt Goto Github PK

Currently, we use a non-versioned copy of OpenNMT which becomes difficult to update when there is an update upstream. Figure out a way to update OpenNMT while keeping our wrappers intact.

Add the lints to Azure Pipelines.

Where can I get the parallel data to train this on?

Currently, the results from the API is in this form:

{
  "result": "Today 's weather is beautiful\nToday is \nToday 's \nThis day \nThe weather \nToday , ",
  "attn": [
    1,
    1,
    1,
    1
  ],
  "partial": "",
  "ppl": 3.230648083472593,
  "avg": -1.172682762145996
}

While this works for our case, it would be great to restructure it:

• "result" can be two lists - "full_sentence": [f1] and "part_sentence: [p1, p2, p3, ...]. The length of each list is controlled by #22
• "attn" can be restructured so that it makes sense independently. Maybe add the source sentence as well in the output?
• "ppl" and "avg" can be moved into a "metrics" section as both represent a way to measure the output.

There are some issues with the text pointer interactions. The common ones observed are:

• If a translation box is left blank, the pointer behavior on preview tab is buggy.
• After clicking the preview and coming back, the pointer behavior is buggy.
• Reverse pointer does not work in the preview tab.

Currently, the translation engine configurations are ingrained in the code, decouple the configurations to a separate json file and include in the .gitignore file.

the page show that:
This link has been disabled.
Sorry, access through this link has been removed by admin policy. Please contact the person who shared it with you.

Currently, the maximum number of suggestions in the drop-down box is 5. Can we add that as a parameter to the API?

Look at the question of the last person, do not know who to find the new model download address.

Currently, recording of keystrokes get replaced every time the translator opens the translation interface.

TODOs:

• Add mechanism to append keystrokes rather than replacing them.
• Record interaction keystrokes - probably use xpath to record these.

this secret needs to be removed and rotated ASAP.

This includes:

• Source Side BPE tokens - This includes applying BPE code while translating.
• Target Side BPE tokens - This includes joining of suggestions.

Currently, every time a translation is needed, the translation API is called and the request is processed for that source and partial input. Implementing Translation Memory would mean that we can use translation suggestions which were previously done by the translators.

Should the TM be user specific or global? It can probably be a combination of both - maybe batch and compare the individual TM and add that to global if it is commonly occurring.

Should the TM suggestion be shown with a different color explicitly to specify it comes from the TM whereas the others come from NMT?

Currently, we are performing beam search based on the inputs at the end. However, there should be a method to change inputs in between, so that the suggestion can be made based on the prefix and suffix.

Relevant Literature:

1. Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation
2. Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search
3. Guided Open Vocabulary Image Captioning with Constrained Beam Search

Currently, the transliteration helping mechanism takes the english input and then retrieves the appropriate transliteration and shows to the user. However, there are interaction flaws with this:

• Showing up of English characters in the translation box, which overlaps with the hindi suggestion.

• Issues for users of Language Keyboards as even with that input, the transliterator calls the API. This can be fixed by limiting call to transliteration API only if the input unicode lies in the range of English (Latin) script.