Korean doesn't seem to work about scriptshifter

[scossu]

Closing this as it's related to a Korean issue that no longer exists. #72 should cover the table parsing issue.

[olovy]

Hi,

The problem seems to be these lines containing invalid unicode escape sequences (e.g. \u030)

17104    - "\uD5C8\uB4DD\uB7C9": "Ho\u0306 Tu\u030ng-nyang_"
33923    - "\uAC1C\uBCF": "kaebok_"
36281    - "Ho\u0306 Tu\u030ng-nyang": "\uD5C8\uB4DD\uB7C9_"
63564    - "kaebok": "\uAC1C\uBCF_"

In Libris we are are investigating using using your tables converted [1] to ICU transform rules [2] for automatic romanization.
In addition to to our Libris-specific tables [3].
[1] https://github.com/libris/librisxl/tree/develop/whelk-core/src/main/resources/romanizer/loc
[2] https://unicode-org.github.io/icu/userguide/transforms/general/rules.html
[3] https://github.com/libris/librisxl/tree/develop/whelk-core/src/main/resources/romanizer

[scossu]

I see that the \u030 in Tu\u030ng is not a valid Unicode sequence. I defer to a Korean specialist as to which symbol is meant to be placed there.

Note that Korean transliteration & transcription will be completely overhauled in Phase 2 (current), and the current table won't be used. However, if the current table is of any use for the time being, it may be worth fixing it.

[scossu]

This highlights a separate problem: each table is only read (and validated) when transliteration is requested for the first time in the application life cycle for that particular table. This is done to reduce application startup time (especially with the huge Korean table), however it also hides problems like this one, when a table contains encoding errors, which are only caught when transliteration is requested.

I think the best way to resolve this is to have automated tests built into a CI pipeline rather than forcing a read of all tables on startup.