ISO 2022-jp encoding/decoding support about encoding

[r12a]

The figure for iso2022jp-encode-href-errors-misc ought to be 79.

[r12a]

List of bugs raised:

• https://bugs.webkit.org/show_bug.cgi?id=159887
• https://bugzilla.mozilla.org/show_bug.cgi?id=1285398
• https://bugs.chromium.org/p/chromium/issues/detail?id=626399
• https://developer.microsoft.com/en-us/microsoft-edge/platform/issues/8202212/

[jungshik]

Chromium:

1. form/href-encoding-misc:
   Out of 93, 30 characters(Cf, default ignorable) share the same cause as #58, #59, #61, #62.
   The rest seems to be half-width Katakana. I remember raising an issue with this somewhere (and taking action), but I couldn't find it.
2. form/href-encoding: 373 characters. Mostly CJK Ideographs. Chromium treats them as not covered by ISO-2022-JP. Need to investigate. ISO-2022-JP in ICU (used by Chrome) share the table with Shift_JIS (and Shift_JIS in chromium passes the tests).

[r12a]

@jungshik is this the bug report about half-width katakana that you were looking for?

https://bugs.chromium.org/p/chromium/issues/detail?id=544402&thanks=544402&ts=1445064020

[jungshik]

@r12a, No, that's not what I had in mind. I remember doing something - at least filing a bug - to take a look at what you reported here; Half-width Katana in ISO-2022-JP, but I couldn't find it.

If you have the same issue in UTF-8 at http://r12a.github.io/uniview/?block=halfwidth_and_fullwidth_forms (which I couldn't reproduce on my Mac Chrome in the past. I didn't try it today) , it cannot be related to the encoding conversion.

[hsivonen]

Two decode expectations for malformed sequences seem wrong:
Fail escape start: 1B 65 79 56 1B 28 42 assert_equals: expected "�eByV" but got "�eyV"
Fail escape: 1B 24 65 79 56 1B 28 42 assert_equals: expected "�e$yV" but got "�$eyV"

[vyv03354]

Firefox Nightly 56 got much improvement, but still encoding errors has 63 failures and decoding errors has 2 failures.

[r12a]

Today and yesterday i updated the results at https://www.w3.org/International/tests/repo/results/encoding-dbl-byte.en#iso2022jp for Firefox, FNightly, Chrome, and Canary. The latest summary is:

[hsivonen]

Firefox Nightly 56 got much improvement, but still encoding errors has 63 failures and decoding errors has 2 failures.

Per earlier comment, the decoder error handling failures are test suite bugs.

The encoder failures are due to the test suite not having been updated to account for the spec change to half-width katakana handling.

[r12a]

The encoder failures are due to the test suite not having been updated to account for the spec change to half-width katakana handling.

@hsivonen i updated the encoder algorithm used by the tests. I haven't updated the results page yet, but you can run the tests from https://www.w3.org/International/tests/repo/results/encoding-dbl-byte.en#iso2022jp (click on the title in the left column).

The result of that fix is that Chrome and Safari now pass iso2022jp-encode-form-errors-misc.html cleanly. Firefox however still sticks on 8 characters (not katakana), so i'm guessing that may be a FF bug(?)

[r12a]

Two decode expectations for malformed sequences seem wrong:
Fail escape start: 1B 65 79 56 1B 28 42 assert_equals: expected "�eByV" but got "�eyV"
Fail escape: 1B 24 65 79 56 1B 28 42 assert_equals: expected "�e$yV" but got "�$eyV"

@hsivonen @annevk i fixed the first, which was indeed a bug. (Wrong expectations.)
however the second test you mention above still fails in FF, Chrome and Safari.

I suspect it may be a bug in the decoder algorithm at https://encoding.spec.whatwg.org/#iso-2022-jp-decoder

step escape.8 says:

Prepend lead and byte to stream.

To get what the browsers are actually returning, i think it needs to say
"Prepend byte and lead to stream"

Or perhaps better, specify explicitly the order in which those two should end up when prepended.

[r12a]

I should mention that i tested those in FF 55.

[annevk]

What it means is that they're to be prepended together in the order specified as specified at https://encoding.spec.whatwg.org/#concept-stream-prepend. I'm not sure how you can read it any other way.

[annevk]

It would seem weird to prepend lead and then prepend byte (aka prepend byte and lead) as that would have them be in the reverse order when you read that stream.

[hsivonen]

encoding_rs went into Firefox 56, so testing 56 is more useful than testing 55.

I see one failure (escape: 1B 24 65 79 56 1B 28 42 | assert_equals: expected "�e$yV" but got "�$eyV") in Firefox Nightly. This is clearly a test case bug with the test case having e and $ reversed compared to the ASCII interpretation of the input bytes.

[r12a]

What it means is that they're to be prepended together in the order specified as specified at https://encoding.spec.whatwg.org/#concept-stream-prepend. I'm not sure how you can read it any other way.

I read

those tokens must be inserted, in given order

as "insert the first one, and then insert the second one". Depends whether 'in given order' refers to 'insert' or 'the tokens'. I read it as "prepend lead, then byte to stream". (I'll admit that i was following the wording rather than the deep logic of what was going on.)

Something like you just said may be clearer, eg. Prepend lead and byte together to stream.

Anyway, i'll fix it.

[annevk]

Now that Firefox passes all these tests and a year has passed, I'm happy to consider this done. A new issue would also be less noisy at this point, were one warranted.

If you want to pursue changing the wording here I'd be open to that by the way, but let's discuss that in a new issue.

[jungshik]

2. form/href-encoding: 373 characters. Mostly CJK Ideographs. Chromium treats them as not covered by ISO-2022-JP. Need t

It's now fixed in Chromium's ToT. It'll be included in next canary and Chrome 72 (will turn stable in January. dev/beta before that). https://crbug.com/901255 .