Why Big5 index contains unmappable characters? about encoding

[hsivonen]

The Big5 encoder and decoder are asymmetric (like the EUC-JP encoder and decoder). The visualizations visualize what can be decoded. The spec excludes part of the decoding space from round-tripping via the encoder in order for HTML form submission not to generate extension-range bytes that some server-side recipients may not support.

For EUC-JP, the asymmetry is based on historical experience. For Big5, it is by prudent analogy of the problem initially seen with EUC-JP. Also, for Big5, the exclusion for Big5 is questionable and possibly by accident excluding less than what was intended: The encoder only excludes the extension part below the original Big5 range but doesn't exclude the other extension part above the original Big5 range.

[Mingun]

Well, probably that information should be included somewhere in the spec, probably here

encoding/encoding.bs

Lines 959 to 961 in 4f549cd

	<p class="note no-backref">All <a lt=index>indexes</a> are also available as a non-normative
	<a href=indexes.json>indexes.json</a> resource. (<a>Index gb18030 ranges</a> has a slightly
	different format here, to be able to represent ranges.)

because it was a little surprising when I used indexes.json for my own goals