CanonicalizeLanguageTag should remove duplicate attributes/keywords in a Unicode extension, consistent with Intl.Locale about proposal-intl-locale

[jswalden]

I can't find anything in TR35 or RFC 6067 that describes the removal of duplicate attributes/keywords. So it doesn't look like there's anything in TR35 we could invoke to perform this operation, and it has to be hand-rolled -- as this patch does for CanonicalizeLanguageTag, and as Intl.Locale in this specification already does. :-|

[jswalden]

@anba You should probably take a look at and comment on this, seeing as you understand this stuff better than everyone else here. :-)

[aphillips]

@jswalden 6067 says something about this:

Only the first occurrence of an attribute or key conveys meaning in a
language tag. When interpreting tags containing the Unicode locale
extension, duplicate attributes or keywords are ignored in the
following way: ignore any attribute that has already appeared in the
tag and ignore any keyword whose key has already occurred in the tag.

This was intended to allow duplicates to be removed without effect (although I don't think we went on to encourage/permit it).

[jswalden]

@aphillips Yeah, the semantics of a tag with duplicates in it are clear enough even if they aren't removed. Just seems to me if we remove them one place -- and we don't really have a choice about it in Intl.Locale unless we wanted to just insert out-of-band options stringwise at the start of any existing Unicode locale extension, but that seems batty -- we should be consistent and do it everywhere.

[aphillips]

The canonical form is in a specific order. I would just go ahead and remove duplicates - - - extension tags are already long and unwieldy even without useless cruft in them.

[zbraniecki]

#83 fixes this.