tc39 / proposal-intl-localematcher

Proposal for Intl.LocaleMatcher

Intl.LocaleMatcher

Motivation

i18n-supported websites often get a list of preferred locales via Accept-Language header or navigator.languages. They then try to determine the best available locale based on the set of locales that they support (and have translations for).

This operation currently exists within ECMA-402 but is only available as an abstract operation. Surfacing this functionality as a top level API would improve locale negotiation correctness and developer productivity as sites will be able to reliably handle not only matching, but also aliases, fallbacks and such.

Use cases

1. Given a set of locales an application has translations for and the set of locales a user requests, find the best matching locales.
2. JS runtimes (& polyfills) are not required to guarantee supporting all locales. Given a set of locales it supports and what the user requests, find the best matching locales.
3. An application can also provide different "tones" of the same locales (e.g casual, formal), utilizing -x- private tag. Given a set of locales with extensions and what the user preference might be, find the best matching locales.

Status

Stage 1

Ponyfill: https://formatjs.io/docs/polyfills/intl-localematcher

Champion

• Long Ho (@longlho)

API

interface Options {
    algorithm: 'lookup' | 'best fit'
}

Intl.LocaleMatcher.match(
    requestedLocales: string[],
    availableLocales: string[],
    defaultLocale: string,
    options?: Options
): string

Options

1. lookup would continue to be the existing LookupMatcher implementation within ECMA-402.
2. best fit would be implementation-dependent.

Examples

Intl.LocaleMatcher.match(["fr-XX", "en"], ["fr", "en"], "en"); // 'fr'

Prior Arts

@hapi/accept

This is the core of hapijs header parsing with quality preferences. This however does a naive hierarchy with exact matches only. For example:

Accept.language("en;q=0.7, fr-XX;q=0.8", ["fr", "en"]); // language === "en"

which would not be accurate.

koa

Similarly, Koa's request.acceptsLanguages follow similar exact match algorithm.

UTS35 LanguageMatching

This details a more sophisticated locale negotiation algorithm that is more accurate than hapi/koa

RFC4647 Section 3.4

This is the lookup algorithm in ECMA-402.

cldrjs's lookup implementation

Similar to UTS35 LanguageMatching.

References

• tc39/ecma402#513
• tc39/ecma402#46
• vercel/next.js#18676

proposal-intl-localematcher's Issues

return type of locale negotiation

cc @zbraniecki

In addition to returning an array of matched locales, I was wondering if we can actually return Intl.Locale[] instead. The benefit being that this solidify Intl.Locale as the data model representing Locale within ECMA402 and it also come already parsed with extensions builtin. Thoughts?

Move to tc39

This proposal was approved for Stage 1 on 2021-01-28. It should now be moved to the tc39 org. Please first transfer the repo to tc39-transfer, and then open an admin/business request for a chair to move it to tc39.

Extension support

An application can also provide different "tones" of the same locales (e.g casual, formal), utilizing -x- private tag. Given a set of locales with extensions and what the user preference might be, find the best matching locales.

Ref: formatjs/formatjs#2642

Matching multiple ordered candidates

It is occasionally useful to be able to test if multiple locales are good candidates according a user's preferences. For example:

• Configuring spellcheckers for multiple languages.
• Display a locale <select> with the user's preferences hoisted to the top.

I will say it's totally possible to do this with just this API, it's just slightly involved:

let requestedLocales = ["fa-FR", "fr-FR"]
let availableLocales = ["en-US", "fr-FR", "fa-IR", "zh-Hans-CN"]
let defaultLocale = "en-US"
let options = { algorithm: "best fit" }
let fakeDefaultLocale = "en-x-ignore";

let matchedLocales = []

requestedLocales.forEach((requestedLocale) => {
  let match = Intl.LocaleMatcher.match(
    [requestedLocale], 
    availableLocales, 
    fakeDefaultLocale,
    options
  )
  if (match !== fakeDefaultLocale) {
    matchedLocales.push(match)
  }
})

if (matchedLocales.length === 0) {
  matchedLocales.push(defaultLocale)
}

// matchedLocales == ["fa-IR", "fr-FR"]

However, it seems like an easy thing to just bake into this API to make it more useful:

let matchedLocales = Intl.LocaleMatcher.matchMany(
  requestedLocales, 
  availableLocales, 
  defaultLocale,
  options
)
// matchedLocales == ["fa-IR", "fr-FR"]

Add the API as a Intl.prototype function instead.

I suggest we add this functionality as a function in Intl instead of adding a new LocaleMatcher object.

Instead of

Intl.LocaleMatcher.match(["fr-XX", "en"], ["fr", "en"], "en"); // 'fr'

How about

Intl.matchLocale(["fr-XX", "en"], ["fr", "en"], "en"); // 'fr'

Add boilerplate from template-for-proposals

I should have mentioned that proposal repos should be based on the following template:

https://github.com/tc39/template-for-proposals

You can add this retroactively by merging the main branch from that repo into the main branch of this one.

What use cases is this intended to support?

A relevant observation about how users don't really have locale preference rankings, at least not in many cases: https://twitter.com/hsivonen/status/1301456494534234117

The user’s preference order of languages is the big fallacy of language negotiation in software. In reality, the top-preferred languages don’t have a fixed order but form an accept set and the user wants the least translated version within the accept set.

The ECMA-402 algorithms are not necessarily well-adapted to support that.