elixir-cldr / cldr

Elixir implementation of CLDR/ICU

Getting Started with Cldr

Change to :json_library configuration {: .warning}

As of ex_cldr version 2.37.2 the configuration parameter :json_library does not attempt to read the configuration of either Phoenix of Ecto.

Specifing the :json_library parameter under the :ex_cldr configuration key in config.exs is recommended however when executing on OTP versions below OTP 27 but the availability of Jason or Poison will still be automatically detected and configured if the :json_library key is not set.

Introduction

ex_cldr is an Elixir library for the Unicode Consortium's Common Locale Data Repository (CLDR). The intentions of CLDR, and this library, is to simplify the locale specific formatting and parsing of numbers, lists, currencies, calendars, units of measure and dates/times. As of April 28th 2023 and ex_cldr Version 2.37.0, ex_cldr is based upon CLDR version 43.0.

The first step is to define a module that will host the desired ex_cldr configuration and the functions that serve as the public API. This module is referred to in this documentation as a backend module. For example:

defmodule MyApp.Cldr do
  @moduledoc """
  Define a backend module that will host our
  Cldr configuration and public API.

  Most function calls in Cldr will be calls
  to functions on this module.
  """
  use Cldr,
    locales: ["en", "fr", "zh", "th"],
    default_locale: "en"

end

This strategy means that different configurations can be defined and it also means that one ex_cldr implementation won't interfere with implementations in other, potentially dependent, applications.

The functions you are mostly likely to use are:

• MyApp.Cldr.default_locale/0
• MyApp.Cldr.put_locale/1
• MyApp.Cldr.get_locale/0
• MyApp.Cldr.known_locale_names/0
• MyApp.Cldr.Locale.new/1
• MyApp.Cldr.validate_locale/1

Use Case

Use this library if you need to:

• Support multiple languages and locales in your application

• Support formatting numbers, dates, times, date-times, units and lists in one language or many

• Need to access the data maintained in the CLDR repository in a functional manner

• Parse an Accept-Language http header or a language tag

It is highly likely that you will also want to install one or more of the dependent packages that provide localization and formatting for a particular data domain. See Additional Cldr Packages below.

Elixir Version Requirements

• ex_cldr requires Elixir 1.12 or later.

Installation

Add ex_cldr and the JSON library of your choice as a dependencies to your mix project. If running on OTP 27 or later, this step is not required since ex_cldr will detect and use the built-in :json module.

defp deps do
  [
    {:ex_cldr, "~> 2.37"},
    # Poison or any other compatible json library
    # that implements `encode!/1` and `decode!/1`
    # :jason is recommended
    {:jason, "~> 1.0"}
    # {:poison, "~> 2.1 or ~> 3.0"}
  ]
end

then retrieve ex_cldr and the JSON library from hex:

mix deps.get
mix deps.compile

Additional Cldr Packages

ex_cldr includes only basic functions to maintain the CLDR data repository in an accessible manner and to manage locale definitions. Additional functionality is available by adding additional packages:

• Plugs for setting the locale from an HTTP request: ex_cldr_plugs
• Number formatting: ex_cldr_numbers
• List formatting: ex_cldr_lists
• Unit or measure formatting: ex_cldr_units
• Date/Time/DateTime formatting: ex_cldr_dates_times
• Route localization for Phoenix: ex_cldr_routes
• Locale name localisation: ex_cldr_locale_display
• HTML select helpers: ex_cldr_html
• Calendars: ex_cldr_calendars
• Calendar formatting: ex_cldr_calendars_format
• Printf-like formatting: ex_cldr_print
• Collation: ex_cldr_collation
• ICU Message formatting: ex_cldr_messages
• Person name formatting: ex_cldr_person_names
• Territories localization and information: ex_cldr_territories by @Schultzer
• Languages localization: ex_cldr_languages by @lostkobrakai

Each of these packages includes ex_cldr as a dependency so configuring any of these additional packages will automatically install ex_cldr.

Configuration

ex_cldr attempts to maximise runtime performance at the expense of additional compile time. Where possible ex_cldr will create functions to encapsulate data at compile time. To perform these optimizations for all 541 locales known to Cldr wouldn't be an effective use of your time or your computer's. Therefore ex_cldr requires that you configure the locales you want to use.

ex_cldr is configured in your backend module. This removes any dependency on your mix.exs and therefore simplifies deployment as a release.

Backend Module Configuration

use Cldr {: .info}

When you use Cldr, a number of functions are generated that encapsulate CLDR data. A module that invokes use Cldr is referred to as a Cldr backend module.

The functions in a Cldr backend module form the primary recommended API for ex_cldr.

In additional, a number of additional modules may be generated with names that are prefixed by the name of the module in which use Cldr is invoked. The number and names of these additional modules is determined by the modules configured under the :providers option to use Cldr.

It is not recommended that a module that invokes use Cldr define any other functions.

defmodule MyApp.Cldr do
  use Cldr,
    default_locale: "en",
    locales: ["fr", "en", "bs", "si", "ak", "th"],
    add_fallback_locales: false,
    gettext: MyApp.Gettext,
    data_dir: "./priv/cldr",
    otp_app: :my_app,
    precompile_number_formats: ["¤¤#,##0.##"],
    precompile_transliterations: [{:latn, :arab}, {:thai, :latn}],
    providers: [Cldr.Number],
    generate_docs: true,
    force_locale_download: false
end

Otp App Configuration

In the backend configuration example above the :otp_app key has been defined. This means that ex_cldr will look for additional configuration, defined under the key :my_app with the sub-key MyApp.Cldr. For example:

# cldr.ex
defmodule MyApp.Cldr do
  use Cldr,
    otp_app: :my_app,
    default_locale: "en",
    gettext: MyApp.Gettext,
    json_library: Jason,
    data_dir: "./priv/cldr",
    precompile_number_formats: ["¤¤#,##0.##"],
    providers: [Cldr.Number]
end

# config/config.exs
config :my_app, MyApp.Cldr,
  # a single locale, for fast compilation in dev / test
  locales: ["en"]

# config/production.exs
config :my_app, MyApp.Cldr,
  # these will take a while to compile
  locales: ["fr", "en", "bs", "si", "ak", "th"],
  precompile_transliterations: [{:latn, :arab}, {:thai, :latn}]

Multiple backends can be configured under a single :otp_app if required.

Global configuration.

In config.exs a global configuration can be defined under the :ex_cldr key. Although any valid configuration keys can be used here, only the keys :json_library, :default_locale, :default_backend, :cacertfile, :data_dir, :force_locale_download are considered valid. Other configuration keys may be used to aid migration from ex_cldr version 1.x but a deprecation message will be printed during compilation. Here's an example of global configuration:

config :ex_cldr,
  default_locale: "en",
  default_backend: MyApp.Cldr,
  json_library: Jason,
  cacertfile: "path/to/cacertfile"

Note that the :json_library key can only be defined at the global level since it is required during compilation before any backend module is compiled.

On most platforms other than Windows the :cacertfile will be automatically detected. Any configured :cacertfile will take precedence on all platforms.

If configuration beyond the keys :default_locale, :cacertfile or :json_library are defined a deprecation warning is printed at compile time noting that configuration should be moved to a backend module.

Configuration Priority

When building the consolidated configuration the following priority applies:

• Consider the global configuration
• Merge the otp_app configuration over the top of the global configuration
• Merge the backend module configuration over the top

Backend Configuration Keys

The configuration keys available for ex_cldr are:

• default_locale specifies the default locale to be used for this backend. The default locale in case no other locale has been set is "en-001". The default locale calculated as follows:

  • If set by the :default_locale key, then this is the priority
  • If no :default_locale key, then a configured Gettext default locale for this backend is chosen
  • If no :default_locale key is specified and no Gettext module is configured, or is configured but has no default set, use Cldr.default_locale/0 which returns either the default locale configurated in mix.exs under the ex_cldr key or then the system default locale will is currently en-001

• locales: Defines what locales will be configured in ex_cldr. Only these locales will be available and an exception Cldr.UnknownLocaleError will be raised if there is an attempt to use an unknown locale. This is the same behaviour as Gettext. Locales are configured as a list of binaries (strings). For convenience it is possible to use wildcard matching of locales which is particulalry helpful when there are many regional variances of a single language locale. For example, there are over 100 regional variants of the "en" locale in CLDR. A wildcard locale is detected by the presence of ., [, * and + in the locale string. This locale is then matched using the pattern as a regex to match against all available locales. The example below will configure all locales that start with en- and the locale fr.

use Cldr,
  default_locale: "en",
  locales: ["en-*", "fr"]

• There is one additional setting which is :all which will configure all 541 locales. This is highly discouraged since it will take many minutes to compile your project and will consume more memory than you really want. This setting is there to aid in running the test suite. Really, don't use this setting.

• :add_fallback_locales is a boolean key which when true results in the fallback locales being added for each of the configured locales. The default is false. The reason to set this option to true is that some data such as rules based number formats and subdivision data are inherited from their language roots. For example, the locale en-001 is inherited from the locale en. Locale en-001 does not have any rules based number formats or subdivision data defined for it. However locale en does. Including the fallback locales maximises the opportunity to resolve localised data.

• :gettext: specifies the name of a Gettext module that informs ex_cldr to use that module as an additional source of locales you want to configure. Since Gettext uses the Posix locale name format (locales with an '_' in them) and ex_cldr uses the Unicode format (a '-' as the subtag separator), ex_cldr will transliterate locale names from Gettext into the ex_cldr canonical form. For example:

use Cldr,
  default_locale: "en",
  gettext: MyApp.Gettext,
  locales: ["en-*", "fr"]

• :data_dir: indicates where downloaded locale files will be stored. The default is :code.priv_dir(otp_app) where otp_app is the app defined under the :otp_app configuration key. If that key is not specified then the :ex_cldr app is used. It is recommended that an :otp_app key is specified in your backend module configuration.

• :precompile_number_formats: provides a means to have user-defined format strings precompiled at application compile time. This has a performance benefit since precompiled formats execute approximately twice as fast as formats that are not precompiled.

• :precompile_transliterations: defines those transliterations between the digits of two different number systems that will be precompiled. The is a list of 2-tuples where each tuple is of the form {from_number_system, to_number_system} where each number system is expressed as an atom. The available number systems is returned by Cldr.Number.System.systems_with_digits/0. The default is the empty list [].

• :precompile_date_time_formats: provides a means to have user-defined date, time and date time format strings precompiled at application compile time. This has a performance benefit since precompiled formats execute approximately twice as fast as formats that are not precompiled. These formats are used by ex_cldr_date_times.

• :precompile_interval_formats: provides a means to have user-defined interval format strings precompiled at application compile time. This has a performance benefit since precompiled formats execute approximately twice as fast as formats that are not precompiled. These formats are used by ex_cldr_date_times.

• :default_currency_format determines whether Cldr.Number.to_string/2 will use :currency or :accounting if no format is specified but a currency is. The default is nil which means that the format will be derived from the locale.

• :providers: a list of modules that provide ex_cldr functionality to be compiled into the backend module. See the providers section below.

• :generate_docs defines whether or not to generate documentation for the modules built as part of the backend. Since these modules represent the public API for ex_cldr, the default is true. Setting this key to false (the atom false, not a falsy value) will prevent the generation of docs for this backend.

• :suppress_warnings defines whether warnings are logged when a provider module is configured but not available. It also controls whether warnings are logged when a number format is compiled at runtime. Its purpose is to help identify those formats which might best be added to the :precompile_number_formats configuration. The default is false. Warning are not logged when set to true.

• :force_locale_download determines whether to always download locale files during compilation. Locale data is ex_cldr version dependent. When a new version of ex_cldr is installed, no locales are installed and therefore locales are downloaded at compilation time as required. This ensures that the right version of the locale data is always associated with the right version of ex_cldr. However if locale data is being cached in CI/CD there is some possibility that there can be a version mismatch. Since reproducible builds are important, setting the force_locale_download: true in a backend or in global configuration adds additional certainty. The default setting is false thereby retaining compatibility with existing behaviour. The configuration can also be made dependent on mix environment as shown in this example:

defmodule MyApp.Cldr do
  use Cldr,
    locales: ["en", "fr"],
    default_locale: "en",
    force_locale_download: Mix.env() == :prod
end

• :https_proxy is the URL of a proxy host that will be used when downloading locales to be installed. When downloading, this configuration key has priority, followed by the environment variables HTTPS_PROXY and https_proxy. The default is nil.

Providers

The data maintained by CLDR is quite large and not all capabilities are required by all applications. Hence ex_cldr has additional optional functionality that can be provided through additional hex packages. In order to support compile-time additions to a configured backend, any package can define a provider that will be called at compile time.

The currently known providers and their hex package names are:

Hex Package	Provider Module	Comment
ex_cldr_numbers	Cldr.Number	Formatting of numbers, currencies
ex_cldr_lists	Cldr.List	Formatting of lists
ex_cldr_units	Cldr.Unit	Formatting of SI and Imperial units
ex_cldr_person_names	Cldr.PersonName	Formatting person names
ex_cldr_currencies	Cldr.Currency	Currency definitions and localizations
ex_cldr_territories	Cldr.Territory	Formatting of territory (country) data
ex_cldr_languages	Cldr.Language	Formatting of language information
ex_cldr_dates_times	Cldr.DateTime	Formatting of dates, times & datetimes
ex_cldr_locale_display	Cldr.LocaleDisplay	Localising locale names
ex_cldr_routes	Cldr.Route	Localized routes and route helpers
ex_cldr_messages	Cldr.Message	Formatting of ICU-formatted messages
ex_money	Money	Operations on and formatting of a money type

Any library author can create a provider module by exposing a function called cldr_backend_provider/1 that takes a Cldr.Config struct as a single parameter. The function should return an AST that is inserted into the backend module being compiled.

Providers are configured on each backend module under the :providers key. It must be a list of provider modules. For example:

defmodule MyApp.Cldr do
  use Cldr,
    locales: ["en", "zh"],
    default_locale: "en",
    providers: [Cldr.Number, Cldr.List]
end

If :providers is nil (the default), ex_cldr will attempt to configure all of the providers described above if they have been installed as deps. If you don't wish to invoke any providers, use the empty list [].

Migrating from Cldr 1.x

1. Create a backend module by following the configuration instructions
2. Delete any duplicated global configuration in any config.exs files. Only the keys :default_locale and :json_library are supported in the global configuration
3. Update any plugs to configure the desired backend
4. Adjust any API calls from Cldr.some_function to MyApp.Cldr.some_function. Or better still, alias your backend module where required. ie. alias MyApp.Cldr, as: Cldr

Downloading Locales

ex_cldr can be installed from either github or from hex.

• If installed from github then all 571 locales are installed when the repo is cloned into your application deps.

• If installed from hex then only the locales "en", "en-001" and "und" are installed. When you configure additional locales these will be downloaded during application compilation.

Localizing Numbers

The Cldr.Number module implemented in the ex_cldr_numbers package provides number formatting. The public API for number formatting is MyApp.Cldr.Number.to_string/2. Some examples:

iex> MyApp.Cldr.Number.to_string 12345
"12,345"

iex> MyApp.Cldr.Number.to_string 12345, locale: "fr"
"12 345"

iex> MyApp.Cldr.Number.to_string 12345, locale: "fr", currency: "USD"
"12 345,00 $US"

iex> MyApp.Cldr.Number.to_string 12345, format: "#E0"
"1.2345E4"

iex(> MyApp.Cldr.Number.to_string 1234, format: :roman
"MCCXXXIV"

iex> MyApp.Cldr.Number.to_string 1234, format: :ordinal
"1,234th"

iex> MyApp.Cldr.Number.to_string 1234, format: :spellout
"one thousand two hundred thirty-four"

See h MyApp.Cldr.Number and h MyApp.Cldr.Number.to_string in iex for further information.

Localizing Lists

The Cldr.List module provides list formatting and is implemented in the ex_cldr_lists package. The public API for list formatting is Cldr.List.to_string/2. Some examples:

iex> MyApp.Cldr.List.to_string(["a", "b", "c"], locale: "en")
"a, b, and c"

iex> MyApp.Cldr.List.to_string(["a", "b", "c"], locale: "en", format: :unit_narrow)
"a b c"

iex> MyApp.Cldr.List.to_string(["a", "b", "c"], locale: "fr")
"a, b et c"

See h MyApp.Cldr.List and h MyApp.Cldr.List.to_string in iex for further information.

Localizing Units

The Cldr.Unit module provides unit localization and is implemented in the ex_cldr_units package. The public API for unit localization is Cldr.Unit.to_string/3. Some examples:

iex> MyApp.Cldr.Unit.to_string 123, :gallon
"123 gallons"

iex> MyApp.Cldr.Unit.to_string 1234, :gallon, format: :long
"1 thousand gallons"

iex> MyApp.Cldr.Unit.to_string 1234, :gallon, format: :short
"1K gallons"

iex> MyApp.Cldr.Unit.to_string 1234, :megahertz
"1,234 megahertz"

iex> MyApp.Cldr.Unit.available_units
[:acre, :acre_foot, :ampere, :arc_minute, :arc_second, :astronomical_unit, :bit,
 :bushel, :byte, :calorie, :carat, :celsius, :centiliter, :centimeter, :century,
 :cubic_centimeter, :cubic_foot, :cubic_inch, :cubic_kilometer, :cubic_meter,
 :cubic_mile, :cubic_yard, :cup, :cup_metric, :day, :deciliter, :decimeter,
 :degree, :fahrenheit, :fathom, :fluid_ounce, :foodcalorie, :foot, :furlong,
 :g_force, :gallon, :gallon_imperial, :generic, :gigabit, :gigabyte, :gigahertz,
 :gigawatt, :gram, :hectare, :hectoliter, :hectopascal, :hertz, :horsepower,
 :hour, :inch, ...]

See h MyApp.Cldr.Unit and h MyApp.Cldr.Unit.to_string in iex for further information.

Localizing Dates

Formatting of relative dates and date times is supported in the Cldr.DateTime.Relative module implemented in the ex_cldr_dates_times package. The public API is MyApp.Cldr.DateTime.to_string/2 and MyApp.Cldr.DateTime.Relative.to_string/2. Some examples:

iex> MyApp.Cldr.Date.to_string Date.utc_today()
{:ok, "Aug 18, 2017"}

iex> MyApp.Cldr.Time.to_string Time.utc_now
{:ok, "11:38:55 AM"}

iex> MyApp.Cldr.DateTime.to_string DateTime.utc_now
{:ok, "Aug 18, 2017, 11:39:08 AM"}

iex> MyApp.Cldr.DateTime.Relative.to_string 1, unit: :day, format: :narrow
{:ok, "tomorrow"}

iex> MyApp.Cldr.DateTime.Relative.to_string(1, unit: :day, locale: "fr")
"demain"

iex> MyApp.Cldr.DateTime.Relative.to_string(1, unit: :day, format: :narrow)
"tomorrow"

iex> MyApp.Cldr.DateTime.Relative.to_string(1234, unit: :year)
"in 1,234 years"

iex> MyApp.Cldr.DateTime.Relative.to_string(1234, unit: :year, locale: "fr")
"dans 1 234 ans"

Gettext Pluralization

gettext allows for user-defined plural forms modules to be configured for a gettext backend.

To define a plural forms module that uses CLDR plural rules create a new module and then use Cldr.Gettext.Plural. For example:

defmodule MyApp.Gettext.Plural do
  use Cldr.Gettext.Plural, cldr_backend: MyApp.Cldr
end

This module can then be used in the configuration of a gettext backend. For example:

defmodule MyApp.Gettext do
  use Gettext, plural_forms: MyApp.Gettext.Plural
end

Note that MyApp.Gettext.Plural does not guarantee to return the same plural index as Gettext's own pluralization engine which can introduce some compatibility issues if you plan to mix plural engines. See Cldr.Gettext.Plural for more information.

About Language Tags

Note that ex_cldr defines locale strings according to the IETF standard as defined in RFC5646. ex_cldr also implements the u extension as defined in RFC6067 and the t extension defined in RFC6497. This is also the standard used by W3C.

The IETF standard is slightly different to the ISO/IEC 15897 standard used by Posix-based systems; primarily in that ISO 15897 uses a "_" separator whereas IETF and W3C use "-".

Locale string are case insensitive but there are common conventions:

• Language codes are lower-cased
• Territory codes are upper-cased
• Script names are capital-cased
• All other subtags are lower-cased

Sigil_l

As of ex_cldr version 2.23.0, a sigil is available to simplify creating t:Cldr.LanguageTag structs. Usage is:

iex> import Cldr.LanguageTag.Sigil
Cldr.LanguageTag.Sigil

# Returns a locale that is valid and known to
# the default backend module
iex> ~l(en-US)
#Cldr.LanguageTag<en-US [validated]>

# Same, but specifying the backend module
# MyApp.Cldr specifically
iex> ~l(en-US|MyApp.Cldr)
#Cldr.LanguageTag<en-US [validated]>

# The `u` flag will parse and validate
# the language tag but it may not be known
# as a configured locale
iex> ~l(zh)u
#Cldr.LanguageTag<zh [canonical]>

# Language tags can convey a lot more information
# than might be initially expected!
iex> ~l(en-u-ca-ethiopic-cu-aud-sd-gbsct-t-d0-lower-k0-extended-m0-ungegn-x-ux)
#Cldr.LanguageTag<en-t-d0-lower-k0-extended-m0-ungegn-u-ca-ethiopic-cu-aud-sd-gbsct-x-ux [validated]>

Locale extensions

Unicode defines the U extension which support defining the requested treatment of CLDR data formats. For example, a locale name can configure the requested:

• calendar to be used for dates
• collation
• currency
• currency format
• number system
• first day of the week
• 12-hour or 24-hour time
• time zone
• and many other items

For example, the following locale name will request the use of the timezone Australia/Sydney, and request the use of accounting format when formatting currencies:

iex> MyApp.Cldr.validate_locale "en-AU-u-tz-ausyd-cf-account"
{:ok,
 %Cldr.LanguageTag{
   canonical_locale_name: "en-Latn-AU",
   cldr_locale_name: "en-AU",
   extensions: %{},
   gettext_locale_name: "en",
   language: "en",
   language_subtags: [],
   language_variants: nil,
   locale: %Cldr.LanguageTag.U{cf: :account, timezone: "Australia/Sydney"},
   private_use: [],
   rbnf_locale_name: "en",
   requested_locale_name: "en-AU",
   script: :Latn,
   territory: :AU,
   transform: %{}
 }}

The implementation of these extensions is governed by each library in the ex_cldr family. As of January 2020, ex_cldr_numbers version 2.10 implements the following U extension keys:

• cf (currency format)
• cu (currency)
• nu (number system)

Other libraries in the family will progressively implement other extension keys.

Notes

• A language code is an ISO-3166 language code.
• Potentially one or more modifiers separated by - (dash), not a _. (underscore). If you configure a Gettext module then ex_cldr will transliterate Gettext's _ into - for compatibility.
• Typically the modifier is a territory code. This is commonly a two-letter uppercase combination. For example pt-PT is the locale referring to Portuguese as used in Portugal.
• In ex_cldr a locale name is always a binary and never an atom. Internally a locale is parsed and stored as a t:Cldr.LanguageTag struct.
• The locales known to ex_cldr can be retrieved by Cldr.known_locale_names/1 to get the locales known to this configuration of ex_cldr and Cldr.all_locale_names/0 to get the locales available in the CLDR data repository.

Developing ex_cldr

See the file DEVELOPMENT.md in the github repository.

Testing

Tests cover the full ~700 locales defined in CLDR. Since ex_cldr attempts to maximize the work done at compile time in order to minimize runtime execution, the compilation phase for tests is several minutes.

Tests are run on Elixir 1.11 and later. ex_cldr is not supported on Elixir versions before 1.11.

cldr's Issues

Cldr.AcceptLanguage.Parse unexpected Sort Order

Problem

When parsing a list of languages with the same quality, the list is inverted.

One would expect the list to remain unsorted if the sort criteria is identical for all members of the list.

Why this is an issue

This is problematic when trying to determine locales through WebSockets. In this scenario, we would be sending as a param on connection, and it would most likely derive from navigator.languages. For most browsers, this provides the same as the accept-languages header, but without the quality.

Example

iex> Cldr.AcceptLanguage.tokenize("en-US,en,es-ES,es")
[{1.0, "en-us"}, {1.0, "en"}, {1.0, "es-es"}, {1.0, "es"}]

Note the order matches the accepted list.

Now if we parse the list:

iex(8)> Cldr.AcceptLanguage.parse("en-US,en,es-ES,es")
{:ok,
 [
   {1.0,
    %Cldr.LanguageTag{
      canonical_locale_name: "es-Latn-ES",
      cldr_locale_name: "es",
      extensions: %{},
      gettext_locale_name: nil,
      language: "es",
      language_subtags: [],
      language_variant: nil,
      locale: %{},
      private_use: [],
      rbnf_locale_name: "es",
      requested_locale_name: "es",
      script: "Latn",
      territory: "ES",
      transform: %{}
    }},
   {1.0,
    %Cldr.LanguageTag{
      canonical_locale_name: "es-Latn-ES",
      cldr_locale_name: "es",
      extensions: %{},
      gettext_locale_name: nil,
      language: "es",
      language_subtags: [],
      language_variant: nil,
      locale: %{},
      private_use: [],
      rbnf_locale_name: "es",
      requested_locale_name: "es-ES",
      script: "Latn",
      territory: "ES",
      transform: %{}
    }},
   {1.0,
    %Cldr.LanguageTag{
      canonical_locale_name: "en-Latn-US",
      cldr_locale_name: "en",
      extensions: %{},
      gettext_locale_name: nil,
      language: "en",
      language_subtags: [],
      language_variant: nil,
      locale: %{},
      private_use: [],
      rbnf_locale_name: "en",
      requested_locale_name: "en",
      script: "Latn",
      territory: "US",
      transform: %{}
    }},
   {1.0,
    %Cldr.LanguageTag{
      canonical_locale_name: "en-Latn-US",
      cldr_locale_name: "en",
      extensions: %{},
      gettext_locale_name: nil,
      language: "en",
      language_subtags: [],
      language_variant: nil,
      locale: %{},
      private_use: [],
      rbnf_locale_name: "en",
      requested_locale_name: "en-US",
      script: "Latn",
      territory: "US",
      transform: %{}
    }}
 ]}

The list is inverted as identified by requested_locale_name on the language tags.

module Poison is not available

When switching to use Jason as the JSON library, I get the following warning:

warning: function Poison.decode!/1 is undefined (module Poison is not available)
lib/cldr/config/rbnf_config.ex:106

Do I need to include both Posion and Jason as dependencies?

Flow is optional so compiler issues warnings if its not installed

The cldr.consolidate mix task consolidates multiple different json files into one consolidated locale file. This process uses Experimental.Flow but that declared as an option hex package and therefore not installed by default. Therefore the compiler issues a warning when compiling Cldr.Consolidate when Experimental.Flow is not installed.

`default_locale` issue

I went from 1.3.2 to master with the following config:

config :ex_cldr,
  default_locale: "en",
  locales: ["en"],
  gettext: MyApp.Gettext

But I got this error:

 ** (FunctionClauseError) no function clause matching in Keyword.get/3

     The following arguments were given to Keyword.get/3:
     
         # 1
         nil
     
         # 2
         :default_locale
     
         # 3
         nil
     
     Attempted function clauses (showing 1 out of 1):
     
         def get(keywords, key, default) when is_list(keywords) and is_atom(key)
     
     stacktrace:
       (elixir) lib/keyword.ex:179: Keyword.get/3
       (ex_cldr) lib/cldr/config/config.ex:245: Cldr.Config.gettext_locales/0
       (ex_cldr) lib/cldr.ex:369: Cldr.known_gettext_locale_name?/1
       (ex_cldr) lib/cldr.ex:448: Cldr.known_gettext_locale_name/1
       (ex_cldr) lib/cldr/locale.ex:328: Cldr.Locale.first_match/2
       (ex_cldr) lib/cldr/locale.ex:317: Cldr.Locale.gettext_locale_name/1
       (ex_cldr) lib/cldr/locale.ex:299: Cldr.Locale.set_gettext_locale_name/1
       (ex_cldr) lib/cldr/locale.ex:249: Cldr.Locale.canonical_language_tag/1
       (ex_cldr) lib/cldr.ex:536: Cldr.validate_locale/1

To fix it, I had to add the following:

config :my_app, MyApp.Gettext, default_locale: "en"

Is this expected behavior? If it is, then it should have a proper warning.

For nan-TW, script should be "Hant"

The locale "nan-tw", refers to the Taiwanese language (also known as Hokkien and Minnanyu). The script should be traditional Chinese ("Hant"), not simplified Chinese ("Hans").

Cldr.Plug.Locale - design thoughts

The last functional part to be developed before Cldr version 1.0 is a plug to parse the accept-language header and set the locale. I'm looking for feedback on the following.

There are at least 3 places in an HTTP request to look for a locale

• accept-language header
• a locale parameter like https://domain.com/content?locale=en
• a locale segment in an URL like https://domain.com/en/content

Should the plug look for a locale in

1. All three by default? If so, what precedence?
2. All/any of the three by configuration?
3. Only the accept-language header only?

The second option would require configuration. Something like:

pipeline :browser do
  plug Cldr.Plug.Locale, [:accept_language, :query_parameter]
  plug Cldr.Plug.Locale, [:query_parameter]
  plug Cldr,Plug.Locale  # What would the default be?
end

Thoughts much appreciated.

Gettext get_locale different from get_cldr_locale

I just added cldr as a dependency as it great to handle proper i18n, and after configuring I still cannot get it to work well with Gettext.

In config.exs, I put:

config :gettext,
  default_locale: "fr",
  locales: ~w(fr en es ko)

config :ex_cldr,
  default_locale: "fr", # necessary only for router "default: Cldr.default_locale,"
  gettext: TotoWeb.Gettext,
  json_library: Jason

In router.exs, I added the SetLocale plug:

    plug Cldr.Plug.SetLocale,
      apps: [:cldr, :gettext],
      from: [:path, :cookie, :accept_language],
      param: "locale",
      default: Cldr.default_locale,
      gettext: TotoWeb.Gettext

Per the doc, by adding the :gettext to that "apps" list, the plug should set the locale for gettext, but in my view Gettext.get_locale() remains the default "fr", whereas Cldr.Plug.AcceptLanguage.get_cldr_locale(@conn) is updated correctly.

Did I miss something? Do I have to set the locale manually for gettext after the setLocale plug?

Number.to_string doesn't correctly select number system for a locale when using :default

Number.to_string 123, locale: "ar" should use the :default number system for locale "ar" but it uses instead the default of the current locale.

GenStage/Flow in mix.exs

Can we limit the usage of genstage/flow to only a certain mix env or maybe make it runtime: false? From what I can tell it's not going to be used in compiled production code.

Scientific format doesn't honour forced positive sign on exponent

A format like #E+0 should force a localised + sign in the exponent. Negative exponents are fine.

Now:

iex> Number.to_string 12456, format: "#E+0"
"1.2456E4"

Should be:

iex(1)> Number.to_string 12456, format: "#E+0"
"1.2456E+4"

RBNF private rulesets are defined as public

RBNF defines access to a ruleset as either 'public' or 'private'. Cldr is defining them all as public which pollutes the public api.

Cldr.Config.default_locale/0 does not set :gettext_locale

Cldr.Config is responsible for creating the language tag for the default locale since most of Cldr is not available at the compile time of Cldr.Config. As a result Cldr.Config has code the duplicates parts of Locale.canonical_language_tag/1 but it's not setting :gettext_locale in the language tag map.

Download locale data only when required to reduce install package size

The CLDR repository in json form is over 600Mb which is too large to be reasonable for an installable package. Given that most production uses would be apply considerably less than the available 511 locales a different strategy is required. The planned approach is as follows:

1. The full data will remain in the github repo and the code can continue to be used directly from github.
2. A mix task cldr.consolidate will consolidate all locale-specific information for each locale into a single directory in the Cldr repo called cldr
3. The non locale-specific data in CLDR (cldr-core, common and cldr-rbnf) will be copied to the cldr directory
4. A compressed archive file will be created for each locale that includes the single locale-specific json file plus the common non-locale specific content.
5. An additional compressed archive will be created that contains all locales
6. A mix task cldr.get_locales will read the configuration and download the locales requested from github directly and expand them into a user configurable directory that will default to ./priv/cldr. The download will be come from the github branch reflecting the installed version of the Cldr package to ensure repeatable outcomes even as the package evolves.

Not found locale for `pt-BR`

Using {:ex_money, "~> 2.1.0-rc.1"}
Elixir 1.6

== Compilation error in file lib/cldr.ex ==
** (Cldr.UnknownLocaleError) Some locale names are configured that are not known to CLDR. Compilation cannot continue until the configuration includes only locales names known in CLDR.

Configured locales names: ["en", "en-001", "en-GB", "es", "pt", "pt-BR", "root", "ru"]
Gettext locales names:    []
Unknown locales names:    ["pt-BR"]

Issue compiling on windows

Cldr.Install.ensure_client_dirs_exist! does try to create the following directories on windows:

Generating Cldr for 10 locales ["de", "de-AT", "de-BE", "de-CH", "de-IT", ...] with default locale "en"
["/c:", "/Users", "/fklement", "/Desktop", "/connect", "/_build", "/dev",
 "/lib", "/ex_cldr", "/priv", "/cldr", "/locales"]

For the following correct path: c:/Users/fklement/Desktop/connect/…

Noticed by @fklement

Does not support exponent grouping in scientific formats

TR35 states that for scientific formats (i.e. mantissa and exponent):

The maximum number of integer digits, if present, specifies the exponent grouping. The most common use of this is to generate engineering notation, in which the exponent is a multiple of three, for example, "##0.###E0". The number 12345 is formatted using "##0.####E0" as "12.345E3".

Cldr does not currently support such functionality.

Cldr.known_currencies spec vs example

The spec suggests a list of strings being returned, the example says it's a list of atoms.

tk

Support Poison 3.0

Most of the libraries I'm using have moved to Poison 3.0+, leaving this package as the last holdout.

Sometimes, both major versions are supported by specifying "~> 2.2 or ~> 3.0", including Phoenix 1.3.

Moreover, devinus/poison doesn't show a 2.0 branch so I'm not even sure if any fixes are backported.

Downloading configured locales does not work if ex_cldr is installed from Hex

Due to my misunderstanding on how the priv dir works and how Mix fits into hex packages and releases the mechanism to download configured locales is quite incorrect. As a result, configured locales are not installed as expected and the mix task cldr.install.locales fails miserably.

compile failed with ex_cldr 1.8.2

I got this error when running mix compile:
I'm using: ex_cldr 1.8.2, Elixir: 1.6.5, erlang 20.2

== Compilation error in file lib/cldr/language_tag/rfc5646_parser.ex ==
** (FunctionClauseError) no function clause matching in NimbleParsec.lookahead/2

    The following arguments were given to NimbleParsec.lookahead/2:

        # 1
        [choice: [[{:label, [{:traverse, [{:label, [choice: [[{:traverse, [{:traverse, [{:traverse, [{:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [5, {:integer, [], NimbleParsec}]}]}, {:times, [{:bin_segment, [97..122, 65..90], [], [:integer]}], 0, 3}], :post, [{NimbleParsec, :__runtime_string__, [5, 8, {:integer, [], NimbleParsec}]}]}], :post, [{NimbleParsec, :__unwrap_and_tag__, [:language]}]}], [{:traverse, [{:traverse, [{:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [4, {:integer, [], NimbleParsec}]}]}], :post, [{NimbleParsec, :__unwrap_and_tag__, [:language]}]}], [{:label, [{:traverse, [{:traverse, [{:traverse, [{:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [2, {:integer, [], NimbleParsec}]}]}, {:times, [{:bin_segment, [97..122, 65..90], [], [:integer]}], 0, 1}], :post, [{NimbleParsec, :__runtime_string__, [2, 3, {:integer, [], NimbleParsec}]}]}], :post, [{NimbleParsec, :__unwrap_and_tag__, [:language]}]}, {:choice, [[{:traverse, [{:traverse, [{:bin_segment, '-', [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {:integer, [], NimbleParsec}]}]}], :constant, [{NimbleParsec, :__constant__, [[]]}]}, {:label, [choice: [[{:label, [{:traverse, [{:traverse, [{:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [4, {:integer, [], NimbleParsec}]}]}], :post, [{NimbleParsec, :__unwrap_and_tag__, [:script]}]}], "a script id of four alphabetic character"}], [{:traverse, [{:traverse, [{:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [3, {:integer, [], NimbleParsec}]}]}, {:traverse, [{:traverse, [{:bin_segment, '-', [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {:integer, [], ...}]}]}], :constant, [{NimbleParsec, :__constant__, [[]]}]}, {:traverse, [{:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [3, {:integer, [], NimbleParsec}]}]}, {:traverse, [{:traverse, [{:bin_segment, '-', [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {...}]}]}], :constant, [{NimbleParsec, :__constant__, [[]]}]}, {:traverse, [{:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [3, {:integer, [], ...}]}]}], :post, [{NimbleParsec, :__tag__, [:language_subtags]}]}, {:traverse, [{:traverse, [{:bin_segment, '-', [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {:integer, [], NimbleParsec}]}]}], :constant, [{NimbleParsec, :__constant__, [[]]}]}, {:label, [{:traverse, [{:traverse, [{:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [],[:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [4, {:integer, ...}]}]}], :post, [{NimbleParsec, :__unwrap_and_tag__, [:script]}]}], "a script id of four alphabetic character"}], [{:traverse, [{:traverse, [{:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [3, {:integer, [], NimbleParsec}]}]}, {:traverse, [{:traverse, [{:bin_segment, '-', [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1,{:integer, ...}]}]}], :constant, [{NimbleParsec, :__constant__, [[]]}]}, {:traverse, [{:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [3, {:integer, [], NimbleParsec}]}]}], :post, [{NimbleParsec, :__tag__, [:language_subtags]}]}, {:traverse, [{:traverse, [{:bin_segment, '-', [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {:integer, [], NimbleParsec}]}]}], :constant, [{NimbleParsec, :__constant__, [[]]}]}, {:label, [{:traverse, [{:traverse, [{:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [...]}], :post, [{NimbleParsec, :__compile_string__, [4, {...}]}]}], :post, [{NimbleParsec, :__unwrap_and_tag__, [:script]}]}], "a script id of four alphabetic character"}], [{:traverse, [{:traverse, [{:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [3, {:integer, [], NimbleParsec}]}]}, {:traverse, [{:traverse, [{:bin_segment, '-', [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {...}]}]}], :constant, [{NimbleParsec, :__constant__, [[]]}]}], :post, [{NimbleParsec, :__tag__, [:language_subtags]}]}, {:label, [{:traverse, [{:traverse, [{:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [...]}], :post, [{NimbleParsec, :__compile_string__, [4, {...}]}]}], :post, [{NimbleParsec, :__unwrap_and_tag__, [:script]}]}], "a script id of four alphabetic character"}], [{:traverse, [{:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [3, {:integer, [], NimbleParsec}]}]}, {:traverse, [{:traverse, [{:bin_segment, '-', [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {:integer, [], ...}]}]}], :constant, [{NimbleParsec, :__constant__, [[]]}]}, {:traverse, [{:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [3, {:integer, [], NimbleParsec}]}]}, {:traverse, [{:traverse, [{:bin_segment, '-', [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {...}]}]}], :constant, [{NimbleParsec, :__constant__, [[]]}]}, {:traverse, [{:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [3, {:integer, [], ...}]}]}], [{:traverse, [{:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [3, {:integer, [], NimbleParsec}]}]}, {:traverse, [{:traverse, [{:bin_segment, '-', [],[:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {:integer, ...}]}]}], :constant, [{NimbleParsec, :__constant__, [[]]}]}, {:traverse, [{:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [3, {:integer, [], NimbleParsec}]}]}], [{:traverse, [{:traverse, [{:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [3, {:integer, ...}]}]}], :post, [{NimbleParsec, :__tag__, [:language_subtags]}]}]]], "an ISO-639 language code of between one and three three alphabetic characters"}], []]}], "an ISO-639 language code of two or three alphabetic characters"}]]], "an ISO-639 country code or between 4 and 8 alphabetic characters"}, {:choice, [[{:traverse, [{:traverse, [{:bin_segment, '-', [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {:integer, [], NimbleParsec}]}]}], :constant, [{NimbleParsec, :__constant__, [[]]}]}, {:label, [{:traverse, [{:traverse, [{:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [4, {:integer,[], NimbleParsec}]}]}], :post, [{NimbleParsec, :__unwrap_and_tag__, [:script]}]}], "a script id of four alphabetic character"}], []]}, {:repeat, [{:traverse, [{:traverse, [{:bin_segment, '-', [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {:integer, [], NimbleParsec}]}]}], :constant, [{NimbleParsec, :__constant__, [[]]}]}, {:label, [{:traverse, [choice: [[{:traverse, [{:traverse, [{:bin_segment, [97..122, 65..90, 48..57], [], [:integer]}, {:bin_segment, [97..122, 65..90, 48..57], [], [:integer]}, {:bin_segment, [97..122, 65..90, 48..57], [], [:integer]}, {:bin_segment, [97..122, 65..90, 48..57], [], [:integer]}, {:bin_segment, [97..122, 65..90, 48..57], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [5, {:integer, [], NimbleParsec}]}]}, {:times, [{:bin_segment, [97..122, 65..90, 48..57], [], [:integer]}], 0, 3}], :post, [{NimbleParsec, :__runtime_string__, [5, 8, {:integer, [], NimbleParsec}]}]}], [{:traverse, [{:traverse, [{:bin_segment, [48..57], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {:integer, [], NimbleParsec}]}]}, {:traverse, [{:bin_segment, [97..122, 65..90, 48..57], [], [:integer]}, {:bin_segment, [97..122, 65..90, 48..57], [], [:integer]}, {:bin_segment, [97..122, 65..90, 48..57], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [3, {:integer, [], NimbleParsec}]}]}], :post, [{NimbleParsec, :__reduce__, [{Enum, :join, []}]}]}]]], :post, [{NimbleParsec, :__unwrap_and_tag__, [:language_variant]}]}], "a language variant code of five to eight alphabetic character or a single digit plus three alphanumeric characters"}], {NimbleParsec, :__cont_context__, []}}, {:choice, [[{:traverse, [{:traverse, [{:bin_segment, '-', [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {:integer, [], NimbleParsec}]}]}], :constant, [{NimbleParsec, :__constant__, [[]]}]}, {:label, [{:traverse, [choice: [[{:traverse, [{:bin_segment, [97..122, 65..90], [], [:integer]}, {:bin_segment, [97..122, 65..90], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [2, {:integer, [], NimbleParsec}]}]}], [{:traverse, [{:bin_segment, [48..57], [], [:integer]}, {:bin_segment, [48..57], [], [:integer]}, {:bin_segment, [48..57], [], [:integer]}], :post, [{NimbleParsec, :__compile_integer__, [3]}]}]]], :post, [{NimbleParsec, :__unwrap_and_tag__, [:territory]}]}], "a territory code of two alphabetic character ISO-3166-1 code or a three digit UN M.49 code"}],[]]}, {:repeat, [{:traverse, [{:traverse, [{:bin_segment, '-', [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {:integer, [], NimbleParsec}]}]}], :constant, [{NimbleParsec, :__constant__, [[]]}]}, {:label, [{:traverse, [choice: [[{:traverse, [{:traverse, [{:bin_segment, [97..122, 65..90, 48..57], [], [:integer]}, {:bin_segment, [97..122, 65..90, 48..57], [], [:integer]}, {:bin_segment, [97..122, 65..90, 48..57], [], [:integer]}, {:bin_segment, [97..122, 65..90, 48..57], [], [:integer]}, {:bin_segment, [97..122, 65..90, 48..57], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [5, {:integer, [], NimbleParsec}]}]}, {:times, [{:bin_segment, [97..122, 65..90, 48..57], [], [:integer]}], 0, 3}], :post, [{NimbleParsec, :__runtime_string__, [5, 8, {:integer, [], NimbleParsec}]}]}], [{:traverse, [{:traverse, [{:bin_segment, [48..57], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {:integer, [], NimbleParsec}]}]}, {:traverse, [{:bin_segment, [97..122, 65..90, 48..57], [], [:integer]}, {:bin_segment, [97..122, 65..90, 48..57], [], [:integer]}, {:bin_segment, [97..122, 65..90, 48..57], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [3, {:integer, [], NimbleParsec}]}]}], :post, [{NimbleParsec, :__reduce__, [{Enum, :join, []}]}]}]]], :post, [{NimbleParsec, :__unwrap_and_tag__, [:language_variant]}]}], "a language variant code of five to eight alphabetic character or a single digit plus three alphanumeric characters"}], {NimbleParsec, :__cont_context__, []}}, {:repeat, [{:traverse, [{:traverse, [{:traverse, [{:bin_segment, '-', [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {:integer, [], NimbleParsec}]}]}], :constant, [{NimbleParsec, :__constant__, [[]]}]}, {:choice, [[{:label, [{:traverse, [{:traverse, [{:traverse, [{:traverse, [{:bin_segment,'uU', [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {:integer, [], NimbleParsec}]}]}], :constant, [{NimbleParsec, :__constant__, [[]]}]}, {:choice, [[{:traverse, [{:traverse, [{:traverse, [{:bin_segment, '-', ...}], :post, [{...}]}], :constant, [{NimbleParsec, :__constant__, [...]}]}, {:traverse, [{:traverse, [{:bin_segment, ...}, {...}, ...], :post, [...]}, {:times, [{...}], 0, ...}], :post, [{NimbleParsec, :__runtime_string__, ...}]}, {:repeat, [{:traverse, [{...}], :constant, ...}, {:traverse, [...], ...}], {NimbleParsec, :__cont_context__, []}}], :post, [{NimbleParsec, :__tag__, [:attributes]}]}, {:traverse, [{:repeat, [{:traverse, [{:traverse, ...}], :constant, [...]}, {:label, [{...}, ...], "a valid keyword or keyword-type pair"}], {NimbleParsec, :__cont_context__, []}}], :post, [{NimbleParsec, :__reduce__, [:collapse_keywords]}]}], [{:traverse, [{:repeat, [{:traverse, [{:traverse, ...}], :constant, [...]}, {:label, [{...}, ...], "a valid keyword or keyword-type pair"}], {NimbleParsec, :__cont_context__, []}}], :post, [{NimbleParsec, :__reduce__, [:collapse_keywords]}]}]]}], :post, [{NimbleParsec, :__reduce__, [:combine_attributes_and_keywords]}]}], :post, [{NimbleParsec, :__unwrap_and_tag__, [:locale]}]}], "a BCP-47 language tag locale extension"}], [{:label, [{:traverse, [{:traverse, [{:traverse, [{:traverse, [{:bin_segment, 'tT', [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {:integer, [], ...}]}]}], :constant, [{NimbleParsec, :__constant__, [[]]}]}, {:traverse, [{:traverse, [{:bin_segment, '-', [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {:integer, ...}]}]}], :constant, [{NimbleParsec, :__constant__, [[]]}]}, {:label, [{:label, [{:traverse, [{:traverse, [{:bin_segment, ...}, {...}], :post, [...]}], :post, [{NimbleParsec, :__unwrap_and_tag__, ...}]}], "a key of two alphanumeric characters"}, {:choice, [[{:traverse, [{:traverse, [...], ...}], :constant, [{...}]}, {:label, [{:traverse, ...}, {...}], "a type that is one or more three to eight alphanumeric characters separated by a dash"}], []]}], "a valid keyword or keyword-type pair"}], :post, [{NimbleParsec, :__reduce__, [:collapse_keywords]}]}, {:repeat, [{:traverse, [{:traverse, [{:traverse, [{:bin_segment, 'tT', [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [...]}]}], :constant, [{NimbleParsec, :__constant__, [[]]}]}, {:traverse, [{:traverse, [{:bin_segment, '-', [], [...]}], :post, [{NimbleParsec, :__compile_string__, ...}]}], :constant, [{NimbleParsec, :__constant__, [[]]}]}, {:label, [{:label, [{:traverse, [{...}], :post, ...}], "a key of two alphanumeric characters"}, {:choice, [[{:traverse, ...}, {...}], []]}], "a valid keyword or keyword-type pair"}], :post, [{NimbleParsec, :__reduce__, [:collapse_keywords]}]}], {NimbleParsec, :__cont_context__, []}}], :post, [{NimbleParsec,:__unwrap_and_tag__, [:transform]}]}], "a BCP-47 language tag transform extension"}], [{:label, [{:traverse, [{:traverse, [{:traverse, [{:label, [{:traverse, [{:bin_segment, [48..57, 97..115, 65..83, ...], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [...]}]}], "a single alphanumeric character that is not 'x', 'u' or 't'"}], :post, [{NimbleParsec, :__unwrap_and_tag__, [:type]}]}, {:traverse, [{:traverse, [{:traverse, [{:bin_segment, '-', [], [...]}], :post, [{NimbleParsec, :__compile_string__, ...}]}], :constant, [{NimbleParsec, :__constant__, [[]]}]}, {:traverse, [{:traverse, [{:bin_segment, [97..122, ...], [], ...}, {:bin_segment, [...], ...}], :post, [{NimbleParsec, ...}]}, {:times, [{:bin_segment, [...], ...}], 0, 6}], :post, [{NimbleParsec, :__runtime_string__, [2, ...]}]}], :post, [{NimbleParsec, :__unwrap_and_tag__, [:attribute]}]}, {:repeat, [{:traverse, [{:traverse, [{:traverse, [{...}], :post, ...}], :constant, [{NimbleParsec, ...}]}, {:traverse, [{:traverse, [...], ...}, {:times, ...}], :post, [{...}]}], :post, [{NimbleParsec, :__unwrap_and_tag__, [:attribute]}]}], {NimbleParsec, :__cont_context__, []}}], :post, [{NimbleParsec, :__reduce__, [:collapse_extension]}]}], :post, [{NimbleParsec, :__unwrap_and_tag__, [:extension]}]}], "a valid BCP-47 language tag extension"}]]}], :post, [{NimbleParsec, :__reduce__, [:collapse_extensions]}]}], {NimbleParsec, :__cont_context__, []}}, {:choice, [[{:traverse, [{:traverse, [{:bin_segment, '-', [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {:integer, [], NimbleParsec}]}]}], :constant, [{NimbleParsec, :__constant__, [[]]}]}, {:label, [{:traverse, [{:traverse, [{:traverse, [{:bin_segment, 'xX', [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {:integer, [], NimbleParsec}]}]}], :constant, [{NimbleParsec, :__constant__, [[]]}]}, {:traverse, [{:traverse, [{:bin_segment, '-', [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {:integer, [], NimbleParsec}]}]}], :constant, [{NimbleParsec, :__constant__, [[]]}]}, {:traverse, [{:traverse, [{:bin_segment, [97..122, 65..90, 48..57], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {:integer, [], NimbleParsec}]}]}, {:times, [{:bin_segment, [97..122, 65..90, 48..57], [], [:integer]}], 0, 7}], :post, [{NimbleParsec, :__runtime_string__, [1, 8, {:integer, [], NimbleParsec}]}]}, {:repeat, [{:traverse, [{:traverse, [{:bin_segment, '-', [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {:integer, [], NimbleParsec}]}]}], :constant, [{NimbleParsec, :__constant__, [[]]}]}, {:traverse, [{:traverse, [{:bin_segment, [97..122, 65..90, 48..57], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {:integer, [], NimbleParsec}]}]}, {:times, [{:bin_segment, [97..122, 65..90, 48..57], [], [:integer]}], 0, 7}], :post, [{NimbleParsec, :__runtime_string__, [1, 8, {:integer, [], NimbleParsec}]}]}], {NimbleParsec, :__cont_context__, []}}], :post, [{NimbleParsec, :__tag__, [:private_use]}]}], "an 'x' representing a private use tag"}], []]}], :post, [{NimbleParsec, :__post_traverse__, [flatten: []]}]}], "a valid BCP-47 language tag"}], [{:label, [{:traverse, [{:traverse, [{:traverse, [{:bin_segment, 'xX', [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {:integer, [], NimbleParsec}]}]}], :constant, [{NimbleParsec, :__constant__, [[]]}]}, {:traverse, [{:traverse, [{:bin_segment, '-', [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {:integer, [], NimbleParsec}]}]}], :constant, [{NimbleParsec, :__constant__, [[]]}]}, {:traverse, [{:traverse, [{:bin_segment, [97..122, 65..90, 48..57], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {:integer, [], NimbleParsec}]}]}, {:times, [{:bin_segment, [97..122, 65..90, 48..57], [], [:integer]}], 0, 7}], :post, [{NimbleParsec, :__runtime_string__, [1, 8, {:integer, [], NimbleParsec}]}]}, {:repeat, [{:traverse, [{:traverse, [{:bin_segment, '-', [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {:integer, [], NimbleParsec}]}]}], :constant, [{NimbleParsec, :__constant__, [[]]}]}, {:traverse, [{:traverse, [{:bin_segment, [97..122, 65..90, 48..57], [], [:integer]}], :post, [{NimbleParsec, :__compile_string__, [1, {:integer, [], NimbleParsec}]}]}, {:times,[{:bin_segment, [97..122, 65..90, 48..57], [], [:integer]}], 0, 7}], :post, [{NimbleParsec, :__runtime_string__, [1, 8, {:integer, [], NimbleParsec}]}]}], {NimbleParsec, :__cont_context__, []}}], :post, [{NimbleParsec, :__tag__, [:private_use]}]}], "an 'x' representing a private use tag"}], [{:label, [{:traverse, [choice: [[{:label, [{:traverse, [choice: [[string: "en-GB-oed"], [string: "i-ami"], [string: "i-bnn"], [string: "i-default"], [string: "i-enochian"], [string: "i-hak"], [string: "i-klingon"], [string: "i-lux"], [string: "i-mingo"], [string: "i-navajo"], [string: "i-pwn"], [string: "i-tao"], [string: "i-tay"], [string: "i-tsu"], [string: "sgn-BE-FR"], [string: "sgn-BE-NL"], [string: "sgn-CH-DE"]]], :post, [{NimbleParsec, :__unwrap_and_tag__, [:irregular]}]}], "one of the irregular language tags in BCP-47"}], [{:label, [{:traverse, [choice: [[string: "art-lojban"], [string: "cel-gaulish"], [string: "no-bok"], [string: "no-nyn"], [string: "zh-guoyu"], [string: "zh-hakka"], [string: "zh-min"], [string: "zh-min-nan"], [string: "zh-xiang"]]], :post, [{NimbleParsec, :__unwrap_and_tag__, [:regular]}]}], "one of the regular language tags in BCP-47"}]]], :post, [{NimbleParsec, :__tag__, [:grandfathered]}]}], "a grandfathered language tag"}]]]

        # 2
        :error_on_remaining

    Attempted function clauses (showing 1 out of 1):

        def lookahead(combinator, to_lookahead) when is_list(combinator) and is_list(to_lookahead)

    lib/nimble_parsec.ex:849: NimbleParsec.lookahead/2
    lib/cldr/language_tag/rfc5646_parser.ex:31: (module)
    (stdlib) erl_eval.erl:670: :erl_eval.do_apply/6
could not compile dependency :ex_cldr, "mix compile" failed. You can recompile this dependency with "mix deps.compile ex_cldr", update it with "mix deps.update ex_cldr" or clean it with "mix deps.clean ex_cldr"

Documentation for 1.3.2 uses `config :cldr` (instead of `config :ex_cldr` ?)

Following https://hexdocs.pm/ex_cldr/readme.html#getting-started, I just installed cldr from hex adding this in my mix.exs:

{:ex_cldr, "~> 1.0"},

This gets me 1.3.2.
The documentation (and the README.md from master) explain to configure with:

config :cldr,
   default_locale: ....

However I could not download any locale other than ["en-001", "root"]. After some googling I found that this other way to configure existed:

config :ex_cldr,
   default_locale: ....

This seems to work better. Is it just a documentation issue ? Which one should I use ?

Replace phoenix dependency with just plug

I'm not sure if phoenix is currently used elsewhere besides the tests (github showed 48 pages of searchresults), but if not we should probably replace it with just plug. No need to recompile cldr if just phoenix is updated.

Code.LoadError with 2.x

Hi,

I'm not sure if this is coming from ex_cldr_numbers or ex_cldr itself but upgrading to 2.x makes my phoenix not compile anymore with Code.LoadError raised randomly (e.g. the error below happens in different files upon several mix compile runs)

mix compile:

== Compilation error in file lib/decapx_web/controllers/pdf_controller.ex ==
** (Code.LoadError) could not load /Users/julien/dev/repos/decapx/deps/postgrex/lib/decapx_web/controllers/pdf_controller.ex
    (elixir) lib/code.ex:1147: Code.find_file/2
    (elixir) lib/code.ex:928: Code.compile_file/2
    (elixir) lib/kernel/parallel_compiler.ex:206: anonymous fn/4 in Kernel.ParallelCompiler.spawn_workers/6

iex gives a bit more info (note the different file):
iex -S mix

== Compilation error in file lib/decapx/works/chantier.ex ==
** (Code.LoadError) could not load /Users/julien/dev/repos/decapx/deps/ex_cldr/lib/decapx/works/chantier.ex
    (elixir) lib/code.ex:1147: Code.find_file/2
    (elixir) lib/code.ex:928: Code.compile_file/2
    (elixir) lib/kernel/parallel_compiler.ex:206: anonymous fn/4 in Kernel.ParallelCompiler.spawn_workers/6
** (MatchError) no match of right hand side value: {:error, :bad_directory}
    (mix) lib/mix/tasks/compile.all.ex:24: anonymous fn/1 in Mix.Tasks.Compile.All.run/1
    (mix) lib/mix/tasks/compile.all.ex:39: Mix.Tasks.Compile.All.with_logger_app/1
    (mix) lib/mix/task.ex:316: Mix.Task.run_task/3
    (mix) lib/mix/tasks/compile.ex:94: Mix.Tasks.Compile.run/1
    (mix) lib/mix/task.ex:316: Mix.Task.run_task/3
    (mix) lib/mix/tasks/app.start.ex:57: Mix.Tasks.App.Start.run/1
    (mix) lib/mix/task.ex:316: Mix.Task.run_task/3
    (mix) lib/mix/tasks/run.ex:129: Mix.Tasks.Run.run/5
    (mix) lib/mix/tasks/run.ex:85: Mix.Tasks.Run.run/1
    (mix) lib/mix/task.ex:316: Mix.Task.run_task/3
    (mix) lib/mix/cli.ex:79: Mix.CLI.run_task/2
    (elixir) lib/code.ex:767: Code.require_file/2

Here are the differences from 1.x to 2.x:

config/config.exs

-config :ex_cldr,
-  default_locale: "fr",
-  locales: ["fr"]

lib/decapx/cldr.ex

+defmodule Decapx.Cldr do
+  use Cldr,
+    default_locale: "fr",
+    locales: ["fr"]
+end

mix.exs

-      {:ex_cldr_numbers, "~> 1.0"},
+      {:ex_cldr_numbers, "~> 2.0"},

and where I'm using it, the difference is simply Cldr.Number.to_string becoming Decapx.Cldr.Number.to_string.

Looking around, I found this issue that looks somewhat similar but it looks like it's been resolved.

Also, I wonder if I need to keep cldr in the compilers list, e.g.

compilers: [:phoenix, :gettext] ++ Mix.compilers() ++ [:cldr],

(keeping it as is or removing it didn't make any difference)

Running "Erlang/OTP 21 [erts-10.1.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe] [dtrace]" and Elixir 1.7.4.

Let me know if you need more info, thanks!

Elixir 1.4 issues warnings for unused module attributes

This is mostly a good thing, but in the few cases where credo hints are applied with the @lint module document we end of with some undesired warnings. Better to remove them and solve the credo issues by configuration or another way,

Not loading gettext locales (v1.3)

Hi,
How am I supposed to use cldr with gettext in a phoenix app? I configure my Gettext module properly and set it in the cldr config. But Cldr.known_gettext_locale_names just returns []

Surprisingly, if I change your code to

  @known_gettext_locale_names Config.gettext_locales()
                              |> Enum.map(&Locale.locale_name_from_posix/1)

  @spec known_gettext_locale_names :: [Locale.locale_name(), ...] | []
  def known_gettext_locale_names do
    Config.gettext_locales()
    |> Enum.map(&Locale.locale_name_from_posix/1)
  end

Everything is working as expected. Somehow the gettext locales are not available at compile time

Locale extraction with Cldr.Plug.SetLocale

The locale parameter extraction in this plug puzzles me. See i.e. this route:

scope "/:locale", MyApp do
   #stuff
end

Which configuration should i use? SetLocale, apps: [:gettext, :cldr], from: [:url] isn't working. Looking at the code I don't understand the parameter url_params. conn.url_params doesn't even exist here

Have a nice day!

[BUG] Cldr.Normalize.Currency remove or overwrite essential data

Hi @kipcole9,

There is a bug in Cldr.Normalize.Currency

When we normalize the currencyData.json we end up with a map but the problem is that some countries have multiple entries of the same currency code for example "PS" L2568-L2591 the JOD appears twice so when we create a map from that data we get malformed data because an elixir map can only contain unique keys

with these limitation I would recommend that we use Keyword list where we also can preserve the JSON structere.

We get this

%{
  ILP: %{from: ~D[1967-06-01], to: ~D[1980-02-22]},
  ILS: %{from: ~D[1985-09-04]},
  JOD: %{from: ~D[1996-02-12], to: ~D[1967-06-01]}
}

but it should be this

[
  JOD: %{from: ~D[1950-07-01], to: ~D[1967-06-01}
  ILP: %{from: ~D[1967-06-01], to: ~D[1980-02-22]},
  ILS: %{from: ~D[1985-09-04]},
  JOD: %{from: ~D[1996-02-12]},
]

:json_library or :json_lib

Cldr.Config does check for :json_lib in the application env instead of the documented :json_library.

Cldr.DateTime.to_string seems extremely slow

Today we noticed that a specific page in our production Elixir/Phoenix application would load much slower than expected, sometimes taking 5+ seconds to load completely. Initially we suspected slow Javascript load times, but local debugging led us to this call in one of our view helpers:

Cldr.DateTime.to_string(payment.inserted_at, format: :medium, locale: "nl")

Where payment.inserted_at is a UTC DateTime from a database table, retrieved with Ecto. I ran a profile of this specific function with ex_prof on a list of 25 payment records:

import ExProf.Macro

profile do
  Enum.map(payments, fn(payment) ->
    Cldr.DateTime.to_string(payment.inserted_at, format: :medium, locale: "nl")
  end)
end

The full results are here.

Benchmarking the same call with Benchee shows that the 99th% takes 2.13s:

Benchee.run(%{ "ex_cldr" => fn -> Enum.map(payments, fn(payment) -> Cldr.DateTime.to_string(payment.inserted_at, format: :medium, locale: "nl") end) end })
Operating System: macOS
CPU Information: Intel(R) Core(TM) i5-6600K CPU @ 3.50GHz
Number of Available Cores: 4
Available memory: 16 GB
Elixir 1.6.0
Erlang 20.2.2
Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
parallel: 1
inputs: none specified
Estimated total run time: 7 s


Benchmarking ex_cldr...

Name              ips        average  deviation         median         99th %
ex_cldr          0.49         2.05 s     ±4.17%         2.09 s         2.13 s

This is our configuration for ex_cldr:

config :ex_cldr,
       default_locale: "en",
       locales: ["en", "nl"],
       gettext: MyApp.Web.Gettext

Are we missing something in our configuration or is this an actual issue in ExCldr?

Missing github tag

I currently get those errors with the rc.1:

Failed to download locale "de-LU" from https://raw.githubusercontent.com/kipcole9/cldr/v1.4.0-rc.1/priv/cldr/locales/de-LU.json. HTTP Error: (404) 'Not Found'

Always using international currency symbols

Hello!
Is this possible to always use international currency symbols (EUR, USD) etc, instead of localized symbols (€, $)? Per the spec it should be left to the application [1].

E.g.: I'd like to be able always represent numbers in the format 123,456.78 EUR regardless of the locale (however, , and . would be replaced with appropriate symbols specific to locale)

If it's not currently possible, I'd be happy to give it a shot but I'd probably need some pointers.

PS thank you for this library, it seems it's exactly what we need for our application!

[1] http://cldr.unicode.org/translation/number-patterns, section "General Purpose Numbers"

Doesn't compile

I'm trying to incorporate your library ex_money into my phoenix app. When I try to compile, this is what I get:

Could you please fix asap?

Creating an elixir-cldr organization

Hi,

Would it be worth it to create an organization for ex_cldr and its supporting libraries?

Pros

• Code unification across libraries with mix format
• One CI for testing across the libraries
• We can help each other out and delegate responsibility
• Code reviews

I'm not sure about the cons, let me know what you think.

cc @kipcole9 @LostKobrakai

Conditional inclusion of CLDR components

Cldr consists of a lot of static text, multiplied by up to 511 locales. Today the number of locales can be configured but not the CLDR components requested. As more functionality is added, the more the memory footprint will grow.

The idea is to be able to configure in mix.exs which components are to be included in an installation. Something like:

config :ex_cldr,
  components: [:number, :currency, :territory, :date]

And then conditionally compile those modules into the application.

:priv_dir files from a sibling dependency get deleted using cldr compiler

First of all, thanks for the useful lib!!!

I've been chasing this bug on production, where my monitoring tool (appsignal) stopped reporting data, and it was because on compile time, appsignal downloads some files.

# relevant part of mix compile --force
Generated sylar app
-----> appsignal on: before deps.clean
       Directory /tmp/build_5ef2fa06241626cf7c9113fb77ca137a/_build/prod/lib/appsignal/priv
       appsignal-agent
       appsignal.architecture
       appsignal_extension.so
       appsignal.h
       appsignal.version
       cacert.pem
       libappsignal.a

It turns out, after the compile step is done, those files are not there anymore. They get deleted.
But, if I change my configuration from:
compilers: [:phoenix, :gettext] ++ Mix.compilers() ++ [:cldr],
to
compilers: [:phoenix, :gettext] ++ Mix.compilers(),
the files stay there.

Extrangely, this does not happen on my local machine, not even compiling in prod environment. But it does happen 100% of the time on heroku and my staging servers (normal docker setup)

Can you shed any light on this? how can I help on the debug?

Cldr.Normalize.LanguageName incomplete copy of Cldr.Normalize.TerritoryNames

It seems like Cldr.Normalize.LanguageName was copied from Cldr.Normalize.TerritoryName, but things like moduledoc / function names wheren't updated.

Dialyzer Errors in v2

Let me first thank you for the great library!

I have upgraded my CLDR dependencies to the newest major release. Now I have a lot of Dialyzer Errors:

$ mix dialyzer --halt-exit-status
Finding suitable PLTs
Checking PLT...
[:asn1, :certifi, :cldr_utils, :compiler, :connection, :cowboy, :cowlib, :crypto, :csv, :db_connection, :decimal, :ecto, :ecto_enum, :ecto_sql, :eex, :elixir, :ex_cldr, :ex_cldr_currencies, :ex_cldr_numbers, :file_system, :flow, :gen_stage, :gettext, :hackney, :html_sanitize_ex, :idna, :inets, :jason, :kernel, :logger, :mariaex, :metrics, :mime, :mimerl, :mochiweb, :nimble_parsec, :oauth2, :parallel_stream, :phoenix, :phoenix_active_link, :phoenix_ecto, :phoenix_html, :phoenix_live_reload, :phoenix_pubsub, :plug, :plug_cowboy, :plug_crypto, :public_key, :ranch, :runtime_tools, ...]
PLT is up to date!
Starting Dialyzer
[
  check_plt: false,
  init_plt: '/acme/_build/dev/dialyxir_erlang-21.2.3_elixir-1.8.0_deps-dev.plt',
  files_rec: ['/acme/_build/dev/lib/acme_web/ebin',
   '/acme/_build/dev/lib/acme/ebin'],
  warnings: [:unknown]
]
:0:unknown_type
Unknown type: Cldr.Locale.name/0.
________________________________________________________________________________
:0:unknown_type
Unknown type: List.t/0.
________________________________________________________________________________
:0:unknown_type
Unknown type: Map.t/0.
________________________________________________________________________________
:0:unknown_type
Unknown type: Math.number/0.
________________________________________________________________________________
:0:unknown_type
Unknown type: AcmeWeb.Cldr.Locale.locale_name/0.
________________________________________________________________________________
:0:unknown_type
Unknown type: AcmeWeb.Cldr.Locale.name/0.
________________________________________________________________________________
:0:unknown_type
Unknown type: System.name/0.
________________________________________________________________________________
apps/acme_web/lib/screen24_sales_provision_web/cldr.ex:1:invalid_contract
Invalid type specification for function.

Function:
AcmeWeb.Cldr.validate_locale/1

Success typing:
@spec validate_locale(binary() | %Cldr.LanguageTag{_ => _}) ::
  {:error, {Cldr.UnknownLocaleError, <<_::64, _::size(8)>>}}
  | {:ok, %Cldr.LanguageTag{_ => _}}
________________________________________________________________________________
apps/acme_web/lib/screen24_sales_provision_web/cldr.ex:1:invalid_contract
Invalid type specification for function.

Function:
AcmeWeb.Cldr.Currency.currency_for_code/2

Success typing:
@spec currency_for_code(binary(), Keyword.t()) ::
  {:error, {%{:__exception__ => true, :__struct__ => atom(), atom() => _}, binary()}}
  | {:ok,
     %Cldr.Currency{
       :cash_digits => non_neg_integer(),
       :cash_rounding => non_neg_integer(),
       :code => binary(),
       :count => %{},
       :digits => non_neg_integer(),
       :iso_digits => non_neg_integer(),
       :name => binary(),
       :narrow_symbol => binary(),
       :rounding => non_neg_integer(),
       :symbol => binary(),
       :tender => boolean()
     }}
________________________________________________________________________________
apps/acme_web/lib/screen24_sales_provision_web/cldr.ex:1:invalid_contract
Invalid type specification for function.

Function:
AcmeWeb.Cldr.Currency.new/2

Success typing:
@spec new(_, _) ::
  {:error,
   {Cldr.CurrencyAlreadyDefined, <<_::64, _::size(8)>>}
   | {Cldr.UnknownCurrencyError, <<_::64, _::size(8)>>}}
  | {:ok, %{:__struct__ => atom(), atom() => _}}
________________________________________________________________________________
apps/acme_web/lib/screen24_sales_provision_web/cldr.ex:1:invalid_contract
Invalid type specification for function.

Function:
AcmeWeb.Cldr.Number.to_range_string/2

Success typing:
@spec to_range_string(%Range{:first => _, :last => _, _ => _}, Keyword.t() | map()) :: any()
________________________________________________________________________________
apps/acme_web/lib/screen24_sales_provision_web/cldr.ex:1:invalid_contract
Invalid type specification for function.

Function:
AcmeWeb.Cldr.Number.Format.all_formats_for/1

Success typing:
@spec all_formats_for(binary() | %Cldr.LanguageTag{:cldr_locale_name => <<_::16, _::size(16)>>, _ => _}) ::
  {:error, {Cldr.UnknownLocaleError, <<_::64, _::size(8)>>}} | {:ok, %{:latn => map()}}
________________________________________________________________________________
apps/acme_web/lib/screen24_sales_provision_web/cldr.ex:1:invalid_contract
Invalid type specification for function.

Function:
AcmeWeb.Cldr.Number.Format.all_formats_for!/1

Success typing:
@spec all_formats_for!(binary() | %Cldr.LanguageTag{:cldr_locale_name => <<_::16, _::size(16)>>, _ => _}) :: %{
  :latn => map()
}
________________________________________________________________________________
apps/acme_web/lib/screen24_sales_provision_web/cldr.ex:1:invalid_contract
Invalid type specification for function.

Function:
AcmeWeb.Cldr.Number.Symbol.number_symbols_for/1

Success typing:
@spec number_symbols_for(binary() | %Cldr.LanguageTag{:cldr_locale_name => nil | binary(), _ => _}) ::
  {:error, {Cldr.UnknownLocaleError, <<_::64, _::size(8)>>}} | {:ok, %{:latn => map()}}
________________________________________________________________________________
apps/acme_web/lib/screen24_sales_provision_web/cldr.ex:1:invalid_contract
Invalid type specification for function.

Function:
AcmeWeb.Cldr.Number.System.number_system_names_for/1

Success typing:
@spec number_system_names_for(binary() | %Cldr.LanguageTag{:cldr_locale_name => nil | binary(), _ => _}) ::
  {:error, {Cldr.UnknownLocaleError, <<_::64, _::size(8)>>}} | {:ok, [any()]}
________________________________________________________________________________
lib/backend/format.ex:249:pattern_match
The pattern
{:error, {_, _}}

can never match the type
{:ok, 1}
________________________________________________________________________________
lib/backend/transliterate.ex:1:no_return
Function transliterate/1 has no local return.
________________________________________________________________________________
lib/backend/transliterate.ex:1:no_return
Function transliterate/2 has no local return.
________________________________________________________________________________
lib/backend/transliterate.ex:107:call
The call:
AcmeWeb.Cldr.Number.Transliterate.transliterate(
  _ :: any(),
  %Cldr.LanguageTag{
    :canonical_locale_name => binary(),
    :cldr_locale_name => nil | binary(),
    :extensions => %{},
    :gettext_locale_name => _,
    :language => binary(),
    :language_subtags => [binary()],
    :language_variant => nil | binary(),
    :locale => %{},
    :private_use => [binary()],
    :rbnf_locale_name => binary(),
    :requested_locale_name => binary(),
    :script => nil | binary(),
    :territory => nil | binary(),
    :transform => %{}
  },
  :default
)

breaks the contract
(String.t(), Cldr.LanguageTag.t(), String.t()) :: String.t()
________________________________________________________________________________
lib/backend/transliterate.ex:108:call
The call:
AcmeWeb.Cldr.Number.Transliterate.transliterate(_ :: any(), _ :: any(), :default)

breaks the contract
(String.t(), Cldr.LanguageTag.t(), String.t()) :: String.t()
________________________________________________________________________________
lib/backend/transliterate.ex:150:pattern_match
The pattern
{:error, {_, _}}

can never match the type
binary()
________________________________________________________________________________
lib/cldr/backend.ex:1:no_return
Function known_currency?/1 has no local return.
________________________________________________________________________________
lib/cldr/backend.ex:153:call
The call:
AcmeWeb.Cldr.Currency.known_currency?(_ :: any(), [])

will never return since the success typing is:
(binary(), [
  %Cldr.Currency{
    :cash_digits => non_neg_integer(),
    :cash_rounding => non_neg_integer(),
    :code => binary(),
    :count => %{},
    :digits => non_neg_integer(),
    :iso_digits => non_neg_integer(),
    :name => binary(),
    :narrow_symbol => binary(),
    :rounding => non_neg_integer(),
    :symbol => binary(),
    :tender => boolean()
  },
  ...
]) :: boolean()

and the contract is
(Cldr.Currency.code(), [Cldr.Currency.t(), ...]) :: boolean()
________________________________________________________________________________
Total errors: 23, Skipped: 0
done in 0m13.91s
done (warnings were emitted)

Territory Subdivisions

Hi, @kipcole9
I would like to contribute to CLDR.

I'm in need of Territory Subdivisions data so I can list all the cities from each country or list all countries.

how would you like to see it implemented.?
If you got any advise it would be nice.

Thanks for this library ❤️

[Question] - Fail to find ex_cldr files on heroku (prod)

From @lypborges on February 19, 2018 0:41

Thanks for the hard work on this project. But I have a question, in my dev (docker) env all works great when I try to deploy I get this error

(File.Error) could not read file /tmp/build_47658951629831269e9a6090a7cd2754/lypborges-maestro_api-1052cc38a34d0a37d7ea741de1d3420eed528652/_build/prod/lib/ ex_cldr/priv/cldr/available_locales.json": no such file or directory

Is there any other step/configuration for work in prod?

Tks again.

Copied from original issue: kipcole9/money#54

ex_cldr 1.1.0 not compiling under elixir 1.6.0-rc.0

When trying to compile my app which depends on ex_cldr using Elixir 1.6.0-rc.0, it fails with this message:

$ mix deps.clean ex_cldr
* Cleaning ex_cldr
$ mix deps.get
...
$ mix deps.compile ex_cldr
==> ex_cldr
Compiling 2 files (.erl)
Compiling 30 files (.ex)
Generating Cldr for 2 locales named ["en-001", "root"] with a default locale named "en-001"
could not compile dependency :ex_cldr, "mix compile" failed. You can recompile this dependency with "mix deps.compile ex_cldr", update it with "mix deps.update ex_cldr" or clean it with "mix deps.clean ex_cldr"
** (ArgumentError) argument error
    :erlang.get_module_info(Plug.Conn, :deprecated)
    Plug.Conn.__info__/1
    (mix) lib/mix/tasks/xref.ex:335: Mix.Tasks.Xref.load_exports_and_deprecated/1
    (mix) lib/mix/tasks/xref.ex:326: Mix.Tasks.Xref.load_exports_and_deprecated_into_acc/2
    (elixir) lib/enum.ex:1899: Enum."-reduce/3-lists^foldl/2-0-"/3
    (mix) lib/mix/tasks/xref.ex:295: anonymous fn/2 in Mix.Tasks.Xref.source_warnings/2
    (elixir) lib/enum.ex:1899: Enum."-reduce/3-lists^foldl/2-0-"/3
    (mix) lib/mix/tasks/xref.ex:291: Mix.Tasks.Xref.source_warnings/2

Compiler failing under elixir 1.7-rc

Details are already here: https://elixirforum.com/t/elixir-v1-7-0-rc-0-released/15133/20?u=lostkobrakai

could not compile dependency :ex_cldr_dates_times, "mix compile" failed. You can recompile this dependency with "mix deps.compile ex_cldr_dates_times", update it with "mix deps.update ex_cldr_dates_times" or clean it with "mix deps.cleanex_cldr_dates_times"
** (FunctionClauseError) no function clause matching in Mix.Dep.loaded/1

    The following arguments were given to Mix.Dep.loaded/1:

        # 1
        [env: :prod]

    Attempted function clauses (showing 1 out of 1):

        def loaded([])

    (mix) lib/mix/dep.ex:156: Mix.Dep.loaded/1
    lib/mix/tasks/compile.cldr.ex:86: Mix.Tasks.Compile.Cldr.compile_cldr_files/0
    lib/mix/tasks/compile.cldr.ex:67: Mix.Tasks.Compile.Cldr.run/1
    (mix) lib/mix/task.ex:316: Mix.Task.run_task/3
    (mix) lib/mix/tasks/compile.all.ex:68: Mix.Tasks.Compile.All.run_compiler/2
    (mix) lib/mix/tasks/compile.all.ex:52: Mix.Tasks.Compile.All.do_compile/4
    (mix) lib/mix/tasks/compile.all.ex:23: anonymous fn/1 in Mix.Tasks.Compile.All.run/1
    (mix) lib/mix/tasks/compile.all.ex:39: Mix.Tasks.Compile.All.with_logger_app/1

Mix.Dep.loaded is deprecated and to be removed in 1.8

Migration config v2 : A Cldr backend module must be configured

Could you be more clear about the migration? I used to have a working config for my pet project in config.ex:

config :ex_cldr,
  default_locale: "fr",
  gettext: TotoWeb.Gettext,
  json_library: Jason

I removed it to create a module, and as you never say where to put it (and it's a pet project to learn Elixir so I don't know enough to guess your intent) I put it in lib/toto/Cldr.ex. The compilation goes well, but when I try to go on a webpage I get the error "A Cldr backend module must be configured" (from calls to plug Cldr.Plug.SetLocale). Wasn't the Toto.Cldr module the backend module I had to configure? What else am I supposed to do?

Fails to compile with Distillery supplied logger level

We recent ran into an issue with compilation and ex_cldr as it concerns Logger during compilation.

Our production release uses the following config format, as specified by distillery:

config :logger, level: :"${LOG_LEVEL}"

When ex_cldr installs locales at compile time, it uses the Logger to output the message. However, at this point, the config value is :"${LOG_LEVEL}", rather than an atom value expected by the logger. This results in the following error:

== Compilation error in file lib/cldr.ex ==
       ** (FunctionClauseError) no function clause matching in Logger.level_to_number/1
       
       The following arguments were given to Logger.level_to_number/1:
       
       # 1
       :"${LOG_LEVEL}"
       
       Attempted function clauses (showing 4 out of 4):
       
       defp level_to_number(-:debug-)
       defp level_to_number(-:info-)
       defp level_to_number(-:warn-)
       defp level_to_number(-:error-)
       
       (logger) lib/logger.ex:492: Logger.level_to_number/1
       (logger) lib/logger.ex:489: Logger.compare_levels/2
       (logger) lib/logger.ex:615: Logger.bare_log/3
       lib/cldr/install.ex:90: Cldr.Install.do_install_locale_name/2
       (elixir) lib/enum.ex:737: Enum."-each/2-lists^foreach/1-0-"/2
       (elixir) lib/enum.ex:737: Enum.each/2
       lib/cldr/install.ex:25: Cldr.Install.install_known_locale_names/0
       lib/cldr.ex:43: (module)

Here is the specific line: https://github.com/kipcole9/cldr/blob/master/lib/cldr/install.ex#L90

Some RBNF functions need the "root" locale but its not configured by default

To output roman numbers and several other RBNF formats the "root" locale is required but its not configured by default.

Remove nimble_parsec runtime dependency

I'm wondering if we could remove the nimble_parsec dependency at runtime all together using: https://hexdocs.pm/nimble_parsec/Mix.Tasks.NimbleParsec.Compile.html#content

Umbrella app usage

With Gettext, each app in an umbrella app has its own configuration. However, with Cldr there's just one global configuration. This is an issue since you can configure Cldr to use a Gettext module, but there can be multiple Gettext modules.

I guess the best solution is to allow for individual app configuration, and make sure the combined Cldr data from all configurations has been downloaded. However, I'm not sure how this works with compile, or if it's at all possible. It could be something like this:

# App specific configuration
config :ex_cldr, MyApp,
  gettext: MyApp.Gettext

# Pull all configs
Application.get_all_env(:ex_cldr)

@kipcole9 do you have any suggestions for how to make Cldr work in an umbrella app context?

Plug Warning

If the Cldr.Plug.AcceptLanguage plug is used, a warning is logged.

Endpoint Code:

defmodule Acme.Endpoint do
  use Phoenix.Endpoint, otp_app: :acme

  # ...

  plug(Cldr.Plug.AcceptLanguage)

  # ...
end

Log:

16:16:24.920 [warn]  Elixir.Cldr.Plug.AcceptLanguage does not support configuration options. Please remove [] from the plug invokation.

Library Versions:

• phoenix - 1.4.0
• plug - 1.7.1
• ex_cldr - 1.8.1

Fails to compile when default locale isn’t "en"

Steps to Replicate

1. Create a new Mix project and add Cldr as a dependency.

def deps do
  [{:ex_cldr, "0.4.1"}]
end

2. Set default_locale to en-SG.

config :ex_cldr,
  default_locale: "en-SG"

3. Run mix do deps.get, compile.

Stacktrace

==> ex_cldr
Compiling 6 files (.erl)
Compiling 40 files (.ex)
Generating Cldr for 3 locales ["en-SG", "root"] with default locale "en-SG"

== Compilation error on file lib/cldr.ex ==
** (UndefinedFunctionError) function Cldr.all_locales/0 is undefined (module Cldr is not available)
    Cldr.all_locales()
    lib/cldr/install.ex:62: Cldr.Install.install_locale/2
    lib/cldr/config.ex:398: Cldr.Config.get_locale/1
    lib/cldr.ex:189: anonymous fn/1 in :elixir_compiler_13.__MODULE__/1
    (elixir) lib/enum.ex:645: Enum."-each/2-lists^foreach/1-0-"/2
    (elixir) lib/enum.ex:645: Enum.each/2
    lib/cldr.ex:188: (module)

Reason

Under these circumstances, compile-time code in the Cldr module ends up calling itself with Cldr.all_locales/0, which doesn’t exist yet.

Cldr.Number.to_string 123, currency: :AUD should default for format :currency but isn't

Cldr.Number.to_string 123, currency: :AUD should expand to Cldr.Number.to_string 123, format: :currency, currency: :AUD but isn't, its defaulting to :standard.