grosser / i18n_data

Ruby: country/language names and 2-letter-code pairs, in 85 languages, for country/language i18n

• Present users coutries/languages in their language
• Convert a country/language-name to its 2-letter-code
• List of 2-letter-code/name pairs for all countries/languages in all languages

Translations

Through pkg-isocodes:

• 185 Language codes (iso 639 - 2 letter) in 68 Languages
• 246 Country codes (iso 3166 - 2 letter) in 86 Languages
• Country specific codes e.g. zh_TW are also available, have a look at the isocodes website for all options

Install

gem install i18n_data

Usage

require 'i18n_data'
...
I18nData.languages        # {"DE"=>"German",...}
I18nData.languages('DE')  # {"DE"=>"Deutsch",...}
I18nData.languages('FR')  # {"DE"=>"Allemand",...}
...

I18nData.countries        # {"DE"=>"Germany",...}
I18nData.countries('DE')  # {"DE"=>"Deutschland",...}
...

I18nData.language_code('German')       # DE
I18nData.language_code('Deutsch')      # DE
I18nData.language_code('Allemand')     # DE
..

I18nData.country_code('Germany')       # DE
I18nData.country_code('Deutschland')   # DE
..

Data Providers

• FileDataProvider: FAST (default) (loading data from cache-files)
• LiveDataProvider: SLOW (fetching up-to-date data from svn repos)

Development

• update FileDataProvider caches after each code-change to make changes available to users rake write_cache_for_file_data_provider
• FileDataProvider tests might fail if caches are not updates

Alphabetical localized sorting

If you would like to have the countries list sorted alphabetically in different languages there is a gem called sort_alphabetical for that.

TODO

• include other language/country code formats (3-letter codes...) ?
• parse list of files on isocodes for write_cache instead of hardcoding country-specific ones
• rake task that checks that the readme is updated with language/country counters

Authors

Contributors

• Barry Allard
• Richard Doe

Michael Grosser
[email protected]
License: MIT

i18n_data's Issues

Find out certain languageName...

Hi,
I am not sure, if this would work - I need the languageName (i.e. "German"), when I put in "de".

Please let me know how I can achieve this - I would like to use this gem instead of writing a mapping on my own...

Thank you

3-letter language & country codes

I would like to implement this todo from the README:

include other language/country code formats (3-letter codes...) ?

In the process, I would also like to propose some refactoring. What I have in mind here is:

1. Use 3-letter language and country codes in filenames and for lookup keys in data caches, because all languages have a 3-letter code, but not all have a 2-letter code.
2. Use codes in the case they are specified in the relevant ISO standard, i.e. lower-case for language codes, upper-case for country codes.
3. Change the storage format of the cached data to JSON, e.g.
   Languages in English:

   {
     "eng": "English",
     "fra": "French",
     "spa": "Spanish; Castilian"
   }

4. Rename the cache files thus:
   • Countries: /cache/countries/eng.json, /cache/countries/fra.json, etc
   • Languages: /cache/languages/eng.json, /cache/languages/fra.json, etc
5. Make the live data provider add to the cache two JSON dictionaries of alternate codes (2-letter/bibliographic/numeric) mapping them to their 3-letter equivalent, e.g.
   Language code dictionary:

   {
     "en": "eng",
     "es": "spa",
     "fr": "fra",
     "chi": "zho"
   }

   Country code dictionary:

   {
     "GMB": "GM",
     "887": "YEM"
   }

6. When performing lookups by code, if the supplied code is unknown, i.e. has no matching cache file, lookup a 3-letter equivalent in the relevant dictionary and use that.

Before I get started working on a PR for the above, is there general support for some/any/all of this proposal?

External encoding of files should be defined in code

Running LANG=C rspec spec results in failures such as:

 21) I18nData using I18nData::FileDataProvider .countries translated has default for languages that only have subtypes
     Failure/Error: I18nData.countries('ZH')['DK'].should == '丹麦'
     ArgumentError:
       invalid byte sequence in US-ASCII
     # /builddir/build/BUILD/rubygem-i18n_data-0.4.0/usr/share/gems/gems/i18n_data-0.4.0/lib/i18n_data/file_data_provider.rb:32:in `split'
     # /builddir/build/BUILD/rubygem-i18n_data-0.4.0/usr/share/gems/gems/i18n_data-0.4.0/lib/i18n_data/file_data_provider.rb:32:in `block in read_from_file'
     # /builddir/build/BUILD/rubygem-i18n_data-0.4.0/usr/share/gems/gems/i18n_data-0.4.0/lib/i18n_data/file_data_provider.rb:31:in `each'
     # /builddir/build/BUILD/rubygem-i18n_data-0.4.0/usr/share/gems/gems/i18n_data-0.4.0/lib/i18n_data/file_data_provider.rb:31:in `read_from_file'
     # /builddir/build/BUILD/rubygem-i18n_data-0.4.0/usr/share/gems/gems/i18n_data-0.4.0/lib/i18n_data/file_data_provider.rb:7:in `codes'
     # /builddir/build/BUILD/rubygem-i18n_data-0.4.0/usr/share/gems/gems/i18n_data-0.4.0/lib/i18n_data.rb:19:in `block in countries'
     # /builddir/build/BUILD/rubygem-i18n_data-0.4.0/usr/share/gems/gems/i18n_data-0.4.0/lib/i18n_data.rb:52:in `fetch'
     # /builddir/build/BUILD/rubygem-i18n_data-0.4.0/usr/share/gems/gems/i18n_data-0.4.0/lib/i18n_data.rb:18:in `countries'
     # /builddir/build/BUILD/rubygem-i18n_data-0.4.0/usr/share/gems/gems/i18n_data-0.4.0/spec/i18n_data_spec.rb:122:in `block (6 levels) in <top (required)>'
     # /usr/share/gems/gems/rspec-core-2.13.1/lib/rspec/core/example.rb:114:in `instance_eval'
     # /usr/share/gems/gems/rspec-core-2.13.1/lib/rspec/core/example.rb:114:in `block in run'
     # /usr/share/gems/gems/rspec-core-2.13.1/lib/rspec/core/example.rb:254:in `with_around_each_hooks'
     # /usr/share/gems/gems/rspec-core-2.13.1/lib/rspec/core/example.rb:111:in `run'
     # /usr/share/gems/gems/rspec-core-2.13.1/lib/rspec/core/example_group.rb:390:in `block in run_examples'
     # /usr/share/gems/gems/rspec-core-2.13.1/lib/rspec/core/example_group.rb:386:in `map'
     # /usr/share/gems/gems/rspec-core-2.13.1/lib/rspec/core/example_group.rb:386:in `run_examples'
     # /usr/share/gems/gems/rspec-core-2.13.1/lib/rspec/core/example_group.rb:371:in `run'
     # /usr/share/gems/gems/rspec-core-2.13.1/lib/rspec/core/example_group.rb:372:in `block in run'
     # /usr/share/gems/gems/rspec-core-2.13.1/lib/rspec/core/example_group.rb:372:in `map'
     # /usr/share/gems/gems/rspec-core-2.13.1/lib/rspec/core/example_group.rb:372:in `run'
     # /usr/share/gems/gems/rspec-core-2.13.1/lib/rspec/core/example_group.rb:372:in `block in run'
     # /usr/share/gems/gems/rspec-core-2.13.1/lib/rspec/core/example_group.rb:372:in `map'
     # /usr/share/gems/gems/rspec-core-2.13.1/lib/rspec/core/example_group.rb:372:in `run'
     # /usr/share/gems/gems/rspec-core-2.13.1/lib/rspec/core/example_group.rb:372:in `block in run'
     # /usr/share/gems/gems/rspec-core-2.13.1/lib/rspec/core/example_group.rb:372:in `map'
     # /usr/share/gems/gems/rspec-core-2.13.1/lib/rspec/core/example_group.rb:372:in `run'
     # /usr/share/gems/gems/rspec-core-2.13.1/lib/rspec/core/command_line.rb:28:in `block (2 levels) in run'
     # /usr/share/gems/gems/rspec-core-2.13.1/lib/rspec/core/command_line.rb:28:in `map'
     # /usr/share/gems/gems/rspec-core-2.13.1/lib/rspec/core/command_line.rb:28:in `block in run'
     # /usr/share/gems/gems/rspec-core-2.13.1/lib/rspec/core/reporter.rb:34:in `report'
     # /usr/share/gems/gems/rspec-core-2.13.1/lib/rspec/core/command_line.rb:25:in `run'
     # /usr/share/gems/gems/rspec-core-2.13.1/lib/rspec/core/runner.rb:80:in `run'
     # /usr/share/gems/gems/rspec-core-2.13.1/lib/rspec/core/runner.rb:17:in `block in autorun'

Running LANG=en_US.utf8 rspec spec works just fine => external encoding for files should be defined in source code to work correctly no matter what system encoding is.

ES;;Espagnol missing from languages-fr

.rb:9: syntax error, unexpected keyword_end, expecting $end

!/usr/bin/ruby

require 'tmail'
require 'net/smtp'
require 'net/pop'

sub ng

end

puts "Hello World"

when I run this script, I get above error. The whole context is:

/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require': iconv will be deprecated in the future, use String#encode instead.
Hello World

Is there a patch, or how do I fix this. otherwise, I might need to use perl to do the problem solving.

get country_code by ZH_TW

I18nData.country_code('奧蘭群島')
get nil
expect AX

Ruby 1.9.3 fails to run the test suite

$ RUN_CODE_RUN=1 rspec spec/i18n_data_spec.rb
/usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/configuration.rb:698:in `load': /builddir/build/BUILD/rubygem-i18n_data-0.2.7/usr/share/gems/gems/i18n_data-0.2.7/spec/i18n_data_spec.rb:59: invalid multibyte char (US-ASCII) (SyntaxError)
/builddir/build/BUILD/rubygem-i18n_data-0.2.7/usr/share/gems/gems/i18n_data-0.2.7/spec/i18n_data_spec.rb:59: invalid multibyte char (US-ASCII)
/builddir/build/BUILD/rubygem-i18n_data-0.2.7/usr/share/gems/gems/i18n_data-0.2.7/spec/i18n_data_spec.rb:59: syntax error, unexpected $end, expecting keyword_end
            I18nData.languages('DE')['DA'].should == 'Dänisch'
                                                         ^
    from /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/configuration.rb:698:in `block in load_spec_files'
    from /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/configuration.rb:698:in `map'
    from /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/configuration.rb:698:in `load_spec_files'
    from /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/command_line.rb:22:in `run'
    from /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/runner.rb:80:in `run_in_process'
    from /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/runner.rb:69:in `run'
    from /usr/share/gems/gems/rspec-core-2.8.0/lib/rspec/core/runner.rb:10:in `block in autorun'

The patch [1] should fix it.

[1] https://gist.github.com/1670672

Ruby 1.9.2 support?

Is there any reason why we require Ruby >= 1.9.3

If there is no specific reason, can you please switch the ruby dependency to Ruby >= 1.9.2 ?

One of my apps is using Ruby 1.9.2 and its depending on countries gem v1.2.5. Which in turn wants i18n_data gem v0.7.0

Incorrect translation of "North Macedonia" to Serbian

Reported to countries/countries#703, but the data comes from this gem.

In Serbian (SR) and Macedonian (MK) cache files, "North Macedonia" is translated to "Нова Каледонија" which is "New Caledonia"

The correct translation should be "Северна Македонија", as per https://salsa.debian.org/iso-codes-team/iso-codes/-/blob/main/iso_3166-1/sr.po

counties-ZH_CN.txt line 32 波利维亚 玻利维亚

玻利维亚 is Correct translation

I18nData.language_code returns nil for known languages

Hi,

Running this:

> I18nData.languages.map { |code,name| I18nData.language_code(name) }
=> ["AA", "AB", "AF", "AK", "SQ", "AM", "AR", "AN", "HY", "AS", "AV", "AE", "AY", "AZ", "BA", "BM", "EU", "BE", "BN", "BH", "BI", "BS", "BR", "BG", "MY", nil, "CH", "CE", "ZH", nil, "CV", "KW", "CO", "CR", "CS", "DA", nil, nil, "DZ", "EN", "EO", "ET", "EE", "FO", "FJ", "FI", "FR", "FY", "FF", "KA", "DE", nil, "GA", "GL", "GV", "EL", "GN", "GU", nil, "HA", "HE", "HZ", "HI", "HO", "HR", "HU", "IG", "IS", "IO", nil, "IU", nil, "IA", "ID", "IK", "IT", "JV", "JA", nil, "KN", "KS", "KR", "KK", "KM", nil, "RW", nil, "KV", "KG", "KO", nil, "KU", "LO", "LA", "LV", nil, "LN", "LT", nil, "LU", "LG", "MK", "MH", "ML", "MI", "MR", "MS", "MG", "MT", nil, "MN", "NA", nil, nil, nil, "NG", "NE", nil, nil, "NO", nil, "OC", "OJ", "OR", "OM", nil, nil, "FA", "PI", "PL", "PT", nil, "QU", "RM", "RO", "RN", "RU", "SG", "SA", nil, "SK", "SL", "SE", "SM", "SN", "SD", "SO", "ST", nil, "SC", "SR", "SS", "SU", "SW", "SV", "TY", "TA", "TT", "TE", "TG", "TL", "TH", "BO", "TI", "TO", "TN", "TS", "TK", "TR", "TW", nil, "UK", "UR", "UZ", "VE", "VI", "VO", "CY", "WA", "WO", "XH", "YI", "YO", nil, "ZU"]

I would expect to get a language code for each known language? But each language containing multiple names, e.g. Spanish; Castilian, returns nil. I'm expecting to receive ES, which is the key of Spanish; Castilian when running I18nData.languages?

Add "Mandarin" & "Cantonese"

Hey guys, would be cool to have access to these languages as some of our users do not consider themselves as "Chinese" speakers. Thanks.

French countries (and maybe others) are not titleized

Hi! :)

It seems that French countries are not titleized when you do:

> I18nData.languages("FR")
=> {"AA"=>"afar", "AB"=>"abkhaze", "AF"=>"afrikaans", "AK"=>"akan", "SQ"=>"albanais", "AM"=>"amharique", "AR"=>"arabe", "AN"=>"aragonais", "HY"=>"arménien", "AS"=>"assamais", "AV"=>"avar", "AE"=>"avestique", "AY"=>"aymara", "AZ"=>"azéri", "BA"=>"bachkir", "BM"=>"bambara", "EU"=>"basque", "BE"=>"biélorusse", "BN"=>"bengali", "BH"=>"biharies, langues", "BI"=>"bichlamar", "BS"=>"bosniaque", "BR"=>"breton", "BG"=>"bulgare", "MY"=>"birman", "CA"=>"catalan", "CH"=>"chamorro", "CE"=>"tchétchène", "ZH"=>"chinois", "CU"=>"Church Slavic; Old Slavonic; Church Slavonic; Old Bulgarian; Old Church Slavonic", "CV"=>"tchouvache", "KW"=>"cornique", "CO"=>"corse", "CR"=>"cree", "CS"=>"tchèque", "DA"=>"danois", "DV"=>"maldivien", "NL"=>"néerlandais", "DZ"=>"dzongkha", "EN"=>"anglais", "EO"=>"espéranto", "ET"=>"estonien", "EE"=>"éwé", "FO"=>"féroïen", "FJ"=>"fidjien", "FI"=>"finnois", "FR"=>"français", "FY"=>"frison occidental", "FF"=>"peul", "KA"=>"géorgien", "DE"=>"allemand", "GD"=>"gaélique ; gaélique écossais", "GA"=>"irlandais", "GL"=>"galicien", "GV"=>"mannois ; manx", "EL"=>"grec moderne (après 1453)", "GN"=>"guarani", "GU"=>"gujarati", "HT"=>"haïtien ; créole haïtien", "HA"=>"haoussa", "HE"=>"hébreu", "HZ"=>"herero", "HI"=>"hindi", "HO"=>"hiri motu", "HR"=>"croate", "HU"=>"hongrois", "IG"=>"igbo", "IS"=>"islandais", "IO"=>"ido", "II"=>"yi de Sichuan", "IU"=>"inuktitut", "IE"=>"interlingue", "IA"=>"interlingua (langue auxiliaire internationale)", "ID"=>"indonésien", "IK"=>"inupiaq", "IT"=>"italien", "JV"=>"javanais", "JA"=>"japonais", "KL"=>"groenlandais", "KN"=>"kannada", "KS"=>"kashmiri", "KR"=>"kanouri", "KK"=>"kazakh", "KM"=>"khmer central", "KI"=>"kikuyu", "RW"=>"rwanda", "KY"=>"kirghiz", "KV"=>"komi", "KG"=>"kongo", "KO"=>"coréen", "KJ"=>"kuanyama", "KU"=>"kurde", "LO"=>"lao", "LA"=>"latin", "LV"=>"letton", "LI"=>"limbourgeois", "LN"=>"lingala", "LT"=>"lituanien", "LB"=>"luxembourgeois", "LU"=>"luba-katanga", "LG"=>"ganda", "MK"=>"macédonien", "MH"=>"marshall", "ML"=>"malayalam", "MI"=>"maori", "MR"=>"marathe", "MS"=>"malais", "MG"=>"malgache", "MT"=>"maltais", "MO"=>"moldave", "MN"=>"mongol", "NA"=>"nauruan", "NV"=>"navaho", "NR"=>"ndébélé du Sud", "ND"=>"ndébélé du Nord", "NG"=>"ndonga", "NE"=>"népalais", "NN"=>"norvégien nynorsk", "NB"=>"norvégien Bokmål", "NO"=>"norvégien", "NY"=>"nyanja", "OC"=>"occitan (après 1500)", "OJ"=>"ojibwa", "OR"=>"oriya", "OM"=>"galla", "OS"=>"ossète", "PA"=>"pendjabi", "FA"=>"persan", "PI"=>"pali", "PL"=>"polonais", "PT"=>"portugais", "PS"=>"pachto", "QU"=>"quechua", "RM"=>"romanche", "RO"=>"roumain", "RN"=>"rundi", "RU"=>"russe", "SG"=>"sango", "SA"=>"sanskrit", "SI"=>"singhalais", "SK"=>"slovaque", "SL"=>"slovène", "SE"=>"sami du Nord", "SM"=>"samoan", "SN"=>"shona", "SD"=>"sindhi", "SO"=>"somali", "ST"=>"sotho du Sud", "ES"=>"castillan", "SC"=>"sarde", "SR"=>"serbe", "SS"=>"swati", "SU"=>"soundanais", "SW"=>"swahili", "SV"=>"suédois", "TY"=>"tahitien", "TA"=>"tamoul", "TT"=>"tatar", "TE"=>"télougou", "TG"=>"tadjik", "TL"=>"tagalog", "TH"=>"thaï", "BO"=>"tibétain", "TI"=>"tigrigna", "TO"=>"tongan (Îles Tonga)", "TN"=>"tswana", "TS"=>"tsonga", "TK"=>"turkmène", "TR"=>"turc", "TW"=>"twi", "UG"=>"ouïgour", "UK"=>"ukrainien", "UR"=>"ourdou", "UZ"=>"ouszbek", "VE"=>"venda", "VI"=>"vietnamien", "VO"=>"volapük", "CY"=>"gallois", "WA"=>"wallon", "WO"=>"wolof", "XH"=>"xhosa", "YI"=>"yiddish", "YO"=>"yoruba", "ZA"=>"zhuang", "ZU"=>"zoulou"}

And so when you want to get the language code from the country you have to downcase it, like so:

> I18nData.language_code("japonais")
=> "JA"
# But in english: 
> I18nData.language_code("japanese")
=> nil
> I18nData.language_code("Japanese")
=> "JA"

Otherwise it returns nil. But for other languages like English, it works when the language is titleized... This bug may also be for other languages.

Could it be fixed for French?
Thanks :)

Kulgar.

Capitalization of languages in Norwegian and Swedish

The capitalization of languages in Norwegian seems inconsistent. For example: NO;;Norsk is capitalized, but NE;;nepalsk is not. According to e.g. https://help.unbabel.com/hc/en-us/articles/360034571294-Language-Guidelines-Norwegian#33capitalization, language names should NOT be capitalized in Norwegian.

For Swedish, it looks like all the language names are currently capitalized (https://github.com/grosser/i18n_data/blob/master/cache/file_data_provider/languages-SV.txt). However, according to e.g. https://blogs.transparent.com/swedish/use-of-capital-letters/, this is not correct.

Would you consider adjusting the use of capitalization for Norwegian and Swedish?

languages-KA.txt

Hi

I can see countries-KA.txt but no languages-KA.txt
could it be possible to add it ?

thanks

Incorrect Translations

So thank you for this great gem. We have recently swapped out the countries gem to leverage the i18n_data and we have found some issues, that are rooted in some bad translations. What is the best ways to get these corrected? As the debian repo's aren't exactly "approachable".

See countries/countries#243 (comment)

Add for United States !!!!!!!!

client_language = I18nData.language_code('United States')
=> nil

Does not support some languages

My code is crashing because it returns nil, wouldn't it be better to throw an empty string to prevent crashes?

Ability to sort country names using locales

It would be great if there was a way to sort country names differently based on locales (something in databases is done using COLLATE - http://www.postgresql.org/docs/9.4/static/collation.html). This way we can present properly sorted country names to the user.

Just an idea.

NoTranslationAvailable -- countries-ZH_CN

When I try to retrieve all countries in Simplified Chinese I get the following error message:

I18nData.countries('ZH')
>> NoTranslationAvailable -- countries-ZH_CN

I think this is caused by the I18nData#normal_to_region_code method:

def normal_to_region_code(normal)
  {
     "ZH" => "ZH_CN",
     "BN" => "BN_IN",
  }[normal] || normal
end

Whereas the file with the translations is called i18n_data-0.3.2/cache/file_data_provider/countries-zh_CN.txt

nil instead of raise

Hi grosser,
I needed a list of languages translated in their own language and I used your gem. I did this helper.

def languages
  languages = {}

  I18nData.languages.each do |code, language|
    begin
      languages.merge!(I18nData.languages(code)[code] => code.downcase)
    rescue Exception => e
      languages.merge!(I18nData.languages('EN')[code] => code.downcase)
    end
  end
  languages
end

It would better not to work with rescue but there it is this line that uses raises instead of return nil:

i18n_data/lib/i18n_data/file_data_provider.rb

Line 8 in b9d0db3

raise NoTranslationAvailable, "#{type}-#{language_code}"

Wouldn't be better to use nil?

Thank you for your gem

Incorrect Translations

The data update in #56 applied incorrect translations to a number of country files, such as:

i18n_data/cache/file_data_provider/countries-AN.txt

Line 182 in f76b23d

KP;;Nueva Caledonia

and

i18n_data/cache/file_data_provider/countries-AN.txt

Line 122 in f76b23d

KN;;Sant Cristofo y Nieus

Where KP is being translated to "New Caledonia" and KR to "South Africa". In some cases also KR as "North Macedonia"

It seems to be the potencial issue I mentioned in #53 (comment)

In a quick review this affected country files for: AF, AN, possibly AK, AZ, BR, BS, CA, CS, CY, DA, EU, FI, FO, FY, GA, GL, GN, GV, HA, HT, HU, IA, IO, IT, KI, KW, LT, LV, MS, MT, NA, NN, PL, PT, RO, RW, SK, SL, SO, SQ, SW, TK, TL, UZ, VE, VI, WA, WO, XH, YO, ZU

I didn't check the country files that use non-latin alphabets