go-playground / universal-translator

:speech_balloon: i18n Translator for Go/Golang using CLDR data + pluralization rules

universal-translator

Universal Translator is an i18n Translator for Go/Golang using CLDR data + pluralization rules

Why another i18n library?

Because none of the plural rules seem to be correct out there, including the previous implementation of this package, so I took it upon myself to create locales for everyone to use; this package is a thin wrapper around locales in order to store and translate text for use in your applications.

Features

• Rules generated from the CLDR data, v36.0.1
• Contains Cardinal, Ordinal and Range Plural Rules
• Contains Month, Weekday and Timezone translations built in
• Contains Date & Time formatting functions
• Contains Number, Currency, Accounting and Percent formatting functions
• Supports the "Gregorian" calendar only ( my time isn't unlimited, had to draw the line somewhere )
• Support loading translations from files
• Exporting translations to file(s), mainly for getting them professionally translated
• Code Generation for translation files -> Go code.. i.e. after it has been professionally translated
• Tests for all languages, I need help with this, please see here

Installation

Use go get

go get github.com/go-playground/universal-translator

Usage & Documentation

Please see https://godoc.org/github.com/go-playground/universal-translator for usage docs

Examples:

• Basic
• Full - no files
• Full - with files

File formatting

All types, Plain substitution, Cardinal, Ordinal and Range translations can all be contained within the same file(s); they are only separated for easy viewing.

Examples:

• Formats

Basic Makeup

NOTE: not all fields are needed for all translation types, see examples

{
    "locale": "en",
    "key": "days-left",
    "trans": "You have {0} day left.",
    "type": "Cardinal",
    "rule": "One",
    "override": false
}

Field	Description
locale	The locale for which the translation is for.
key	The translation key that will be used to store and lookup each translation; normally it is a string or integer.
trans	The actual translation text.
type	The type of translation Cardinal, Ordinal, Range or "" for a plain substitution(not required to be defined if plain used)
rule	The plural rule for which the translation is for eg. One, Two, Few, Many or Other.(not required to be defined if plain used)
override	If you wish to override an existing translation that has already been registered, set this to 'true'. 99% of the time there is no need to define it.

Help With Tests

To anyone interesting in helping or contributing, I sure could use some help creating tests for each language. Please see issue here for details.

License

Distributed under MIT License, please see license file in code for more details.

universal-translator's Issues

Is this a bug?

1. New a UniversalTranslator with nil fallback, that work fine; ut := New(nil)
2. call ut.AddTranslator to add new translator but t.fallback is nil so will panic. source code

   universal-translator/universal_translator.go

   Line 85 in 7120149

   if t.fallback.Locale() == translator.Locale() {

Whether to add panic handling ?

go test all fails for all dependent projects

What did you do?

I run go test all for my project which has universal-translator as an indirect dependency and it uses go modules.

What did you expect to see?

All tests passed.

What did you see instead?

> go test all
[...]
ok      github.com/go-playground/locales/nl     (cached)
testdata/translations stat testdata/translations: no such file or directory true
--- FAIL: TestExportImportBasic (0.00s)
    import_export_test.go:106: Expected '<nil>' Got 'mkdir testdata/translations: permission denied'
testdata/translations stat testdata/translations: no such file or directory true
--- FAIL: TestExportImportCardinal (0.00s)
    import_export_test.go:235: Expected '<nil>' Got 'mkdir testdata/translations: permission denied'
testdata/translations stat testdata/translations: no such file or directory true
--- FAIL: TestExportImportOrdinal (0.00s)
    import_export_test.go:372: Expected '<nil>' Got 'mkdir testdata/translations: permission denied'
testdata/translations stat testdata/translations: no such file or directory true
--- FAIL: TestExportImportRange (0.00s)
    import_export_test.go:535: Expected '<nil>' Got 'mkdir testdata/translations: permission denied'
--- FAIL: TestBadExport (0.00s)
    import_export_test.go:771: Expected '<nil>' Got 'mkdir testdata/readonly: permission denied'
FAIL
FAIL    github.com/go-playground/universal-translator   0.042s
ok      go/ast  (cached)
[...]
FAIL

How to reproduce?

The slow but exact way: run docker build . with the following Dockerfile:

FROM golang:1.14
WORKDIR /go/src

RUN printf '\
package main \n\
import "github.com/go-playground/universal-translator" \n\
var _ *ut.UniversalTranslator \n\
func main() {} \n\
' > main.go

RUN go mod init example.com/m \
    && go mod tidy

RUN go test all

The fast and hacky way:

cd $GOPATH/src/github.com/go-playground/universal-translator
ls -ld ./testdata/ # so that you remember the mods
chmod 555 ./testdata/
go test ./...

What is the root cause?

1. The failing tests create temporary directories in the projects testdata dir.
2. When running go test all the Go tool changes directory to $GOPATH/pkg/mod/pkg/mod/github.com/go-playground/[email protected] to run the tests.
3. Directories under $GOPATH/pkg/mod/pkg/mod/` are readonly.
   => The tests fail.

Datatime and currency formats

Am doing language translation.

It's such a huge area .

If the text to be translated has datatime or currency ( money ) in it what to do ?

I was thinking that the translated text must preserver any datetime and / or currency values in it is some sort of meta description.
That was at runtime when using the translated language you can do a final parse for any of this meta data and also convert datatime and / currency before returning it back to the GUI .

That's my current idea. Any other approaches ?

This lib seems not to support this use case.
But I saw some libs using getText for golang which do support it.

Cannot contain "{" and "}" in the parameter text when using func (t *translator) Add(key interface{}, text string, override bool) error

Cannot contain "{" and "}" in the message when using method func (t *translator) Add(key interface{}, text string, override bool) error

I want to custom a password validitor with tag valid-password, which requires password can contains a-zA-Z0-9 and （~!@#$%^&*()[]{}|/?,.<>） only, and I'm using the universal-translator to custom the error message when validate fails. The point is when the customed error message contains "{" and "}",there will be an error!

File format for interchange

Sending files for translation to upstream service providers is a common task. Right now, all my translations are directly in my Go code. I will soon need to send it out.. is there any format in which we could pass all the semantics of universal-translator, back and forth ?

This site lists a bunch of them (in which go-i18n figures!):

https://phraseapp.com/docs/guides/formats/

Perhaps we can pick one that is used in the industry ?

Request for Help creating tests for ALL locales

UPDATE: universal-translator's locales + rules have been split into another repo locales

please see here for helping with locale tests.

Project Focus?

Hey! So I was checking out the different options in the golang ecosystem for translation, and it looks like it really comes down to go-i18n and this project. I just wanted to better understand the tradeoffs and benefits of the two projects. It appears as though go-i18n may be easier to get started with, but the thing that keeps me coming back to universal-translator is its integration with validator(which I'm a big fan of). The thing that's hardest for me to understand with validator, is how to go from a list of FieldErrors to an actual list of messages that can be displayed in a template. Any thoughts on this would be very helpful.

In the end, my biggest outstanding questions would be, what are the major benefits of going with this project as opposed to others over the long run? Is this project still actively maintained?

Translation Parameters order

Hello,

We have the below case:
Translations:
EN: Please enter at least {{currency}} {{amount}}:
AR: Please enter at least {{amount}} {{currency}}:

Following the docs our translation value look like this
EN: Please enter at least {0} {1}:
AR: Please enter at least {0} {1}:

At the moment in order to achieve this we have to manually change the order of the parameters in the language is AR.
Is there any other way to have parameters in the translation without order being important?

Web GUI for Translators

I knwo google sheets is so so basic, but i saw another project doing it and its not half bad.

see here:
https://github.com/jgraph/draw.io/wiki/Translations

• This is actually a great way to do it, mostly because Translators DONT have to screw around with git.
• I can probably write a little tool to keep master in sync with the spreadsheet.
  • Does a Diff between what the code EN trans are and whats in the google sheet.
    • Red for removed from Git
    • Green for added from Git.
    • Translators can then sort by the Red, Gree and their language to see the exact work to be done.

I think there are some decent tools for talking to google sheets from golang.
https://developers.google.com/sheets/quickstart/go

• It looks capable enough..

example full with files failing

cd /Users/apple/workspace/go/src/github.com/go-playground/universal-translator/examples/full-with-files && go run main.go
go: finding github.com/go-playground/pure/examples/middleware/logging-recovery latest
go: finding github.com/go-playground/pure/examples/middleware latest
go: finding github.com/go-playground/pure v4.1.1+incompatible
go: finding github.com/go-playground/pure/examples latest
go: downloading github.com/go-playground/pure v4.1.1+incompatible
go: extracting github.com/go-playground/pure v4.1.1+incompatible
go: finding github.com/go-playground/form v3.1.4+incompatible
go: downloading github.com/go-playground/form v3.1.4+incompatible
go: extracting github.com/go-playground/form v3.1.4+incompatible
build command-line-arguments: cannot load github.com/go-playground/pure/examples/middleware/logging-recovery: cannot find module providing package github.com/go-playground/pure/examples/middleware/logging-recovery

fatal error: concurrent map read and map write

github.com/go-playground/universal-translator.(*translator).Add(0xc000a66cc0, 0x190e280, 0xc0006aff10, 0x1b22580, 0x12, 0x0, 0xc000a66f00, 0xc0006fa288)

Readme States Its the Latest CLDR version, v29?

I can't find any hard statement saying this is a stable release, but it looks like it is to me
http://cldr.unicode.org/index/downloads/cldr-30

Is there any limitation when using Import?

Hello,

Go version: 1.20
Universal Translator version: v0.18.0

I would like to asking if there are any limitation when importing translation using JSON format.
In before, when using this library, I always got the translation I meant. But in couple of days, when I add new key in JSON format, the translator return "unknown translation"

Here is the snippet of my code:

import (
  "github.com/go-playground/locales"
  "github.com/go-playground/locales/en"
  "github.com/go-playground/locales/id"
   ut "github.com/go-playground/universal-translator"
)

func InitTranslator() *ut.UniversalTranslator{
   enLocales := en.New()
   idLocales := id.New()

   supportedLocales := []locales.Translator{
     enLocales, idLocales,
   }

   newTranslator := ut.New(enLocales, supportedLocales...)

   newTranslator.Import(ut.FormatJSON, fmt.Sprintf("%s/%s", pwd, "translation"))
   if err = newTranslator.VerifyTranslations(); err != nil {
		log.Fatal(context.Background(), err)
   }
}

Here is my translation file:

   // ... there already a lot of key and value above these two
   {
     "key": "morning",
     "locales": "en",
     "trans": "Morning"
   },
   {
     "key": "morning",
     "locales": "id",
     "trans": "Pagi"
   },

But when I call with:

  trans, found := newTranslator.GetTranslator("id")
	if !found {
           log.Fatal("not found translation")
	}

	return trans.T("morning")

It will return: Unknown Translation. So, how do I fix it?

Plural Types do not match CLDR Plural Rules, yields gramatically incorrect for many languages.

Just a quick view of some of the rules in plurals.go
It has BN, ES, SW, PT, all as type "2A"
However, they have three different rules...
as I write this, it seems that most of the rules are broken if they are dependent on visible trailing zeros, fractions, or integer-portion rules (that ignore fractional portions)

This ticket is to just bring it to your attention.
Hope you can regenerate the data this in a more automated way

But your code only deals with the pre-2012 plural rules for integer values.
And is not compatible with CLDR-24 (current CLDR-29) / ICU 52+ (current ICU 57) rules, which take into account that plurals are dependent on absolute value, integer portion, trailing portion, and whether there are displayed trailing zero's or not.

It doesn't seem there is currently any means to pass in a formatted number, or to provide a Number Formatting rule to be able to even calculate the correct plural form, as the form returned is dependent on the displayed value (with or without trailing zero's for instance)

The English rule is wrong to boot XD
EN: one = (i = 1 and v = 0)
( 1 book, 1.0 books )
where you have it implemented as one = (n=1)

BN: one = (n=1 || i=0)
ES: one = (n=1)
AST, CA, SW: one = (n=1 && v=0)
so, a number formatted as "1" returns ONE, but "1.0" would return OTHER
PT: one = (n=0 || n=1)

THe Croation logic is way off:
HR:
one = (v=0 && i%10 = 1 && i%100 != 11) || (f % 10 = 1 && f % 100 != 11)
few = (v=0 && i % 10 in 2..4 and i % 100 !in 12..14) or ....
one: 1, 1.1, 2.1
few: 2, 2.2, 2.3
other: 1.0

And can't be merged with Russian which is
one: = (v=0 and i % 10 = 1 and i % 100 != 11)
few = (v=0 and i % 10 in 2.4 and i % 100 ! in 2..4)

Fallback Language Support

CLDR has fallback languages for cases in which a localization is unavailable. I would like to add support for this. Any thoughts or recommendations on implementation?

I would create LocaleResolver which takes in a locale ID, and if the ID is support, returns the ID, otherwise it runs through the list of fallback locales, and returns the first one that is supported. I'd probably need to specify a configurable default language value as well.

http://cldr.unicode.org/development/development-process/design-proposals/transform-fallback
http://cldr.unicode.org/development/development-process/design-proposals/languagedistance