nicksnyder / go-i18n Goto Github PK
View Code? Open in Web Editor NEWTranslate your Go program into multiple languages.
License: MIT License
Translate your Go program into multiple languages.
License: MIT License
Hello
I am looking for a way to build translation into a web-based project. I came across this and it really looks impressive. However, I have no idea where to start; I am puzzled how the whole framework is to be integrated in a project. It would be great if there was a minimum sample project somewhere, showing how this is done ?
Hi Nick, Found out today my vendored version is not compatible with the new version. Specifically this function of mine does not work anymore because the locale package is gone.
My Call to AddTranslation:
i18n.AddTranslation(locale.MustNew(userLocale), tran)
What is the new mechanism for this. I spent a handful of minutes looking but found nothing obvious.
Thanks
// LoadJSON takes a json document of translations and manually
// loads them into the system
func LoadJSON(userLocale string, translationDocument string) error {
tracelog.Startedf("localize", "LoadJSON", "UserLocale[%s] Length[%d]", userLocale, len(translationDocument))
tranDocuments := []map[string]interface{}{}
err := json.Unmarshal([]byte(translationDocument), &tranDocuments)
if err != nil {
tracelog.CompletedErrorf(err, "localize", "LoadJSON", "**************>")
return err
}
for _, tranDocument := range tranDocuments {
tran, err := translation.NewTranslation(tranDocument)
if err != nil {
tracelog.CompletedError(err, "localize", "LoadJSON")
return err
}
i18n.AddTranslation(locale.MustNew(userLocale), tran)
}
tracelog.Completed("localize", "LoadJSON")
return nil
}
I was wondering if we can support ARB format, so i can use google translate toolkit.
Here is a good summary:
https://github.com/dart-lang/intl#extracting-and-using-translated-messages
Background info:
At the moment i use Dart for frontend and golang for backend.
Could you make a more thorough example for V2?
I'm not sure why NewBundle accepts language.English
when later Accept-Language
is used as an input. What is a bundle, for that matter?
RegisterUnmarshalFunc
is set to toml
. Where is there any TOML used?
I think it could be very useful to be able to add custom functions to the internally used template engine.
The Template method that allows it is https://golang.org/pkg/text/template/#Template.Funcs .
I'm not sure where it would the best place would be to implement this. Maybe adding another Field Funcs
of the type template.funcMap
to the Message
struct would be a good idea?
I have started to prototype what v2 of this library might look like.
The overall goals of v2 are to
My development plan is to work on this in a branch until I am happy with the results. Then, I will merge this into master under a subfolder called v2-beta
and tag a 1.x.0
release. Projects can then begin to try out the new API and provide feedback. APIs under v2-beta
subfolder will be subject to change pending community feedback. Once I am happy with the results, I will make a final 1.x.0
release that contains the v2 api under the v2
subfolder. Immediately following that, I will delete the v1 package files, move v2-beta to the root of the repository, and tag 2.0.0
.
I do not have an estimate for when this work will be complete as I am working on it in spare time here and there.
If you have general feedback about this process it can be shared here.
If there are other things you would want to see in a world where breaking API changes are possible, please create a new issue and I will consider it.
I want to use Sourcegraph code search and code review with go-i18n. A project maintainer needs to enable it to set up a webhook so the code is up-to-date there.
Could you please enable go-i18n on @sourcegraph by going to https://sourcegraph.com/github.com/nicksnyder/go-i18n and clicking on Settings? (It should only take 15 seconds.)
Thank you!
Is the support for locale based number and currency handling in your timeline?
Please forgive my question as I'm somewhat new to Golang.
A recent change in go-i18n/i18n/bundle/bundle.go to import gopkg.in/yaml.v2 started causing my build to fail with this error....
src/github.com/nicksnyder/go-i18n/i18n/bundle/bundle.go:7:2: cannot find package "gopkg.in/yaml.v2"
I use "git clone" to pull down all depend projects so I'm not sure if I can only get the package above via "go get" or if there should be a github repo with the changes that I can clone from.
As discussed in this thread:
We use go-i18n in Hugo and it works great, but the number one related question we get on the support forum comes from people having a hard time getting their language files in YAML right (indentation issues, mostly).
So it would be nice if go-i18n could support the simpler TOML format and also optionally support a slightly flatter data format, so we could write:
[[d_days]]
one = "{{.Count}} day"
other = "{{.Count}} days"
[[string_id_2]]
other = "Some other translation"
/cc @moorereason @BoGeM
A flatter data file structure would be beneficial. Implementation should maintain support of existing format.
{
"d_days": {
"one": "{{.Count}} day",
"other": "{{.Count}} days"
},
"string_id_2": {
"other": "Some other translation"
}
}
d_days:
one: "{{.Count}} day"
other: "{{.Count}} days"
string_id_2:
other: Some other translation
[d_days]
one = "{{.Count}} day"
other = "{{.Count}} days"
[string_id_2]
other = "Some other translation"
This replaces the rejected #81
While @nicksnyder is right about thread safety being the client's responsibility, the global pluralSpecs
makes it harder than it should.
And then having RegisterPluralSpec
mutate that state.
In Hugo we run most tests with t.Parallel()
. And while this may report some data races we may not realistically see in real use scenarios, it is well worth it.
In this case, it reports a data race in pluralSpecs
when using the RegisterPluralSpec
. We can probably fix that by adding some global locks.
But these globals also has another unfortunate side-effect:
We cannot run multiple go-i18n
configurations side-by-side. And this has nothing to do with concurrency.
So, it would be much better if you could provide a constructor func that takes the options needed (plural specs etc.)
from @bubenkoff #87 (comment)
@nicksnyder extract
doesn't seem to work.
ran it with json
format and got empty toml file.
goi18n extract -sourceLanguage en-us -format json -outdir locales objects/ service/
got
active.en-US.toml
which is empty
Let's say I have two files as follows:
// en-us.json
[
{
"id": "sample_test",
"translation": {
"other": "{{.Type}} by {{.Owner}}",
"zero": "Nobody here but us chickens!"
}
}
]
// ru-ru.json
[
{
"id": "sample_test",
"translation": {
"other": "{{.Type}} от {{.Owner}}",
"zero": "Здесь никого, кроме нас, цыплят!"
}
}
]
After I get a merge files the following lines:
// en-us.all.json
[
{
"id": "sample_test",
"translation": {
"one": "",
"other": "{{.Type}} by {{.Owner}}"
}
}
]
// ru-ru.all.json
[
{
"id": "sample_test",
"translation": {
"few": "",
"many": "",
"one": "",
"other": "{{.Type}} от {{.Owner}}"
}
}
]
So... WTF?
I want to use the translation only in two cases:
I do not want to duplicate the line, for example, for a value of 1, this is an extra information trash and additional fixes.
I'd love to be able to just write
program_greeting: "Hello world"
instead of
program_greeting:
other: "Hello world"
While I understand the use case of the latter, couldn't it just support a direct string value and default to the "other" mechanic?
Create tag for 1.0 and start using versioned import urls everywhere.
http://gopkg.in/v1/docs
Thank you for the great project!
One thing really missing is message extraction utility.
Any plans for implementing that?
Thanks again!
Hey, @nicksnyder! Ran into an issue today you might find interesting. Here's my situation.
I have three files, all for English: en.all.json
, en-US.all.json
, and en-GB.all.json
. The first holds all English translations that are common between all variations of English we support. The second and third files hold translations specific to their respective locales.
A user requests something from us, and sends us the locale en-US
. We make a TranslationFunc
with preferences en-US, en
. Now here's where we run into a problem: if I ask for a translation that is not in en-US.all.json
, it returns the translationID
instead of looking inside en.all.json
for a relevant translation.
This is due to the way bundle.TfuncAndLanguage
handles language preferences. When I ask for en-US
and the language has any translations, it limits the search to this map, ignoring the remaining language preferences.
I'd like to support the preference fallback at the translation level, instead of the language level. In pseudocode, this is:
translation_id = "my_string"
preferences = %[en-US en]
# Upon request for a translation, iterate through each & return
# if any of the preferences contain the translation.
preferences.each do |pref|
if translations[pref] && translations[pref][translation_id]
# A translation was found.
return translations[pref][translation_id]
end
end
# If no langs have matching translations, return the translation ID.
return translation_id
What do you think?
Given I've added a string to the translations for the default language and run goi18n *.all.json
to update the other languages with the new id without a translation, my program is now in a state where it will return ids for all non-default languages rather than something intelligible, like the value present in the default language.
The README explains that you're meant to send out the *.untranslated.json files for translations after running goi18n *.all.json
, but am I supposed to wait until I get those translations back before I can ship my code?
It would be far more preferable if I could use a default language for when a translation does not exist.
It seems to me that this is not presently possible with go-i18n. I have to merge the untranslated translation files (which contain the default language), which means then that the untranslated files get emptied. Now I cannot easily add another string as my untranslated files will then only contain the new untranslated string.
I would like to be able to maintain untranslated files with all of the untranslated strings such that whenever I get a chance to send them out I have the complete list, and have a default language used when a specific translation ID does not exist.
Does not support single subtag IETF language codes or anything else but 2-subtag codes, while the IETF language code does define other-than-2-subtag syntax.
Since browsers often send short IETF language tags, this doesn't really work with go-i18n. For example, my browser sends: Accept-Language:en-US,en;q=0.8,de;q=0.6,nl;q=0.4
. Only the en-US
is useful for go-i18n, but en
/de
/nl
are not.
It would be great if you request en-US with i18n.Tfunc and only en
exists, it uses en
. This can be extended when any request for a more specific language tag (more extensions) is boiled down to the closest supported language tag
Is it possible to write translation strings for html/template templates which contain HTML fragments?
For example:
{
"id": "userUnauthorized",
"translation": "You are not authorized to view user <span class='entityName'>{{ .userName }}</span>. "
},
Calling {{T "userUnauthorized" .}}
results in escaped HTML. I would prefer to use HTML in my translations because it reduces the total number of keys required, and I want to give my translators more flexibility with sentence construction.
If I were using html/template
directly, I could use the template.HTML
function to declare that the input is HTML and not a string. Is there a clean way to use something like this technique for translations?
I'm a newb to this project, and I think I'm following the instructions, but perhaps I'm missing something?
This code generates the error in the subject line:
bundle := i18n.NewBundle(language.English) bundle.RegisterUnmarshalFunc("toml", toml.Unmarshal) bundle.MustLoadMessageFile(path.Dir(os.Args[0]) + "/i18n/active.fr.toml")
The toml file originates from a goi18n extract.
In addition, the MustLocalize
example on this page generates the same error:
https://godoc.org/github.com/nicksnyder/go-i18n/v2/i18n#pkg-examples
Thank you in advance.
I have an English file with strings foo
and bar
. I run the tool which generates a de-DE.all.json
with empty translations for those strings. I removed the foo
string from the English file and re-ran the tool intending to regenerate the German files and it panics.
~/project ᐳ cat en-US.all.json
[
{
"id": "bar",
"translation": "Bar"
}
]
~/project ᐳ cat de-DE.all.json
[
{
"id": "bar",
"translation": ""
},
{
"id": "foo",
"translation": ""
}
]
~/project ᐳ goi18n *all.json
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x50 pc=0x100afa]
goroutine 1 [running]:
github.com/nicksnyder/go-i18n/i18n/translation.(*singleTranslation).Backfill(0x82043a3a0, 0x0, 0x0, 0x0, 0x0)
/Users/jwalker/go/src/github.com/nicksnyder/go-i18n/i18n/translation/single_translation.go:37 +0xba
main.(*mergeCommand).execute.func2(0x88205bc318, 0x82043a3a0, 0x0, 0x0)
/Users/jwalker/go/src/github.com/nicksnyder/go-i18n/goi18n/merge.go:71 +0x11a
main.filter(0x820444de0, 0x820481b00, 0x0, 0x0, 0x0)
/Users/jwalker/go/src/github.com/nicksnyder/go-i18n/goi18n/merge.go:100 +0x12e
main.(*mergeCommand).execute(0x820481ef0, 0x0, 0x0)
/Users/jwalker/go/src/github.com/nicksnyder/go-i18n/goi18n/merge.go:69 +0xe11
main.main()
/Users/jwalker/go/src/github.com/nicksnyder/go-i18n/goi18n/goi18n.go:78 +0x1ee
Hi!
This might be similar to #30 but that issue is quite old so I'm rather starting a new one here. I'm not sure it's completely related.
I am using simply de
and en
because there is just one translation for each language, no flavours and I want them to be used for any kind of locales of those languages that come along, be it en-GB
, en-US
, de-CH
, de-DE
, etc. Default language is en
in any case, and I really want to get the English text if the German version should not be found for some reason (which best should never happen).
With Accept-Language header de-DE
, the de
translations are not found.
This is becauselang.MatchingTags()
is only applied when adding translations (so in my case de
, en
, and of course the matching tags are only de
and en
).
When the translation function is requested for de-DE
, only lang.Tag
is used to lookup the fallback language and this is then only de-de
(in bundle.translatedLanguage()
).
So, my question is, would it not make more sense to use lang.MatchingTags()
also in bundle.translatedLanguage()
where the fallback is looked up?
Don't need to distinguish these since they have the same pluralization. Also seems inconsistent with pt-BR.
Hey @nicksnyder!
Thanks for writing this great series of packages. We're looking into using them and we're running into a fairly large issue: when we call a Tfunc
, it is automatically cast to a map[string]interface{}
instead of being left alone. As a result, my struct doesn't resolve properties.
Thoughts?
Hi Nick,
Adding strings to json by hand is cumbersome, goi18n could parse the source looking for Tfunc invocations and treat the arguments of such calls as candidates for translation.
I have carved a proof of concept here: https://github.com/rtfb/go-i18n/commits/sift-WIP
You can try it out with $ goi18n -sift src/ l10n/*
, which will extract untranslated stuff to *.untranslated.json
.
Would you be interested in such functionality? I would then shape it up and submit a PR.
Hi,
Thank you for creating this great tool!
I'm using the go-i18n
with the PhraseApp
, which is a platform that provides the API to download the translated file. I want to reload the translated file on the fly when the translators do some changes without re-deploying our microservices.
I noticed that the AddTranslation
method can import the translations and it will apply the lock to the bundle, but I also noticed that you mentioned that "Your Go program should load translations during its initialization". So I wonder that if it's appropriate to reload the translations after the service initialization using AddTranslation
.
The second question is about the goroutine safety of Tfunc
. Is it safe that I get a Tfunc
during the initialization and use it as a global sharing function? From my understanding, it should be ok. I just wanna confirm it here.
Thank you again.
Blocked on response from these tickets
http://unicode.org/cldr/trac/ticket/9006
http://unicode.org/cldr/trac/ticket/9022
see https://github.com/getlantern/i18n
We coudl incorporate the ability to pick detect the users locale. I need this for Desktop apps :)
CLDR provides XML data for all locales so I would like to write a tool that parses the file and generates code and tests for all locales.
If user wants to have custom language code that aren't in Unicode CLDR (like at.yaml, dk.yaml, ch.yaml, gb.yaml, etc.), bundle.ParseTranslationFileBytes
returns no language found in "dk.yaml"
.
I don't think that users should be limited only with CLDR languages. They can freely use language codes they want and it can be technically implemented.
Discussion: https://discuss.gohugo.io/t/i18n-country-codes-as-part-of-the-url/6881
Working with Android/Java the IDE generates a dynamic class R.java
with all strings ids, this method avoid a lot of bugs. In a big project is easy to reference an invalid id or to have a lot of unused ids.
What do you think the command goi18n
generates R.go
?
Example:
en-US.all.json
[
{
"id": "settings_title",
"translation": "Settings"
}
]
$ goi18n -genconst en-US.all.json -outdir ./
this command generates R.go
package R
const SettingsTitle string = "settings_title"
In code the new way will be:
T(R.SettingsTitle)
This is only a suggestion, let me know what do you think.
I can help with this change
thanks
Adding to the bundle in a concurrent program causes panics
Make bundle safe for concurrent writes
fatal error: concurrent map read and map write
goroutine 49987 [running]:
runtime.throw(0x36a3440, 0x21)
/usr/local/go/src/runtime/panic.go:547 +0x90 fp=0xc821f58bc0 sp=0xc821f58ba8
runtime.mapaccess1_faststr(0x28527e0, 0xc8203d3ef0, 0x3458300, 0xd, 0xc8203c5338)
/usr/local/go/src/runtime/hashmap_fast.go:202 +0x5b fp=0xc821f58c20 sp=0xc821f58bc0
github.com/-/-/vendor/github.com/nicksnyder/go-i18n/i18n/bundle.(*Bundle).translate(0xc82040cc90, 0xc8203f4d60, 0x3458300, 0xd, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/-/-/vendor/github.com/nicksnyder/go-i18n/i18n/bundle/bundle.go:237 +0x15c fp=0xc821f58ce8 sp=0xc821f58c20
Comments are useful for translators so they can understand additional context about the translation.
{
"id": "helloName",
"translation": "Hello {{name}}",
"comment": "This is a welcome message for a user. {{name}} is the user's first name."
}
Hi!
We started using go-i18n in one of our projects some time ago. Thank you for making our life easier!
Currently we are in the process of translating everything to several languages and started using an online service to coordinate the translations. We can export the translations from there in json format, and using them is as easy as replacing the name of the keys:
term => id
definition => translation
Is there any chance to handle term and definition as synonyms of id and translation? In that case we would be able to use the files directly without converting them at all.
Here is the format definition of the service we use: https://poeditor.com/localization/files/json
Best Regards
Tobias
Hi,
In one particular situation i want to translate some materials using a cascade of IDs.
Specifically for form error handling.
Reason for that is to be able to have a general translation, and if needed a set of more and more specific translations ID alternative for that specific (input + error) of that specific (form+input+error).
Right now i m doing it by trying to translate each possibility until it returns non empty value which is considered as the translation to consume.
I d prefer to use a HasFunc(ID)
to remove that test on empty response.
what do you think ?
I've spent a few hours trying to figure this out, but it looks like it is not possible for an argument (/insert) on a call to i18n.T to contain a percent character.
anonymized code:
func main() {
input := "%"
fmt.Printf(i18n.T("<my translation ID>",
map[string]interface{}{
"Insert": input,
}))
}
Output (with preceding message text removed):
%! (MISSING)
I've tried substituting % with %25 , url escaping , passing in as byte slice rather than string.
But it looks like a percent anywhere within a value (of the args template data) gets garbled.
Hi, I'm want to override the language used in templates in the handler but I can't find out how to do that... I load the default en-US and in this case I want to switch to nl-NL. Can you let me know how I should do that?
It looks like the recent change to support a flatter format might have some backwards-compatibility issues with yaml
https://github.com/nicksnyder/go-i18n/blob/master/i18n/bundle/bundle.go#L125
This line assumes that a standard format yaml file will start with a -
, which is true unless there's a comment at the start of the file. In that case, it'll assume it's a flat format file and panic/error.
I would love to include my translated files with go-bindata
thus not providing access to a file but a string/byte pointer of the file content.
Is there a way i can load translations from string or an go-bindata
Asset?
see: https://github.com/jteeuwen/go-bindata
How do you handle the format for dates, numbers, etc.. ?
How about stealing this: https://github.com/jbnicolai/closure-library/blob/20702eff5afe0fd9d8c5a42254879fc18c34253b/closure/goog/i18n/datetimepatternsext.js ?
For example, the angular folks leverage it through their https://docs.angularjs.org/api/ng/filter/date filter.. same with number and currency filters..
This is more of a question than a bug report. Given that I am starting a new project which I will want to serve in English and German let's say I start with this file
~/i18n ᐳ cat en-US.all.json
[
{
"id": "point_count",
"translation": {
"one": "Only {{ .Count }} Point...",
"other": "{{ .Count }} Points!"
}
}
]
When I generate the untranslated file I get this
~/i18n ᐳ goi18n *all.json
~/i18n ᐳ cat de-de.untranslated.json
[
{
"id": "point_count",
"translation": {
"one": "{{ .Count }} Points!",
"other": "{{ .Count }} Points!"
}
}
]
If I send that to be translated they won't know how English handles the one
case to generate an appropriate German translation. This seems to be expected by the test files under goi18n/testdata/expected/
but it was unexpected to me. Is this intended?
Thanks!
I have a translation like this
[customer]
one = "Kunde"
other = "Kunden"
And call
{{T "customer"}}
The restult is "customer" where I would expect one of the defined translations. I would suggest to use "other" for undefined amounts.
When using {{T "customer" 2}}
everything works fine.
CLDR zero
key support
[
{
"id": "{{.Count}} users online",
"translation": {
"zero": "Никого нет",
"other": "{{.Count}} пользователей онлайн"
}
}
]
Random translations
The idea is to be able to return a random phrase, if the term translation is an array
[
{
"id": "Hello",
"translation": ["Привет", "Здарова", "Привет, братюня", "Как сам?"]
}
]
...or so...
[
{
"id": "{{.Count}} apples",
"translation": {
"one": "{{.Person}} яблоко",
"other": ["{{.Count}} яблок", "{{.Count}} яблочек"]
}
}
]
It is not used within the package and will be ambiguous when zh-Hant and zh-Hans are merged into just zh for #1
In my templates I'm trying to do something like:
{{T "simple_user_greeting" .UserName}}
With translation:
[
{
"id": "simple_user_greeting",
"translation": "Hello {{.}}"
}
]
It outputs:
Hello map[Count:user1]
I'm expecting:
Hello user1
where user1 is .UserName.
Aparently, the first variable is parsed as 'Count' (unless not int nor string) and then mapped as such in a map. Is there a way to override this? Is it planned? Because I don't want to specify the variable name in the translation file (I'd like to do it in the template).
On this line is it necessary to have the 'string' type too? If it isn't there, I think this would work normally (although I'd understand if you don't want to change that).
{
"id": "UPDATE_CART",
"translation": "Updated cart{{if .Items}} with {{end if .Items}}{{.Items}}{{if .Items}} items{{end if .Items}}"
}
So if {{.Item}} is present then the output would be
Updated cart with 12 items
else
Updated cart
Hi,
First of all, thank you for this nice tool !
My use case :
I am facing what follows with templates :
toml
files : the template contains client application functions : goi18n
application fails to convert it the flat file into json
because those client applications functions are not known at that moment ;
json
files ;Execute()
function is called when getting the template and there is no arguments so far to make the execution correctly.
Digging into the code, I must admit that the Execute()
function on template is always called even if there is no arguments (https://github.com/nicksnyder/go-i18n/blob/master/i18n/translation/template.go#L37 called by https://github.com/nicksnyder/go-i18n/blob/master/i18n/bundle/bundle.go#L387).
Do you have any clue on how to manage this use case : let client application handle template execution ?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.