Scratch project code translator about scratchaddons

[WorldLanguages]

What I just said:

I don't know why we're talking about Google Translate and running our own servers though, when Scratch itself offers the API we want!

(Don't say it's because we want variables named "fuck" to be translatable...)

[mybearworld]

Code that uses "i" as a variable name might be translated into the pronoun for non-English languages.

[WorldLanguages]

Local translation models need 100+ MB to work though

[DNin01]

I'm buying a server soon. I could host a LibreTranslate API there. Note that this would also require to add a host permission pointing to the API's port on the server but at least it's legal.

If we send requests from Scratch webpages, I believe we don't have to request host permissions, but it should still be disclosed by the addon and our privacy policy.

[mxmou]

I wonder how Scratch prevents that API from getting abused.

It doesn't - you can access it by typing the URL into the address bar.

[BroJac5246]

-snip-
Scratch's API has a filter that isn't always accurate. Also, what if someone puts stuff against the CG in code? The addon's duty is to translate with not a single filter.

The Scratch API is free and since the filter is imposed by Scratch, it will be consistent with existing Scratch behavior.

Variable names, custom blocks, list names..., etc Aren't filtered!

If there's something that should be filtered out but isn't then it's probably for the better if we don't translate it, then.

[DNin01]

we'll need to add host permissions to https://translate.google.com/, if we do so the extension will be disabled for everyone until they allow this permission.

We could use optional host permissions instead, then we'd only need to ask for anything when the user enables the addon.

[Joeclinton1]

Translating all the variables and text of a project with a translation service costs money, so the two options would either be to use the Scratch API, or to give a copy and pastable list of named items in the project file that a user can copy into google translate then, copy back to do the translation.

If we used scratch api, we should reduce number of api calls to avoid abusing the api. They allow maximum 1028 characters so. You can probably just insert everything as a list and convert them all in one or a few goes.

[Joeclinton1]

Translating all the variables and text of a project with a translation service costs money, so the two options would either be to use the Scratch API, or to give a copy and pastable list of named items in the project file that a user can copy into google translate then, copy back to do the translation.
If we used scratch api, we should reduce number of api calls to avoid abusing the api. They allow maximum 1028 characters so. You can probably just insert everything as a list and convert them all in one or a few goes.

Umm uhh, what if we host our own translation service? I'm getting a server soon so...

You have no idea the cost or complexity of a good translation service do you? Additionally the functionality of the addon cannot depend on some questionable local server. That's such a bad idea.

[Joeclinton1]

Anyway I like this addon idea since it promotes accessibility. It is probably quite annoying for non-english users to remix the english user projects, which causes community divide.

[Joeclinton1]

Scratch (probably) does have the fee waved due to a partnership with google.

Apparently not: https://scratch.mit.edu/discuss/post/5678948/

Hmm, this makes this idea seem less strong. But we're promoting accessibility and it's within the scratch editor for scratch, so I say "better to ask for forgiveness than permission". Ideally if we batch queries like I suggested this should be but a drop in the bucket compared to usage by kids who abuse it in their projects.

[CST1229]

Does google translate have an API or something then? I fully agree with this and I’ve encountered stuff like this before.

Probably, considering all the other applications that have translation stuff built-in (including Scratch, with the Translate extension).

[WorldLanguages]

In theory we could emulate the "translate text" block from the Scratch editor extension... I wonder how Scratch prevents that API from getting abused.

[Waakul]

If we use Google Translate API and someone enables this addon on turbowarp desktop and they're offline, it won't work! We will have to use some kind of local library for the translation. Something like https://github.com/LibreTranslate/LibreTranslate perhaps...

[DNin01]

If we send requests from Scratch webpages, I believe we don't have to request host permissions

Well, if we need to bypass CORS, then we do need new host permissions. Of course, if we're sending a request to a server we control, we can always set up CORS correctly. I don't know why we're talking about Google Translate and running our own servers though, when Scratch itself offers the API we want!

Scratch's API has a filter that isn't always accurate. Also, what if someone puts stuff against the CG in code? The addon's duty is to translate with not a single filter.

The filter that the translate extension uses is mostly just there to catch swears in case those somehow show up in results.
_{...though it actually fails to catch a few swear words (the BS one, for example)}

I get what you're saying, but I don't think we should fixate on it, at least not if we have no other good API options. This is almost never going to be a problem anyway.

[Waakul]

Translating all the variables and text of a project with a translation service costs money, so the two options would either be to use the Scratch API, or to give a copy and pastable list of named items in the project file that a user can copy into google translate then, copy back to do the translation.

If we used scratch api, we should reduce number of api calls to avoid abusing the api. They allow maximum 1028 characters so. You can probably just insert everything as a list and convert them all in one or a few goes.

Umm uhh, what if we host our own translation service? I'm getting a server soon so...

[Waakul]

Translating all the variables and text of a project with a translation service costs money, so the two options would either be to use the Scratch API, or to give a copy and pastable list of named items in the project file that a user can copy into google translate then, copy back to do the translation.
If we used scratch api, we should reduce number of api calls to avoid abusing the api. They allow maximum 1028 characters so. You can probably just insert everything as a list and convert them all in one or a few goes.

Umm uhh, what if we host our own translation service? I'm getting a server soon so...

Why would we do that, though, when Scratch's API is free? Also, how powerful is your server, and do you have an internet connection that can keep up? And how do you intend to protect against DDoS attacks? Pay extra for Cloudflare?

Additionally, your server could be blocked by some school networks, while Scratch's should remain accessible.

Pretty powerful, it will be kept in a server room with efficient cooling and also be monitored by IT professionals. And cloudflare's ddos attack protection is free right? I do agree that schools might be blocked but if so, then even scratchaddons would probably be blocked. We could also explore free apis tho.

[BroJac5246]

You can probably just insert everything as a list and convert them all in one or a few goes.

I Google Translate charges per character, if that's a consideration. I don't know if Scratch has the fee waved or not.

[Joeclinton1]

You can probably just insert everything as a list and convert them all in one or a few goes.

I Google Translate charges per character, if that's a consideration. I don't know if Scratch has the fee waved or not.

Scratch (probably) does have the fee waved due to a partnership with google. But it has a maximum characters of 1028. Scratch also probably rate limits the api, to maybe 10 requests per second. So, if you had 200 things you needed to translate that would take 20 seconds. But we could bundle it together and do it in like 4 calls, which is just 0.4s.

[DNin01]

According to my testing, there are actually cases where the translation can change slightly depending on whether you request a translation for each word or you put the whole sentence in a single request with each word separated by a line. They aren't always that far off, though. I'll try do some testing and share more information later.

There are, however, a few languages of the ones that Scratch's translate extension offers where, translating something from English, the translator consistently removed all the line breaks, according to my testing:

Basque
Galician
Maori
Scots Gaelic
Zulu

I'm not sure where in the process this happens (if this has to do with the translation algorithm or Scratch's API), but if you translate something that has line breaks and is already in the target language, the line breaks are preserved if nothing changes. Sometimes the line breaks are also lost when translating something with line breaks that's in one of those languages to another language.

So, we can probably use line breaks effectively for the most part, but we should check that the number of line breaks hasn't changed after doing a translation round.

If you want to try this for yourself, here's the program I made:
Multi-line translation tester.sb3

[Joeclinton1]

Code that uses "i" as a variable name might be translated into the pronoun for non-English languages.

Simple solution is to leave any 1 or 2 letter variable name alone.

[DNin01]

I started work at https://github.com/DNin01/ScratchAddons/tree/project-translator. Right now, it's just a proof of concept - it translates a predetermined list to Spanish and logs the translations in the console.

[mxmou]

Scratch (probably) does have the fee waved due to a partnership with google.

Apparently not: https://scratch.mit.edu/discuss/post/5678948/

[BlueGoat11]

Does google translate have an API or something then? I fully agree with this and I’ve encountered stuff like this before.

[BlueGoat11]

Yeah, but is it linked to a google account, or free use?

[BroJac5246]

Yeah, but is it linked to a google account, or free use?

The calculator is estimating $10/1,000,000 characters.

[BroJac5246]

If we use Google Translate API and someone enables this addon on turbowarp desktop and they're offline, it won't work!

We can provide an error pop-up instead or ask GarboMuffin to not include it if need be.

Speaking of Turbowarp, @GarboMuffin does Turbowarp use Scratch's Google Translate API? If so, do you know how we could use that?

[BlueGoat11]

I don’t think anyone wants to pay for this…

[Waakul]

There's a package for google translate (non-official) for js: @vitalets/google-translate-api which is free. but we'll need to add host permissions to https://translate.google.com/, if we do so the extension will be disabled for everyone until they allow this permission.

NOTE: Please note that @vitalets/google-translate-api's repo's ReadMe file has a disclaimer that say's to use the official API (which is paid) to be 100% legal and that it's for pet projects or prototyping.

[Waakul]

I'm buying a server soon. I could host a LibreTranslate API there. Note that this would also require to add a host permission pointing to the API's port on the server but at least it's legal.

[WorldLanguages]

If we send requests from Scratch webpages, I believe we don't have to request host permissions

Well, if we need to bypass CORS, then we do need new host permissions. Of course, if we're sending a request to a server we control, we can always set up CORS correctly.
I don't know why we're talking about Google Translate and running our own servers though, when Scratch itself offers the API we want!

[Waakul]

If we send requests from Scratch webpages, I believe we don't have to request host permissions

Well, if we need to bypass CORS, then we do need new host permissions. Of course, if we're sending a request to a server we control, we can always set up CORS correctly. I don't know why we're talking about Google Translate and running our own servers though, when Scratch itself offers the API we want!

Scratch's API has a filter that isn't always accurate. Also, what if someone puts stuff against the CG in code? The addon's duty is to translate with not a single filter.

[BroJac5246]

If we send requests from Scratch webpages, I believe we don't have to request host permissions

Well, if we need to bypass CORS, then we do need new host permissions. Of course, if we're sending a request to a server we control, we can always set up CORS correctly. I don't know why we're talking about Google Translate and running our own servers though, when Scratch itself offers the API we want!

Scratch's API has a filter that isn't always accurate. Also, what if someone puts stuff against the CG in code? The addon's duty is to translate with not a single filter.

The Scratch API is free and since the filter is imposed by Scratch, it will be consistent with existing Scratch behavior.

[Waakul]

If we send requests from Scratch webpages, I believe we don't have to request host permissions

Well, if we need to bypass CORS, then we do need new host permissions. Of course, if we're sending a request to a server we control, we can always set up CORS correctly. I don't know why we're talking about Google Translate and running our own servers though, when Scratch itself offers the API we want!

Scratch's API has a filter that isn't always accurate. Also, what if someone puts stuff against the CG in code? The addon's duty is to translate with not a single filter.

The Scratch API is free and since the filter is imposed by Scratch, it will be consistent with existing Scratch behavior.

Variable names, custom blocks, list names..., etc Aren't filtered!

[BroJac5246]

Translating all the variables and text of a project with a translation service costs money, so the two options would either be to use the Scratch API, or to give a copy and pastable list of named items in the project file that a user can copy into google translate then, copy back to do the translation.

If we used scratch api, we should reduce number of api calls to avoid abusing the api. They allow maximum 1028 characters so. You can probably just insert everything as a list and convert them all in one or a few goes.

Umm uhh, what if we host our own translation service? I'm getting a server soon so...

Why would we do that, though, when Scratch's API is free? Also, how powerful is your server, and do you have an internet connection that can keep up? And how do you intent to protect against DDoS attacks? Pay extra for Cloudflare?

Additionally, your server could be blocked by some school networks, while Scratch's should remain accessible.

Also, W_L made it clear that he thinks we should use Scratch's API.

[Waakul]

I remember something called yandextranslate, a free translation api: https://rapidapi.com/zakutynsky/api/YandexTranslate/

[DNin01]

You can probably just insert everything as a list and convert them all in one or a few goes.

How context-sensitive is the algorithm? Will we be able to do that or would have to make a separate request for each entry?

[Joeclinton1]

You can probably just insert everything as a list and convert them all in one or a few goes.

How context-sensitive is the algorithm? Will we be able to do that or would have to make a separate request for each entry?

It's very context sensitive. Almost anything you use to delimt the sentences will have edge cases where it accidently merges the two. However I found %0A (the Line Feed symbol) has the magic ability to delimit anything.

Here's an example:
https://translate-service.scratch.mit.edu/translate?language=fr&text=The%0Acat%0Aslept%0Asoundly%0Aon%0Athe%0Asunlit%0Awindowsill.

This returns {"result":"Le\nchat\ndormi\nsainement\nsur\nle\nensoleillé\nrebord de fenêtre."}

Using anything else will fail here.

The \n can then easily be used as the delimiter to split on. It might seem like translations of \n would cause problems but the backslash outputs to \\ so you can distinguish them.

Also tested in various languages and it works.

[DNin01]

It's very context sensitive. Almost anything you use to delimt the sentences will have edge cases where it accidently merges the two. However I found %0A (the Line Feed symbol) has the magic ability to delimit anything.

Here's an example:

https://translate-service.scratch.mit.edu/translate?language=fr&text=The%0Acat%0Aslept%0Asoundly%0Aon%0Athe%0Asunlit%0Awindowsill.

This returns {"result":"Le\nchat\ndormi\nsainement\nsur\nle\nensoleillé\nrebord de fenêtre."}

Using anything else will fail here.

The \n can then easily be used as the delimiter to split on. It might seem like translations of \n would cause problems but the backslash outputs to \\ so you can distinguish them.

So we have a consistent delimiter. Does requesting a translation with line break-separated words differ at all from individually translating each word? If they're the same, we can just put as much as we can fit in one request without problems.

[Joeclinton1]

So we have a consistent delimiter. Does requesting a translation with line break-separated words differ at all from individually translating each word? If they're the same, we can just put as much as we can fit in one request without problems.

Yes this is what i'm saying. I've found the symbol which prevents context from one translation affecting the other. Each of these words translated as if they were separate translations. If they had all been done as one sentence it would've translated to: "Le chat a bien dormi sur le rebord de la fenêtre ensoleillé" instead.

Also doesn't have to be single words. Sentences delimited by this would work fine too.

[Joeclinton1]

According to my testing, there are actually cases where the translation can change slightly depending on whether you request a translation for each word or you put the whole sentence in a single request with each word separated by a line. They aren't always that far off, though. I'll try do some testing and share more information later.

There are, however, a few languages of the ones that Scratch's translate extension offers where, translating something from English, the translator consistently removed all the line breaks, according to my testing:

Basque
Galician
Maori
Scots Gaelic
Zulu

I'm not sure where in the process this happens (if this has to do with the translation algorithm or Scratch's API), but if you translate something that has line breaks and is already in the target language, the line breaks are preserved if nothing changes. Sometimes the line breaks are also lost when translating something with line breaks that's in one of those languages to another language.

So, we can probably use line breaks effectively for the most part, but we should check that the number of line breaks hasn't changed after doing a translation round.

If you want to try this for yourself, here's the program I made: Multi-line translation tester.sb3

I like the testing program. It's a good thing you did the extensive testing, because as with most edge cases this could've easily gone missed.

I think we check if the line breaks is the same, and if it isn't we do it line by line as a workaround.

[mxmou]

I found an old ST comment about the Translate and Text to Speech APIs:

[iqnite]

I wonder how Scratch prevents that API from getting abused.

They don't. If somebody abuses it, it breaks down.