sethrobinson / ugt Goto Github PK

Though uncommon, some games use top-to-bottom (and right-to-left) written Japanese. I found that the system struggled to identify a block of text that was only 1 character wide.

And let's say that there was a block of text that was 3 characters wide by 10 characters tall. Then it would translate it as if it was 10 lines of text that were 3 characters each, resulting in a garbled mess.

Furthermore, the translation would then try to fit this garbled mess of english into a vertical format that it doesn't really fit in...

I'm not sure how this should be fixed, as it's likely a failure on the Cloud side... but maybe there is a way to throw in a hack whenever a block of japanese text that is much taller than wide? Probably between the OCR step and the translate step?

I've set my config to expect to translate Japanese to English. But in some games more than others, it will randomly decide that it is reading chinese text instead of Japanese despite the hints. This means that the audio reads wrong, and the text will often not translate into english without additional button inputs.

Here are my relevant settings:

;source_language_hint|auto
source_language_hint|ja

audio_default_language|ja

I would like a feature that register both picture captured and picture generate with translation, like the log text but for images.

It would be great to be able to use mouse button, both as convenient and for game that skip dialogue for any key pressed.

Hey, this is an awesome project! :D I've been using it a little bit after a few other random OCR solutions for my own Japanese studying, but all solutions seem to be plagued by OCR injecting random spaces into the text, including Google's own Vision API, unfortunately. For example:

謂れも知らず忌み続けてきた

might end up getting parsed by the Vision API as:

謂れ も知らず 忌み 続けて きた

And unfortunately, this results almost 100% of the time in a different translation (sometimes slightly, sometimes severely), so the effects ripple into the experience and into the HTML view as well when trying to parse compounds with rikaikun, altogether slowing down and muddying the experience significantly. However, this seems like something that could be easily fixed prior to the translation phase a la simply (JS as an example):

visionText.replace(/ /g, '')

Since, as you know, Japanese never really has spaces in it anyway, so it's a relatively safe text replacement :) Something I could prob tackle myself, but if it's easy enough for you to throw in in an update or something, this would be super neat! Thanks so much for your hard work!

For example, this image:

Produces the following text in the copy buffer:

I have astral vision and I'm on the lookout for wizards.

With one empty line at start and two in the end.

The log strips it out correctly, btw:

あたしはアストラル視覚で魔法使いを警戒してるわ
I have astral vision and I'm on the lookout for wizards.

Also, the auto-download of update worked fine for me.

Sometimes, the vision API returns stray characters (like the giant I in this image). In this case, what is happening is that it detected ... as い. And it thinks it's a separate textbox.

There are a few possible answers... One would be to try to merge the text. That could be good, but I'm not certain how to know where to insert the text (this is at the end, but it's possible it could be detected in the middle?).

The other idea is to throw away any text bounding box that is fully enclosed within another. This has some obvious benefits as perhaps the most annoying thing is when you have two boxes of text overlapping, because almost always the box that is underneath is the larger, more important box.

What do you think?

Reproduction steps:

0. Open your system CPU monitor, and have a game running.
1. Notice CPU is at 10% utilization
2. perform a window translation of a 1920 x 1080 sized window.
3. Wait for the translation to complete. Keep the translation in the foreground
4. Notice the game is stuttering. CPU Utilization now at 29%
5. Alt-tab to your system monitor. Notice the CPU utilization immediately drops back to 10%
6. Alt-tab back to UGT. System utilization again jumps to 29%

Expected Result:
There is no reason for UGT to be churning through 20% of my CPU while idle.

From what I've read, TEXT_DETECTION is more robust, but DOCUMENT_TEXT_DETECTION is intended for documents.

My hunch is that games are probably closer to Document scans than generic photos of street signs. They are already flat, well oriented, with text that isn't skewed or effected by lighting...

I made a quick experiment to test the difference.

Here is TEXT_DETECTION

Here is DOCUMENT_TEXT_DETECTION

Notice especially the first line, where he's telling us this is his first vacation in a long time.
"It was a physical leisure battle for the first time in a long time."
vs
"It's been a while since I was free."

The second translation is much more natural (though still not perfect). The only problem with the change, is that I think the output format is different than expected. The first line of text is repeated twice, indicating the problem.

I think it would be good to look into changing the API call, and making the associated changes, as it improves the translation dramatically.

Trying to translate this image results in a freeze for me with "2 translations remaining" showing.

It doesn't make sense to play a sound on startup, for example. Or any other time, since every action already has visual indicators.

And though I'm not an expert, the audio indicator doesn't seem helpful for blind users either. it may even make it less accessible, since it's presumably more important to keep the audio channel clear of unnecessary sound effects.

Discussed in #47

So, I'll be using it for a bit, and having a 99 percentile latency in the vision API of 1000ms. Then suddenly it will take 10s of seconds to finish a translation, if it finishes at all. But I don't seem to see anything wrong in the cloud console.

I'm not sure what the issue is.

Some of the hotkeys are customizable, but others do not seem to be.

Right now, using this requires one hand on a controller, and the other using the keyboard to press "Space" to close the translation. I'd like to be able to override that hot key in particular so that I don't need to use the keyboard at all while playing.

Hey Seth :) (btw, spaces fix you did works beautifully, thank you!)

Like you, I find myself relying on rikaikun for a lot of lookup (particularly related to Kanji and Kanji compounds). The HTML view feature you implemented is a fantastic start, but I feel like a couple of small changes would help this tremendously:

1. Add a max-width: 100% or width: 100% to the image CSS (and corresponding Japanese text overlay) so that it doesn't overflow the screen; currently I have a few browser windows open side by side, so on windows not at fullscreen size that the HTML export opens out to, I have to scroll horizontally to view the entire screenshot (probably exacerbated by the fact that I'm playing on a 4k monitor as well, so the images may be larger than what you work with typically if you're working with 1080p or smaller).

2. Add the English translation to the page as well, to make it easier to compare side-by-side. Currently, I have to consult the game window itself with the translation overlay and the HTML export separately to be able to see them, but the English imo should just be rendered alongside the HTML exported original text, even if just below (a direct side-by-side might be kind of tricky). This is primarily an issue because I play with two monitors, with one typically being borderless fullscreen and the second being the one with the actual browser windows, but even if one were playing on the same monitor I feel this would be easier on the eyes :)

Thanks for your hard work as always! お疲れ様です！

cJSON_ArrayForEach(detectedLanguage, detectedLanguages)
  {
    const cJSON *languageCode = cJSON_GetObjectItemCaseSensitive(detectedLanguage, "languageCode");
    textArea.language = languageCode->valuestring;
    //LogMsg("Found language %s", languageCode->valuestring);
  }

This code iterates over the detected languages, and chooses the language code that is the last entry.

I've seen for example, both a ja and no detected as languages with equal confidence. I'm not sure if multiple languages can appear with unequal confidence, but if so, it should preserve the most likely language. In the case of a tie, I think it should prefer the first entry.

If you play with mouse and keyboard, using something like ctrl+right click (or just additional mouse keys) would be easier to use.

Good morning!
I have often used your program without problems, using the right joystick button on my Xbox One controller to translate but after a few months of not using it I today found that the gamepad on any button no longer works. I also tried with another xbox one controller and I have the same problem, but in games they work perfectly. I now do not know if it is caused by some Windows 11 update but I am using its latest version.

I attach the log file here.

log.txt

I now don't know what it is caused by and hopefully it will be fixed.

Thanks in advance!

CL_Vec2f lastVerts[4];
	for (int i = 0; i < 4; i++)
	{
		lastVerts[0] = CL_Vec2f(0, 0);
	}

i is not used in the loop, so this just initializes lastVerts[0] four times instead. This looks like a mistake.
In GameLogicComponent line ~175

Hi

Can you support Sugoi Translator AI model? (and perhaps even some of the new OCR models that have been discussed on the Discord)

Free
Offline
Unlimited
Better there is (even DeepL)

https://www.reddit.com/r/visualnovels/comments/rzhbdz/from_sugoi_translator_developer_i_present_you_the/

I'd recommend joining the Discord if questions arise :)

if (lastVerts[2].y != 0)
		{
			//has valid info, check to see if this word overlaps with the last word drawn in this block.  This is to
			//add the linefeeds which google decides not to include here

So, as it turns out, I don't think this is necessary. You can instead check for a property called detectedBreak on the symbol. That will let you know if that symbol is the end of the line.

"symbols": [
 "property": {
   "detectedBreak": {
     "type": "LINE_BREAK"
   }
 },
 "boundingBox": {
 "vertices": [

I think fixing this may make translation more reliable when spanning multiple lines.

It would be useful to be able to recall the penultimate rectangle drawn. Let me explain: sometimes it is necessary to translate the entire screen, but the command is interpreted as a new rectangle. In these cases it would be useful to recall the penultimate rectangle drawn, because it included a smaller area where the dialogues usually appear. This way you avoid having to redraw the rectangle to be able to retranslate only a small area where the dialogues appear.

JPG Quality in the config is set to 100 by default. This increases the file size, and thus the time it takes to save and transfer the file. The config recommends 95 at most, but the default is 100.

The default should be optimized for common use-cases, which means speed of translation should be prioritized.

Example file sizes:
Quality | Size
100 = 146 KB
95 = 86 KB
90 = 66 KB
85 = 52 KB
80 = 46 KB

Attached is a diff of two screenshots: One done at 100 and one at 80. You can see the difference is a rectangular section which shows some more grayish color pixels. This is the text being translated. (black pixels = no difference)

I found in my test that the translation changed a little at quality 95 and 90, and returned to the original translation at 85 and 80. This tells me that while the translation may change, it is unlikely that the quality is significantly impacting it (it is just as likely that the translation is wrong at 95% instead of at 80%).

I recommend a default of 85%. This would represent a 3x speedup in saving and sending the file. When performing a full screen translation of 1920x1080, the savings represents at a noticeable amount of time on every transfer.

Currently, when translating a window, the screen you look at is actually a screenshot with overlayed text.

Unfortunately, if the screen behind the translation changes, you will not be able to see any changes until you dismiss the translation. In the case of games, where characters may be walking or otherwise animated while dialog is on screen, this can be a minor annoyance.

The same is true when translating intro-movies, where text appears for a few seconds while the intro movie plays (think scrolling text, like opening of Star Wars). The user may translate the text, but they can't see the movie, so they don't know when the next line of dialog is on screen.

Seems the application can only select USB Cameras / Cature cards. Any way to support capture cards connected via PCI-E or even OBS / Xsplit Virtual Cameras?

Would be more efficient for something like a visual novel rather than use the game window instead.

Also are there any plans to add DeepL support?

Good morning!
Is it possible to get the captured text into the clipboard without doing the translation? And getting the translated text without displaying the screenshot with the text on it?

For example, I know very well that after the translation a screenshot with the translated text is displayed and you can also put it in the clipboard. But I would like an option to disable the screenshot without having to press space every time.
I ask for this feature because I find it more comfortable to see the translation directly in a separate window through a program that displays the contents of the clipboard in real time, so the screenshot does not sit on top of the game window.

I also ask if I can use UGT to get the original text without doing the translation. So I get the original text in the clipboard and always display it in a separate window with another program. In the config.txt file I can't disable the translation, so I ask if it's possible.
Thanks a lot in advance!

The issue is around line ~570 in GameLogicComponent.

It reuses the TextArea object instead of creating a new one for each paragraph. Making this change fixes the duplicated text issue...

New code should be something like this

string myLanguage = "";
		cJSON_ArrayForEach(detectedLanguage, detectedLanguages)
		{
			const cJSON *languageCode = cJSON_GetObjectItemCaseSensitive(detectedLanguage, "languageCode");
			myLanguage = languageCode->valuestring;
			//LogMsg("Found language %s", languageCode->valuestring);
		}
		
		const cJSON *paragraph;
		const cJSON *paragraphs = cJSON_GetObjectItemCaseSensitive(block, "paragraphs");
		
		int paraCount = 0;
		cJSON_ArrayForEach(paragraph, paragraphs)
		{
			TextArea textArea;
		//	assert(paraCount == 0 && "Should only be one of these!");
			ReadFromParagraph(paragraph, textArea);
			textArea.language = myLanguage;


			if (textArea.m_rect.get_width() > 5 && textArea.m_rect.get_height() > 5)
				m_textareas.push_back(textArea);
			paraCount++;
		}

Current DeepL request in TextAreaComponant.cpp only pings the free api link (https://api-free.deepl.com)
Does not allow paid DeepL users to connect their api (https://api.deepl.com)

If possible another option to allow the DeepL translation request to swap between the free and paid links based on user config, would be appreciated. Sometimes 500k/month characters isn't enough...

This contradicts the documentation that says that Audiere is the default audio source.

First off, thank you for making this useful program. I had the same idea to make this with google translate for a japanese game I am playing , but found yours already existed which was a nice surprise.

I am using a laptop with a 4k screen with the resolution set at 3840x2160 for my computer. Windows applies recommended scaling to 250% for desktop items such as windows, icons, and tasks. When I use UGT in desktop mode and select the current window, entire desktop, or an area on the screen, it doesn't translate the real area it should. It is always off to the corner of the window I'm in or just the top left corner of the entire desktop. When I change the Windows scaling preference to 100% for my resolution, it works correctly.

Is there anything that could be done to address this issue without having to change my laptop's resolution or scaling to make UGT work?

I tried using this program to translate English>Hebrew but the translated text is reversed.
What should be שלום is םולש.

Any idea how to fix this?

I would like the log file to be able to show 2 translations, for example I would like both English and French translation:

add_switchable_language|en|English
add_switchable_language|fr|French

all_translation_in_log|true

Hello,

First off, great work! The app works amazingly well!

However, for some game where there are multiple languages written (eg, some english names are shown above the dialog box), then the translator tends to just translate the english (which is useless) and not the japanese or other language.

With the draggable area, we could select what region to translate, would solve the issue :-)

Good morning, I am trying to get the text with UGT on Tactics Ogre Reborn but often times the sentence is not unique and it splits, so when doing machine translation it is never correct. I tried editing auto_glue settings but I didn't solve it.

Here are some examples. So is it possible to get a single sentence? Thank you very much!

NO... THIS WON'T END WELL. AND WHAT DO WE "GAIN" BY
TAKING THEIR LIVES, ANYWAY?

THE TURMOIL OF THE LAST WAR HAS ONLY JUST
SETTLED, AND YOU'D START ANOTHER?

Sorry about this but I'm not sure how to properly install the program and set it up using the config file. Is there anyone who could possibly show me how it should look and where to place my sensitive information and codes in the config?

I'd be extremely grateful, thank you!

Each time I need to redraw the rectangle, is there a way to repeat the last rectangle position ?

Do you consider or ever will port this to Linux? I'm a Linux and macOS only user.

Hey Seth, sorry for bombarding you :) This one's more of a feature request!

Typically (as I know you know) with many RPGs, the dialogue box is always in the same place. Being able to assign a box once and then recapture it with the exact same placement/dimensions with a hotkey rather than redrawing it would be super useful. (If this is already a feature, I apologize; wasn't able to find!)

An acceptable response to this (on the surface) might be to say that one can just capture the whole screen repeatedly with the hotkey that already exists for that, but there are some games which have tons of text on the screen besides the dialogue we might want to translate (Final Fantasy XIV Online comes to mind immediately, since that's what I've been often playing), and I have to assume this will unnecessarily query and fetch from Google's APIs, on top of just making things like the HTML export muddier than they would be with just a simple limited recapture of a certain part of the screen that only contains the desired text in question.

Let me know what your thoughts are; Capture2Text, another (less effective) OCR screen capture solution had a feature like this that would have been killer had their program actually had good OCR, but they use something local that does a pretty poor job with large text blocks, so I hardly use that feature :)

;PlayTrans: UGT config file, read at startup. THIS FILE MUST BE RENAMED config.txt!

;To use google vision (which is required to work) you'll need your own Cloud Vision API key. (this is so google can see
;who is using it and charge you money if you hammer it too much. More info here: https://cloud.google.com/vision/docs/before-you-begin
; You need an API key, not a "Service key". To get that,
; Under "Credentials", choose "+Create Credentials" and then choose API key. See pic: https://prnt.sc/s5lc0j

;Also, visit https://console.cloud.google.com/apis/api/translate.googleapis.com and enable the Translation API.
;You also need to enable the Text To Speech API so it can talk: https://console.cloud.google.com/flows/enableapi?apiid=texttospeech.googleapis.com
;Note: it will use the WaveNet voices when it can for the highest quality.

;Note: Don't share your google API key! A google API key looks like (don't include quotes or anything): AIzaSyAvhMEcxpKSBD25ze06p5AlU_IuYtZ2b_g
google_api_key|yourownkeygoeshere

;Optional DeepL API key. This requires that you register at deepl.com. Even if you use deepl, you
;still need the Google key above as well, as that's still used for the OCR and text to speech features.
;Note: you also need to set the "translation_engine" below to use deepl instead of google
;A deepL API key looks like (don't include quotes or anything): 263f19ba-b010-428c-77cf-cc18c977ef4b:fx
deepl_api_key|462c36bd-7e4b-3xpppppppp0d926f3fb:fx

;URL to use for DeepL API calls (need to change this if you aren't using the free version of DeepL)
;If you're using the free version, use https://api-free.deepl.com
;if you're using the paid version, use https://api.deepl.com
deepl_api_url|https://api-free.deepl.com

;Choose which API to use for text translations. Valid options: google, deepl. If you use deepl, you must set your API key above.
;Note: A hotkey to toggle translation engine is T, useful for comparing results
;While Deepl offers better translations, keep in mind it doesn't support as many target languages. It's missing Korean and Hindi for example and will
;give a "Unsupport target_lang" error. Works great for Japanese though.
translation_engine|deepl

;Set the mode to work in.

;desktop - this allows you to translate from things you are doing on your desktop, it works with anything that
;will render to a window (emulators, webpages, etc). Won't work with full screen usually, only windowed mode.

;camera - this is my setup for capturing from live HDMI video. Designed so the output will fill a second monitor
;and be mixed with the HDMI signal later down the chain with a live luma-key effect so I can translate while playing
;on real hardware. This would allow you to translate while playing PS4 games with zero lag for example. Alternatively, desktop mode could be
;used on top of an elgato capture window or something too.
;My retro setup uses a Hydra SCART switcher with two outputs, one for the CRT and one for the OSSC to get HDMI so I can
;play on CRT but see translations on a second monitor.
; Update: These days I just use desktop mode usually, with an elgato capture card for HDMI input for PS3/etc.

;input|camera
input|desktop

;when using camera mode and you have more than one camera available, you can set to id 1 or 2 etc instead of the default which is 0
;Note: It only seems to recognize actual device driver based capture devices/cameras, Virtual Cameras like OBS's don't show up
input_camera_device_id|0

;Enter a full or partial name of an audio device to use that instead. Check the log.txt for a list of them, it's
;scanned and printed there on startup. "default" means use the system default.
audio_device|default

;Set to disabled and we won't call https://www.rtsoft.com/ugt/checking_for_new_version.php?version= at startup
;to see if anything newer has been released. (this check happens in the background, won't slow down the app)
check_for_update_on_startup|enabled

;valid options for the below two settings (written to translation_log.txt and the clipboard):
;disabled (this means don't do anything)
;full (this means copy the pretranslated text boxes along with the translations (if any was done))
;pre (this means only the pretranslated text)
;post (this means only the translated text, if any translation was done)
log_capture_text_to_file|disabled
place_capture_text_on_clipboard|disabled

;target language codes are listed here: https://cloud.google.com/translate/docs/languages
;The first language added is the default, then the rest can be dynamically changed to by using [ and ] or L and R on control pad

;If you don't want any translations done, you can make the first language added be 00|None. (just uncomment the one below this)

;add_switchable_language|00|None
add_switchable_language|en|English
add_switchable_language|pl|Polish

;A source language hint code (same as above) can be given to Google, this is the only way I could get it to read Punjabi. For that situation,
;you would change "auto" to "pa" below. Warning, this does BREAK most other languages though, so not recommended unless you have to.
source_language_hint|auto

;Google allows you request TEXT_DETECTION or DOCUMENT_TEXT_DETECTION for the OCR phase. DOCUMENT_TEXT_DETECTION generally gives better results,
;but sometimes it lumps words from a menu together that should be separate. If you want things "split up more", try setting to TEXT_DETECTION instead.
google_text_detection_command|DOCUMENT_TEXT_DETECTION

;Auto-glue is when vertical lines of text are "glued together" to be translated as a single block. Great for dialog but bad for things that shouldn't be translated together.

;set higher to auto-glue things more even when text is farther apart vertically
auto_glue_vertical_tolerance|0.20

;set higher to auto-glue things more even when the next line doesn't start at the same horizontal place
auto_glue_horizontal_tolerance|0.30

;When uploading the screenshot to google for scanning, 100 means perfect quality (big image) and 0 would mean horrible quality. 95 is probably good
jpg_quality_for_scan|85

;Audio can be set to "fmod" or "audiere". "none" means it won't even try to initialize the audio or play any sounds.
;If this wasn't compiled with FMOD support, audiere will be used instead. (RTsoft releases will have it though)
;fmod seems slightly more compatible, audiere sometimes has weird audio crack/pops when playings mp3s generated with Google's text to speech.
audio|fmod

;Valid key names: 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, -, =, Backspace, Tab, Q, W, E, R, T, Y, U, I, O, P,
;[, ], Enter, Ctrl, A, S, D, F, G, H, J, K, L, ;, ', `, Shift, , Z, X, C, V, B, N,
;M, ,, ., /, Right Shift, Num *, Alt, Space, Caps Lock, F1, F2, F3, F4, F5, F6, F7,
;F8, F9, F10, Pause, Scroll Lock, Num 7, Num 8, Num 9, Num -, Num 4, Num 5, Num 6,
;Num +, Num 1, Num 2, Num 3, Num 0, Num Del, Sys Req, , , F11, F12,

;NOTE: Set to NONE to none use that key at all

;Hotkeys can be set to require other keys down too. Not case sensitive, order doesn't matter. Example to require pressing
;Control-Alt-Shift-F12 to do something:
;hotkey_to_scan_active_window|Control,Alt,Shift,F12|

hotkey_to_scan_active_window|Control,F12|
hotkey_to_scan_whole_desktop|Control,F11|
hotkey_to_scan_draggable_area|Control,F10|
hotkey_to_scan_last_draggable_area_again|Control,F9|

;you can also set a global gamepad button to trigger scanning the active window. This way you can scan while playing an emulator without touching the keyboard,
;or while watching video via an app showing HDMI capture or whatever. Set to "none" or blank to turn this off.
;Note: This will ONLY work if the controller is XInput compatible. (This includes XBox 360 or Xbox One controllers)

;Possible gamepad button names you can use for below:
;left action button, right action button,up action button,down action button,left shoulder button,right shoulder button
;select button,start button,left trigger button,right trigger button
;left joystick button,right joystick button, none

;Note, right joystick button means pushing down (er, in? so it clicks) on the right stick itself, not an actual button

gamepad_button_to_scan_active_window|right joystick button
gamepad_button_to_scan_active_rect_window|left joystick button

;Website for kanji lookups when you left click one (the actual kanji is appended to the end of the URL)
;Leaving commented out for the default
;kanji_lookup_website|https://jisho.org/search/

////// AUDIO/SPEECH OPTIONS ///////

;If set to true, any speech currently playing is rudely cut off when the translation window is toggled off
audio_stop_when_window_is_closed|false

;If a language cannot be autodetected, this language will be used when determined the voice and region to use
audio_default_language|ja

////// CAMERA MODE OPTIONS ////////
;This should match what your camera/streaming device can output, if it doesn't, it will be scaled to this. Oh, the apps
;borderless window will also be set to this size
;NOTE: These are ignored in desktop mode
capture_width|1920
capture_height|1080

;this lets you place the app window in a specific place, useful for putting it on your second monitor (which might be the video switcher input)
window_pos_x|0
window_pos_y|0

;instead of the normal "don't show anything until space is hit", this will just show video as fast it can the whole time,
;use for seeing what your max throughput rate is and verifying the input is working
show_live_video|0

;Causes the app to limit the darkest colors on the RGB channels to a minimum, this way pure black is reserved for
;the lumakeying over the live game image. (0 to 255, 0 is pure black (effectively disabling the option), 255 would force every pixel to pure white)
minimum_brightness_for_lumakey|10

//////////////////////////////////////