Jiotty hangup during upload , wont start again . about jiotty-photos-uploader

[ylexus]

The config file got corrupted. Would you be able to zip /.jiottyphotosuploader/data.json and all the logs under /.jiottyphotosuploader/logs? It’s more interesting to find out why it hung and for that I need all the logs.
Privacy warning: the logs and the config will contain full paths to all the files it attempted to upload. If you don’t want to share them publicly, you may wish to email them to me on [email protected].

[ylexus]

You can copy existing data.json for later investigations and then delete it. This will allow the app to start. It will forget all the things it ever uploaded, though, and will attempt to re-upload them. This is not a big problem, as Google Photos will de-duplicate. You’ll also lose your “ignored files” preferences if you ever customized them.

[axylee]

I delete the data.json , and re-start the Jiotty to upload again , seems working fine now .
Only, some point after 10000 items , the CPU take up to 60% , and upload become very slow , you can see from " main.log.2020-05-29-07-21 " .
Any way , I turn off the app and success to open Jiotty again without delete the data.json now , I will continue to try and keep post here .

The log file is too big to post here ( limited 10MB ) , i send mail to you , thank you .

[ylexus]

The upload must stop after 10,000 items until the next day as it’s a daily limit of the Google Photos API - see the FAQ on the wiki. There are other people using it so it should really stop earlier than 10,000 items. The 60% CPU is troublesome though, that should not happen. Let me check the logs and see if I can figure something out.

[axylee]

I just have a hunch the trouble is relate to large amount of items upload at once , but you just point it out its the 10K limit , thank you.
For the high CPU draining , i realize which happens when the Jiotty is trying to upload the file ( the upload item number is increase , very slow ) , but actually no file has been uploaded , and the fail log is increase with " INVALID_ARGUMENT: Request must have less than 50 items " error . Not sure if this info will help for debug .

I think I should separate the file/folder into smaller segment anyway .

It will be better if the APP can show reach the limit sign and stop trying to upload . Not sure if this is a easy job to do .

I will try again tomorrow with small amount of picture , if its works then we can close this issue .

[ylexus]

Wait, which version are you using? This 50 item bug was fixed a while ago. Can you ensure you’re using the latest version and try again? Moreover, the latest version clearly shows a flashing warning when you’ve reached the limit.

I can see you are using 1.0.5 which is quite old. Please download the latest one and see if any of the issues re-occur.

[axylee]

Oh no ~~ i feel so sorry , some how it's show 1.0.5 , I may made mistake during re-installation .
Now , it's 1.0.9 , and I delete all the log and data.json , upload again in fresh , it looking good now and keep uploading , the CPU stay around 30% , I will keep post here once complete .

[axylee]

Its stuck here fo 30 mins , don't know what's going on .

Attached with log for your reference .
log03.zip

[ylexus]

@axylee did it progress further since you posted? Your log shows that 246 files have been successfully uploaded, but your screenshot shows only 80. I can't explain that. At the time the log was taken the app has been running for only 22 minutes. So if you're saying it's been hanging without any updates for 30 minutes, it means it was actually properly hanging. Was the UI responding when it happened, i.e. you could go to File menu and it opens? Or was it completely frozen?

I'll add a key combination that will dump app threads to file(s) for better ability to diagnose freezes.

As for your first occurrence of a corrupted config, this happened because you had two (or more) instances of the app running at the same time. I'll improve the app so that it ensures the second instance cannot be started.

[ylexus]

1.1.0 is released - please try it and if it freezes again, please press Ctrl+Alt+Shift+D, while in the app. Wait 30 seconds. This would write some thread dump files to .jiottyphotosuploader/threaddumps. They contain no personal data, so please feel free to publish them here.

[axylee]

i delete all the log and data.json , update to 1.1.0 now , re-try to upload each folder , my folder structure like this by years ( from 1996 ~ 2020 ), more close to now more the picture amount .
+Pictures
　　|---1990 <----- ( select the year folder to upload sub folders )
　　　　　|---19900101_Event` one
　　　　　|---19900301_Event two
　　　　　|---19900701_Event three
　　|---2001
　　　　　|---20010101_Event one
　　　　　|---20010301_Event two
　　　　　|---20010501_Event three
I upload by select the year folder each time , and realize

1. starting from 1990 , the year content is under 500 items , upload smooth , with no problem
2. then the year folder content 1000 items , upload smooth , with no problem
3. When the year folder content over 5000 items , the CPU usage went up to 80% , the app behavior like hangup , i check the main.log and the data.json is over 8MB , attached with the threaddumps for reference . Maybe it's cause by huge amount of log in process ?

threaddumps.zip

[axylee]

not sure if this app behavior is only happen on me ? ( over 5000 items upload at a time gets slow down like hangup) , if yes , then I may need to upgrade my NAS PC, which current is Celeron N3050 1.6GHz / 8GB DDR / Windows 10 Pro.

[ylexus]

@axylee, could you please install version 1.1.1, then start uploading your large 5000+ folder, and when it "hangs", press Ctrl+Alt+Shift+H. This will, hopefully, create a heap dump under .jiottyphotosuploader/heapdumps (it will show a dialog confirming it's been successful). Could you zip it and send it to me please?

I'll also try to reproduce myself on a large number of files. Long time ago I uploaded much more than 5000 in one go, but that was on a very old version.

[ylexus]

@axylee also, together with the heap dump, could you please zip all the logs and data.json and email them to me too, for the full investigation?

Your PC spec should be enough, Jiotty should not need much memory or CPU.

When it "hangs", do you see the flashing exclamation mark next to "Uploading media files"? If yes, and it stops for as long as 30 minutes, this it simply means you've hit the daily limit of 10,000 API requests, which is shared by all the users. But I will see if this is what's happening from the log files you'll provide.

[ylexus]

I just uploaded 3000 items and there's no increase in memory consumption whatsoever - it's all good. I don't think it's a memory leak you're experiencing. Anyway, please do send heap dumps, logs and data.json when it all happens again.

[axylee]

Thank you so much for take into this issue !
I just delete all the logs / data.json / threaddump files and update to 1.1.1 . Re-upload the 5000 content folder , I will keep post here the result .

BTW , When it "hangs " , i think it's not actually hang-up , the dialogue GUI just freeze , the uploading number stay still no increasing , the spinning circle on the left is not spinning , but when I click on the " X" to close the window , it does respond and close the window . And no flashing exclamation mark .

[ylexus]

If the circle is not spinning, this means the UI thread is hanging doing something. Can I ask, if this happens again, to take thread dumps again too? It’s now Ctrl+Alt+Shift+T (I changed the hot key), then wait 30 seconds. Then include “threadsumps” directory into the zip. Thanks!

[axylee]

The upload still in progress , has over 4500 now , the circle has stop spinning which is fine , the upload seems slow down , the number is stop rolling , but increase few hundreds after 10~20 mins . I check the data.json file , seems the app is actually running ( can be tell by the file time stamp ) , attached the screen shop and log for your reference . But i think it working now .

threaddumps.zip

threaddumps.zip

BTW, the Ctrl+Alt+Shift+H is not work , will have error window , you can tell there is error log from the main.log . and the red " failure " is fine, it's because there is not support file in the folder .

[axylee]

now the upload is getting really slow . upload number is 5000 now .

threaddumps (2).zip

[ylexus]

Can I have the logs and heap dumps when it’s really slow too please?

[axylee]

I try to hit Ctrl+Alt+Shift+H and there is a blank error window pop up , i check the log show :

[axylee]

wait , the process speed seems picking up , items process per minutes is increase . ( around 2020-06-04 16:16 is slow ) , Mmh ... ,

main - (5).log

[axylee]

it's complete the upload ( see the log ) , only the UI freeze at the end .

The last folder content is wrong ( GP album show 1200 items , but actually 3900 items in folder ) , i will try just this folder again after one day to see if the file issue .

Over all , its at least can complete up to 8000 items at once now .

main (6).log

[axylee]

So , since there is 10000 API upload restriction , plus other user , now I just need to figure out a way to upload below 5000 per day , then will be working fine .

[ylexus]

You don’t need to figure anything out manually. Just stop the upload any time, then later select the same directory again. It will remember what it’s already uploaded and will not re-upload same files again. This is what’s stored in data.json mostly.

[ylexus]

Quick question - will you be interested in helping to translate the app to Chinese? Is it Mandarin?

[axylee]

You don’t need to figure anything out manually. Just stop the upload any time, then later select the same directory again. It will remember what it’s already uploaded and will not re-upload same files again. This is what’s stored in data.json mostly.

Sure , I will give a try on next folder .

[axylee]

Quick question - will you be interested in helping to translate the app to Chinese? Is it Mandarin?

Sure , I’ll do my best , since this is a great app !
Yes , it’s Mandarin ( Traditional Chinese ) , but I can do simplify Chinese as well , just let me know what should I do .

[ylexus]

I try to hit Ctrl+Alt+Shift+H and there is a blank error window pop up , i check the log show :

This is fixed, thanks. Was some rush coding.

[ylexus]

it's complete the upload ( see the log ) , only the UI freeze at the end .

The last folder content is wrong ( GP album show 1200 items , but actually 3900 items in folder ) , i will try just this folder again after one day to see if the file issue .

Over all , its at least can complete up to 8000 items at once now .

main (6).log

I can see there's an error adding some items to albums in main.6.log, which might explain why some albums show less items. But it's hard to investigate what's happening as you only sent the last log file. Can you please always zip the whole "log", data.json, "threaddumps" and "heapdumps" directories into the one archive? This way I can reconstruct the full story of what's been happening. If you have not cleaned it up since last run, could you zip them all now and attach?

[ylexus]

Quick question - will you be interested in helping to translate the app to Chinese? Is it Mandarin?

Sure , I’ll do my best , since this is a great app !
Yes , it’s Mandarin ( Traditional Chinese ) , but I can do simplify Chinese as well , just let me know what should I do .

Thanks for your kind offer. Both traditional and simplified would be great, why not? Let's discuss Chinese translation in this new issue: #37.

[axylee]

it's complete the upload ( see the log ) , only the UI freeze at the end .
The last folder content is wrong ( GP album show 1200 items , but actually 3900 items in folder ) , i will try just this folder again after one day to see if the file issue .
Over all , its at least can complete up to 8000 items at once now .

main (6).log

I can see there's an error adding some items to albums in main.6.log, which might explain why some albums show less items. But it's hard to investigate what's happening as you only sent the last log file. Can you please always zip the whole "log", data.json, "threaddumps" and "heapdumps" directories into the one archive? This way I can reconstruct the full story of what's been happening. If you have not cleaned it up since last run, could you zip them all now and attach?

I just try upload the un-completed folder again ( just 2 sub folder , 4000 items ) , the UI freeze at 121 items , and the log file doesn't move for few hours until i hit the threaddump . I attached the log / threaddump / data.json , would you help to see where went wrong ? If you need to delete the log and try again , please feel free to let me know .

data.zip
main.zip
threaddump-20200605095434517.zip

[ylexus]

Can I ask you if by any chance D:\\_Pictures\\2015-2\\20151030_露營-五峰-瓜麗古老-民都有山\\IMG_2775.JPG and D:\\_Pictures\\2015-2\\20151030_露營-五峰-瓜麗古老-民都有山\\IMG_2579.JPG happen to be the same photo under different file name? And if not, which one of these two this link points to: https://photos.google.com/lr/photo/AM5r2wYnsaw1qbtygypxbMp6Z8HFluP6Qd5SGk8-HdWRbVwJn4-Kli4SHntlshP0qfA-jcz43mg_HSRm2aGv5LmuFcE4wV1E_Q - is it IMG_2775.JPG or IMG_2579.JPG? Both photos have exactly the same "photo taken" time of 2015-10-30 08:33:06 (UTC), so I suspect it's actually the same photo. Google Photos assigned the same media ID to both.

Your last log (main.log) shows that it actually uploaded 8444 items and not 121. It did fail at the end. Also, the thread dumps seem to have been taken after the log files were archived (I cannot see that the thread dumps were requested in the log). Can you please, next time, archive everything at the same time (maybe even in the same zip) after you've taken the thread dumps?

There is definitely:

1. a bug in Jiotty which I now fixed (released in 1.1.2). The bug prevents adding directories with same photos under different file names from being added to albums.
2. There also seem to be a bug in Google Photos java library (referenced above) which I reported, which aborts the whole upload process
3. As for the frozen UI, here I am completely puzzled. The JavaFX thread is alive, as it reacts on your thread dump shortcut, but since it's not refreshing, I suspect it's some JavaFX bug. Not much I can do here, I think, apart from waiting for the next version of JavaFX and hope it's fixed there.

[axylee]

I checked and yes , both IMG_2579.JPG & IMG_2775.JPG are the same photo with different file name ( no idea why ) . An now I update to 1.1.2 , clean up all the log / data.json / threaddump files , and re-upload again the same folder . let's see how it works . thank you for take a time to check .

[axylee]

Look like the upload has been completed , only the UI freeze at 551 which is fine .
I dump the threaddump and heapdump and zip the log and data.json for your reference ( the zip is 22MB toot big to patch here , I will mail to you ). Let me know if you need further test or you can close this issue . Once again thank you for your help and great app .

Cannot mail either . Attached zip file here : please rename the file name as below and open with 7zip ( due to Github not support file name end with 001 )
log202006090910.zip.001
log202006090910.zip.002
log202006090910.zip.003

log202006090910-001.zip

log202006090910-002.zip

log202006090910-003.zip

[ylexus]

@axylee I can't see anything abnormal in any of the files. Looks like this UI freeze is a bug in JavaFX specifically on your hardware. Looks like JavaFX is not particularly stable on various hardware. For example, I have #4 - it crashes with some Linux NVidia drivers. I'll have to close this for now - thanks for all your help, we fixed a few things here and added a lot of diagnostics capabilities :)