Add media upload capability (Two Way Sync) about gphotos-sync

[gilesknap]

As far as I recall, google backup and sync does not handle this well because of the distinction between uploading (free storage) photos/videos and other files and then invoking the aweful Google Drive /Photos integration.

Can we do better with this new API? um ... probably not. We could really do with the following missing features:-

1. modied date on mediaitems
2. ability to assign mediaitems to albums after upload
3. ability to delete media items
4. ability to modify metadata on media items after the fact

Q how are albums handled?

1. I like the idea that the immediate folder name of photos is used as an album name (but pure numeric folders are ignored as per the gphots 'photos' folder)
2. I also like the idea that links are handled so that their containing folder - however this would require a full scan of the file system first to find all links and > album for a single file could not be supported (due to Google Issue).
3. If we could implement the above then uploading a downloaded folder hierarchy to a new account should give the same library.

Q how are conflicts handled?

1. this depends on how we do uploads - if it is upload and forget then there are no conflicts
2. otherwise we are in trouble with no modified date on the server.
3. but upload and forget would be rubbsih since I'd have to remember to clear out the source folder before re-running upload (note that Google Photos will notice the same item and not duplicate it in the library but it is a massive waste of time/bandwitdth)

Q Do I scan the filesystem into the DB?-

1. allows a quick sql query to determine files to upload
2. BUT file system scan would be fast anyway
3. BUT currently, we allow --flush-db so would need to reconstruct the links between Google photos and local files that do not adhere to the folder structure of gphotos-sync downloads
4. BUT... we could do that if we moved the files into the standard structure after successful upload

Q do we support some kind of incremental upload?

1. again this would be possible if we move files into the structure.
2. many won't be keen on having their files restructured (like me for the first 15 years of photo archiving!)
   -> but I guess we add to album based on sub-folder name so the bottom level of structure could be kept.
3. incremental could also be supported by tracking uploads in the db but again this means --flush-db would be a problem

[gilesknap]

I've recently discovered that my Photos Library has been trashed by Insync which was syncing to Google Drive but managed to delete items with the same name from the same month (i.e. same folder). I managed to repair this in Drive but due to the crappy integration, the files have been lost to Google Photos.

So the first goal of this feature should be to repair this.

At first glance, the following seems promising:-

1. Scan google photos and create a DB
2. Scan my filesystem backup from before the issue.
3. Upload anything that is has a filename and create date (or approx date?) that does not match any DB items

Issues:

1. I cannot restore membership of any albums since I don't know it. This may not be an issue since most lost files are old and did not get a Google Photos album yet (they got tagged with their filesystem folder name for future applying of Google Album if that ever becomes possible).
2. It is not that easy to verify that this has worked - although I can compare the downloaded result with what Insync is currently getting from Drive - I believe this to be mostly uncorrupted.

TODO: lets start by doing the comparison scan and listing the missing files - see if its feasible.

[gilesknap]

So the above assumes we use creation date and filename as the key to matching filesystem media to Google Photos media. (EXIF->ImageData-DateTimeOriginal for local and mediaItem->mediaMetadata->creationTime for remote)

This won't work for burst shots unless the camera gives them separate filenames
This won't work if EXIF is not present (such as videos and older JPGs) - could revert to modify date then but this is fragile.

For my use case, this may not be too troublesome since I fixed up metadata before uploading to google.

[gilesknap]

I just read this on SO https://stackoverflow.com/a/50723785/8613945.
Upload using the API counts against quota
I might as well forget about upload. Google will probably not move on this.

[gilesknap]

I have repaired my missing files cause by Insync by running the new comparison feature and then using Google Photos Web API to upload all in the missing folder. I'm happy with this solution and I'm happy to keep gphotos-sync as download only. Thus will never accidentaly destroy someones 20-year photo collection. Closing the this and any future plans to provide upload.

[friesoft]

Great you've found a solution for your problem :)

Sorry for reviving this issue but I'm desperately searching how to proceed with our photos library in the future :( As you seem to have spent significant time already with the photos api maybe you've got an idea on what we could do..

We have more than 1TB of photos currently synced (oneway) to google drive (mainly as backup) and thus automatically imported to google photos (for viewing and the cool autogenerated gifs, ...) - yes we are paying for the google drive storage because we love what google photos does to our library

The sync happens on our Synology NAS using the builtin Cloud Sync (it handles the 1TB of photos like a pro)

As Google now removes the connection between Drive and Photos how could be tackle this problem?
Do you see any chance of supporting this usecase of keeping drive and photos in sync? Or do you know any other tool which could help us?

Thanks!

[gilesknap]

HI,

Thanks for the interest. You allude to an issue which is close to my heart and for which there is no great answer yet.

I used to do exactly what you are doing and backup my Google Drive synced photos. It turns out there are all sorts of issues with this and you may not really know what is backed up and what is not. See here for a (very lengthy) discussion and my comments on the issue https://kunnas.com/google-photos-is-a-disaster/

Because of the above, I developed gphotos-sync and it does an OK job but if you read my known issues here https://github.com/gilesknap/gphotos-sync#known-issues you will see that it still does not give you a perfect backup of your library. I've just commented on Google's bug tracker with my suspicion as to why these issues still persist, https://issuetracker.google.com/issues/112096115#comment13.

On top of all of this, there is the fragility of syncing between a filesystem and a Google library (drive or photos). Because the libraries allow duplicate filenames a scheme is required to manage this in the local filesystem. To date, such schemes have been fragile, even in commercial products such as InSync (as detailed above in this Issue log).

This last point is why I have decided not to make gphotos-sync a two-way process. It is just too hard to know if I will end up trashing precious photos for someone. For me, it would be useless anyway since I use the free service and the API will not upload to the free service (not a problem for you).

My recommendation would be to work like this:-

• treat Google photos as your master
• upload all of your photos there and organize them there
• use a tool like gphotos-sync to back up from Google
• periodically backup to offline disk with a file comparison tool like 'Beyond Compare' which will warn you if unexpected changes have occurred
• also occasionally use Google Takeout to extract your entire library so that you have the means to reconstruct everything (including the GPS data) if you ever want to leave Google Photos

The last point about reconstructing would require some code to scan your gphotos-backup and your google takeout and reconstruct all of your albums with all of their original data. Someone like me will almost certainly write this code if Google Photos is no longer the best option. This would rely on the fact that Google Takeout does indeed give you all your original metadata. I do not know if this is the case but I will now check and report back.

[gilesknap]

I can confirm that google takeout image and video files have the GPS info stripped but that the GPS information is provided alongside in a JSON file. Thus we could reconstruct the original data if necessary.

I expected this because I believe the Data Protection Act requires them to make your own data available to you in some form even if you have to jump through the takeout hoops to get it :-)

[gilesknap]

Oh, and takeout also includes album information too, including comments etc (I did not previously know this)

[gilesknap]

I'm reopening this issue because it is a useful discussion that might benefit others.

[gilesknap]

wrapping up in #119