steilerdev / icloud-photos-sync Goto Github PK

View Code? Open in Web Editor NEW

299.0 9.0 9.0 7.29 MB

One-way sync engine for the iCloud Photos Library into the native file system

Home Page: https://icps.steiler.dev/

License: GNU General Public License v3.0

Dockerfile 0.07% TypeScript 99.01% JavaScript 0.08% Shell 0.68% Liquid 0.16%

icloud icloud-photos icloud-sync photos icloud-api icloud-backup

icloud-photos-sync's People

Contributors

Stargazers

Watchers

Forkers

alexaamaya mhmh261 tonyredondo normalerweise parkr conroy-cheers jasonwaters txwireless sina5w

icloud-photos-sync's Issues

Improve MFA Flow (re-request code / send code through other channels)

LibraryError (FATAL): Unable to verify asset

Thank you for this project!

Describe the bug
After adding the 'public.avi': 'avi' as described here, the library sync appeared to have started, but immediately stopped with the crash described below.

Error code
Unknown error, aborting! caused by LibraryError (FATAL): Unable to verify asset

Logs

Welcome to icloud-photos-sync, v.1.0.1!
Made with <3 by steilerDev
----------------------------------------------------------------------------------------------------------------------------------------------------
Authenticating user...
Device trusted
Sign in successful!
iCloud connection established!
----------------------------------------------------------------------------------------------------------------------------------------------------
Starting sync at 5/2/2023, 12:23:49
----------------------------------------------------------------------------------------------------------------------------------------------------
Loading local & fetching remote iCloud Library state...
iCloudWarning (WARN): Ignoring unknown album type 7
iCloudWarning (WARN): Ignoring unknown album type 7
iCloudWarning (WARN): Ignoring unknown album type 7
iCloudWarning (WARN): Ignoring unknown album type 7
iCloudWarning (WARN): Ignoring unknown album type 7
iCloudWarning (WARN): Ignoring unknown album type 6
iCloudWarning (WARN): Expected 80 CPLMaster & 80 CPLAsset records, but got 79 CPLMaster & 80 CPLAsset records for album 11D481FB-CE91-42F3-9172-FBE6254EDC18
iCloudWarning (WARN): Expected 5 CPLMaster & 5 CPLAsset records, but got 4 CPLMaster & 5 CPLAsset records for album 0B2151CA-1961-431D-ADA5-3F2C2502B122
iCloudWarning (WARN): Not processing record AdSv9JmsjVKNTUIZcCf4vlCd6Dfw caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AdnhtdhQkZuWilG5O/gB8Xw/gQfb caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AQjohVx9S+yE9xnDovrG1GiPe204 caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AecjNnAj02BiDzFU2KysaE4aM2it caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AZFHwwZQzZyWQnISGHRTqjKGEtFa caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AdnhtdhQkZuWilG5O/gB8Xw/gQfb caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AYssE2oGpQ8Z2SK75NEJ8hkNiYUw caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record Aecu/zEIqAlS1MEgqqhZdZPSefCL caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record 43CE49DB-99D9-4010-9FA5-D01D6CD44507 caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AX3hD4R0+LMVXKYaUCvR5A0rUfKM caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AQ4oz8BaZp++b5hLjMZnUN7jU95N caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record 57E5D256-B876-45D7-8AFB-9260A6843C27 caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record 17787FB3-11C4-40AC-82E1-0648293EA832 caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AVMvJz4BZvJIpFswko34xnWM9xoP caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record 4978E257-1488-45D3-AC92-5F3A71E63166 caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AejIKCplWaxd8TJkohyoY3VYwPen caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AX5pU2r4jfwOc8UZ/UVco7fUbmRm caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record E8BF9792-B85C-4024-ADCD-448B80899297 caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AT7LV8Pn2EBwGRtlETCpBSBm+SDC caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AejI7YQAJPnF2xzYFBO6xWx21zXG caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record 1F8ED3CE-63AE-43B2-9579-EF7CF980CB3E caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AcjudN9oiAl7XEJrlKo5aCQZCVOF caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record ASmo6SN8vaotdti68b636p23+Xs6 caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AeWpd6tstNOeAsBZp6SM5dIufQbW caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AXzJ5AG9cXnYO15pH2PPiK2RY/5o caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AQ8sHh5JHroTsQDW8vqkxXwmaulk caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AY/pzCxqg11Pi9C2fFOmgqDqYo0N caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AQvvUb6qo6pF0qj2LX4pyLM0Q7O3 caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AVIgbLv21+L8uOEEuJ0ZR3WLrDSB caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AabJZ+98+QSy4WT8rvmRa5ngRFKy caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record ARUhefG3hbXo3940wNBTwi9Ef6GY caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record 84D283F5-C7D3-4234-B578-70DD9ECE575C caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record A9C9C3B2-8A59-4C7A-9B7D-5E3B52CEB2DD caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record DA42FD03-7DC7-47F3-B6F9-8A72E444FDC3 caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record D912B565-360E-4B31-A46D-A9142B897CED caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record B3D9DC15-14B5-4913-AC34-1684B3197213 caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record ASDf2111qs9O/9GeGOBfdsGO4Gr3 caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record 9DD8E4DB-6BFD-47B2-97B8-E00F25C0D78B caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AYkgHpZjcyAWBp5MUR1fBy1yLzGq caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AXKmfnjMLNG2kuDMSH4Px5FiAGJ+ caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AWBHiEiTMWeBNTiJxDZTyKVvIGEL caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AQmPEc4agjSkggtE1Oe3V1dhrcBI caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Expected 13949 CPLMaster & 13949 CPLAsset records, but got 13900 CPLMaster & 13949 CPLAsset records for album 'All photos'
iCloudWarning (WARN): Expected 328 CPLMaster & 328 CPLAsset records, but got 327 CPLMaster & 328 CPLAsset records for album 2483A873-DD6C-4E1A-886F-ABE869BD44C8
iCloudWarning (WARN): Not processing record AZMrRih+iC7JszPTVfWfOhDAW8u3 caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AYKv87KV6elotzy6EPn//k94wizL caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AV2z3QP2exEKKEn434H0dBr6EqY1 caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AbnrCt6MCOsWbVje0QqypWsIBZoF caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record 79A41F04-E93A-4AB5-9892-6B3E6246D30D caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record FAD3952F-A39B-4EB3-A596-4779E06E3132 caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AcbX+8nj7vIBNWVQtQRKH1oQlyo4 caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AXPYv00D8esreOSWlmQShd4DrfXu caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record 890CE60B-3DBA-4401-9630-7CA652C17682 caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AYPvq7SzYzHk4yXtOPEqVMDe5QNL caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AejIKCplWaxd8TJkohyoY3VYwPen caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AWLZY1ybUlpv4NIJC286NrjiF6Fj caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record FDCAB84A-8DC1-4660-B50C-CD60008566B7 caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record ARv3fiSqFhB5wcu3MlNED8QyeI/N caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record AVSiHFZwJZXynjqiyhmZk2FQKfB6 caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Not processing record ARvkNhRkgJW/nlX2e860rNVGH3xd caused by iCloudWarning (WARN): Duplicate
iCloudWarning (WARN): Expected 4449 CPLMaster & 4449 CPLAsset records, but got 4422 CPLMaster & 4449 CPLAsset records for album 6DB0457F-A06A-4698-BE4A-88DBB7643821
Loaded local state: 5 assets & 0 albums
Fetched remote state: 14584 assets & 167 albums
Diffing remote with local state...
Diffing completed!
Writing diff to disk...
Syncing assets, by keeping 0 and removing 5 local assets, as well as adding 14584 remote assets...
                                          0% | Elapsed: 2s | 0/14584 assets downloaded
SyncWarning (WARN): Error while writing state caused by LibraryError (FATAL): Unable to verify asset ATlktvfjsQdBTWBFtKynRn7alUEn
----------------------------------------------------------------------------------------------------------------------------------------------------
Experienced fatal error at 5/2/2023, 12:24:20: SyncError (FATAL): Sync failed caused by SyncError (FATAL): Unknown error, aborting! caused by LibraryError (FATAL): Unable to verify asset ATlktvfjsQdBTWBFtKynRn7alUEn (Error Code: 0bcdabfb-2521-40ee-a9d7-92f4841ff40d)
----------------------------------------------------------------------------------------------------------------------------------------------------

Operating environmnent

OS: macOS
Version: 13.1 (22C65)
Execution environment: node

[30b9bd] icloud-photos-sync: PhotosLibrary.linkAlbumAssets, PhotosLibrary.writeAlbum, SyncEngine.addAlbum, SyncEngine.writeAl...

	New error: [icloud-photos-sync] 3 occurrences since (Fri Nov 15 2019 11:27:39)

Application	icloud-photos-sync
Attributes	error.message — 3 `Not linking ../../../_All-Photos/AQRQ4qV9tAe-ct43voy0Atzb-tCn.jpeg to /opt/icloud-photos-library/.B4D29B86-4511-45C4-9D4E-350E489EF412/.EFD6CAC4-D9D3-4171-ACFF-545406AA45BB/.8A293F7A-7C5D-4BA9-9134-6CB02017AFEC/DSC02240-edited.jpeg in album Kroatien` — 1 `Not linking ../../../_All-Photos/AVbrg_P-8SLVG_AWUjdaV4fiUReD.jpeg to /opt/icloud-photos-library/.B4D29B86-4511-45C4-9D4E-350E489EF412/.EFD6CAC4-D9D3-4171-ACFF-545406AA45BB/.146C2D79-9CA2-443C-BB2C-40715CCE309A/DSC00394-edited.jpeg in album Portraits` — 1 `Not linking ../../_All-Photos/AQRQ4qV9tAe-ct43voy0Atzb-tCn.jpeg to /opt/icloud-photos-library/.A931B091-DCA4-4D6A-92A5-761DE11FA4F2/.40AA155F-82DB-4B44-910C-47CEC3BA033F/DSC02240-edited.jpeg in album Exported` — 1 icps.description — 3 `LibraryWarning (FATAL): Not linking ../../_All-Photos/AQRQ4qV9tAe-ct43voy0Atzb-tCn.jpeg to /opt/icloud-photos-library/.A931B091-DCA4-4D6A-92A5-761DE11FA4F2/.40AA155F-82DB-4B44-910C-47CEC3BA033F/DSC02240-edited.jpeg in album Exported caused by EEXIST: file already exists, symlink '../../_All-Photos/AQRQ4qV9tAe-ct43voy0Atzb-tCn.jpeg' -> '/opt/icloud-photos-library/.A931B091-DCA4-4D6A-92A5-761DE11FA4F2/.40AA155F-82DB-4B44-910C-47CEC3BA033F/DSC02240-edited.jpeg'` — 1 LibraryWarning (FATAL): Not linking ../../../_All-Photos/AVbrg_P-8SLVG_AWUjdaV4fiUReD.jpeg to /opt/icloud-photos-library/.B4D29B86-4511-45C4-9D4E-350E489EF412/.EFD6CAC4-D9D3-4171-ACFF-545406AA45BB/.146C2D79-9CA2-443C-BB2C-40715CCE309A/DSC00394-edited.jpeg in album Portraits caused by EEXIST: file already exists, symlink '../../../_All-Photos/AVbrg_P-8SLVG_AWUjdaV4fiUReD.jpeg' -> '/opt/icloud-photos-library/.B4D29B86-4511-45C4-9D4E-350E489EF412/.EFD6CAC4-D9D3-4171-ACFF-545406AA45BB/.146C2D79-9CA2-443C-BB2C-40715CCE309A/DSC00394-edited.jpeg' — 1 LibraryWarning (FATAL): Not linking ../../../_All-Photos/AQRQ4qV9tAe-ct43voy0Atzb-tCn.jpeg to /opt/icloud-photos-library/.B4D29B86-4511-45C4-9D4E-350E489EF412/.EFD6CAC4-D9D3-4171-ACFF-545406AA45BB/.8A293F7A-7C5D-4BA9-9134-6CB02017AFEC/DSC02240-edited.jpeg in album Kroatien caused by EEXIST: file already exists, symlink '../../../_All-Photos/AQRQ4qV9tAe-ct43voy0Atzb-tCn.jpeg' -> '/opt/icloud-photos-library/.B4D29B86-4511-45C4-9D4E-350E489EF412/.EFD6CAC4-D9D3-4171-ACFF-545406AA45BB/.8A293F7A-7C5D-4BA9-9134-6CB02017AFEC/DSC02240-edited.jpeg' — 1 icps.uuid — 3 `1f9e4026-1bf4-4053-a7e4-a450a7f16cab` — 1 `e76b64d0-5501-450c-9a89-23d9a57ac7bf` — 1 `ac3f07bc-c47d-4dab-bbd8-dbb0244c376c` — 1 application.version — 3 `0.2.0` — 3
Classifiers	LibraryWarning
Frames	`PhotosLibrary.linkAlbumAssets PhotosLibrary.writeAlbum SyncEngine.addAlbum SyncEngine.writeAlbums SyncEngine.writeState process.processTicksAndRejections`

Native sync scheduling

Currently a scheduled run of the sync needs to be done in a secondary system (e.g. cron).

This should be a native feature of the app.

Docker Compose Pull unset variable

When issuing ‘docker compose pull’ I get a message that says “ WARN[0000] The "tvl" variable is not set. Defaulting to a blank string.”

My AppleID password has a ‘$’ before ‘tvl’ which are the last characters.

Entrypoint does not allow overwrite

icloud-photos-sync/docker/Dockerfile

Line 21 in 99f694c

ENTRYPOINT ["icloud-photos-sync", "daemon"]

This will now always run in daemon mode. I am trying to run it on synology and there is only a way to specify the command, which is now impossible.

Unable to add Asset with filesize > 2 GB

  last run results

Is it possible to handle file larger than 2GiB ?

[2023-01-09T14:00:51.380Z] INFO CLI-Interface: SyncWarning (WARN): Error while writing state caused by File size (2387209258) is greater than 2 GiB
[2023-01-09T14:00:51.380Z] WARN Error-Handler: SyncWarning (WARN): Error while writing state caused by File size (2387209258) is greater than 2 GiB
[2023-01-09T14:00:52.430Z] DEBUG Photos-Library: Writing asset ARCv8Z3JSEgBaRFgOJghRDIW9FKX
[2023-01-09T14:00:55.101Z] INFO CLI-Interface: undefined
[2023-01-09T14:00:55.390Z] INFO CLI-Interface: Experienced fatal error at 1/9/2023, 3:00:55 PM: SyncError (FATAL): Sync failed caused by SyncError (FATAL): Unknown error, aborting! caused by File size (2387209258) is greater than 2 GiB (Error Code: 989aac0b-2a56-4aea-acc3-0fd8bb922c31)
[2023-01-09T14:00:55.390Z] INFO CLI-Interface: undefined
[2023-01-09T14:00:55.390Z] ERROR Error-Handler: SyncError (FATAL): Sync failed caused by SyncError (FATAL): Unknown error, aborting! caused by File size (2387209258) is greater than 2 GiB (Error Code: 989aac0b-2a56-4aea-acc3-0fd8bb922c31)

Thanks !

Originally posted by @Laffreux-dev in #154 (comment)

Shared iCloud Photos Library Support

EAI_AGAIN cvws.icloud-content.com

Describe the bug
sync command exit when this error occurs

Error code

EAI_AGAIN cvws.icloud-content.com

Logs

[2023-01-11T12:36:36.823Z] INFO CLI-Interface:  [0m [33mSyncWarning (WARN): Error while writing state caused by getaddrinfo EAI_AGAIN cvws.icloud-content.com [39m [0m
[2023-01-11T12:36:36.824Z] WARN Error-Handler: SyncWarning (WARN): Error while writing state caused by getaddrinfo EAI_AGAIN cvws.icloud-content.com
[2023-01-11T12:36:40.961Z] DEBUG Photos-Library: Writing asset AbiU0tLsKcgjljuoDcWQ2eUTEYfn
[2023-01-11T12:36:42.886Z] INFO CLI-Interface:  [0m [31mundefined [39m [0m
[2023-01-11T12:36:45.372Z] INFO CLI-Interface:  [0m [31mExperienced fatal error at 1/11/2023, 1:36:42 PM: SyncError (FATAL): Sync failed caused by SyncError (FATAL): Unknown error, aborting! caused by getaddrinfo EAI_AGAIN cvws.icloud-content.com (Error Code: 1f6f3972-56a7-4042-b755-9cd2a48c8da0) [39m [0m
[2023-01-11T12:36:45.372Z] INFO CLI-Interface:  [0m [31mundefined [39m [0m
[2023-01-11T12:36:45.372Z] ERROR Error-Handler: SyncError (FATAL): Sync failed caused by SyncError (FATAL): Unknown error, aborting! caused by getaddrinfo EAI_AGAIN cvws.icloud-content.com (Error Code: 1f6f3972-56a7-4042-b755-9cd2a48c8da0)

Operating environmnent

OS: Synology DSM6
Version: ds1511+
Execution environment: docker

WebUI

Create a WebUI that allows

Exploring the local Photos Library
Archive folders
Perform MFA authentication

This WebUI should be automatically E2E tested.

Apply mtime when archiving

When archiving assets, currently the current time will be applied to the file - sync mtime / ctime from original asset.

Improve API Testing

Use json-schema to validate and test the API.

[8723b9] icloud-photos-sync: iCPSError.toiCPSError, ErrorHandler.handle, ModuleJob.run, async ESMLoader.import, async loadESM...

	New error: [icloud-photos-sync] 1 occurrence since (Fri Nov 15 2019 11:27:39)

Application	icloud-photos-sync
Attributes	error.message — 1 `Unknown error` — 1 icps.description — 1 `iCPSError (FATAL): Unknown error caused by Method not implemented.` — 1 icps.uuid — 1 `05ba3c92-8c34-4bb8-a589-af9f2c78c385` — 1 application.version — 1 `0.2.0` — 1
Classifiers	iCPSError
Frames	`iCPSError.toiCPSError ErrorHandler.handle ModuleJob.run async ESMLoader.import async loadESM async handleMainPromise`

(Chore - v1 Release) Update dependencies

Update dependencies based on Dependabot

error: Unknown filetype descriptor: com.adobe.raw-image

This one looks like its for the .DNG extension
_photo-sync_logs-3.txt

Trust token fails - worked before

Describe the bug
Trying to "sync" or "token" fails.

The installation worked just fine a week ago.
I just did manual install (again) from git from scratch and it does the same.
".trust-token.icloud" is created and has data.

Error code

"No error code!"

Logs

[2023-02-09T11:48:17.450Z] INFO i-Cloud: Initiating iCloud connection
[2023-02-09T11:48:17.451Z] DEBUG MFA-Server: Preparing MFA server on port 8181
[2023-02-09T11:48:17.452Z] DEBUG i-Cloud-Auth: Trying to load trust token from disk
[2023-02-09T11:48:17.452Z] DEBUG i-Cloud-Auth: Acquired trust token from file
[2023-02-09T11:48:17.459Z] INFO i-Cloud: Authenticating user
[2023-02-09T11:48:18.827Z] INFO i-Cloud: Authentication successful
[2023-02-09T11:48:18.827Z] DEBUG i-Cloud-Auth: Processing iCloud authentication response
[2023-02-09T11:48:18.827Z] DEBUG i-Cloud-Auth: Authentication processed, auth secrets populated
[2023-02-09T11:48:18.827Z] DEBUG i-Cloud: Acquired secrets
[2023-02-09T11:48:18.827Z] INFO i-Cloud: Setting up iCloud connection
[2023-02-09T11:48:21.578Z] DEBUG i-Cloud-Auth: Processing iCloud setup response
[2023-02-09T11:48:21.589Z] DEBUG i-Cloud: Account ready
[2023-02-09T11:48:21.589Z] INFO i-Cloud: Getting iCloud Photos Service ready
[2023-02-09T11:48:21.590Z] DEBUG i-Cloud-Photos: Getting iCloud Photos account information
[2023-02-09T11:48:21.906Z] ERROR Error-Handler: TokenError (FATAL): Unable to get trust token caused by iCloudError (FATAL): Authentication failed caused by iCloudError (FATAL): Unable to get iCloud Photos service ready caused by iCloudError (FATAL): Unexpected error while setting up iCloud Photos caused by Request failed with status code 401 (Error Code: No error code! Please enable crash reporting!)

Operating environmnent

OS: Ubuntu 22
Version: 1.0.1 latest from git
Execution environment: shell / npm

Edit

Just guessing possibilities that macth the time frame

iCloud backup issues with 16.3
Expanding Advanced Data Protection outside US.

Edit 2
https://github.com/picklepete/pyicloud Based tool seems to have issues also.

Error: Unknown filetype descriptor: public.tiff

Absolutely love what you're doing here! When deploying the container I'm able to auth w/ iCloud however the the container exits with the attached error. Presumably it seems to not care of the .TIFF extension. Thanks!

_photo-sync_logs-2.txt

Live Photo Support

Syncing Live Photo's video portion as a separate file.

Multiple iCloudWarning (WARN)

got everything installed and authenticated but can't complete the first sync.

iCloud Authentication

Improve reporting of 'Unable to verify asset'

  Hello,

Despite the max-retries at -1 by default, the sync cuts out on various errors such as the one below after less than 10/20 minutes.

If I pass the cron to "* * * * *" do I have a chance of having the script running continuously?
Is the lock released after a fatal error?
If I run the script with the --force param every minute and the script actually runs without crashing, what happens?

Thanks !

Originally posted by @Laffreux-dev in #154

Parsing assets

"Warning: Detected unknown filetype (descriptor with ext)"

The list of supported file types need to be hardcoded.

There is no logic behind supporting additional extensions, besides knowing how the filetype (as returned by the Apple backend) is mapped to a file extension.

If you should come across an error with root cause Unknown filetype descriptor <...>, like this one:

[1/4/2023, 2:04:23 PM] ERROR Error-Handler: SyncError (FATAL): Sync failed caused by LibraryError (FATAL): Error while converting asset caused by LibraryError (FATAL): Unknown filetype descriptor public.mpeg (with potential extension .mpg)

This means that you have an asset in your library whose filetype is not yet supported.

Please report this filetype descriptor in this thread to include it in future releases. In the meantime you can add the filetype descriptor as key and the file extension as value in the above mapped file of your local icloud-photos-sync installation.

In the above example, the required key/value pair would be 'public.mpeg: 'mpg'.

Parsing albums

Advanced Data Protection Support

Describe the bug

$ docker compose exec photos-sync icloud-photos-sync token
-----------------------------------------------------------------------------------------------
Welcome to icloud-photos-sync, v.1.0.1!
Made with <3 by steilerDev
-----------------------------------------------------------------------------------------------
Authenticating user...
Device trusted
Sign in successful!
-----------------------------------------------------------------------------------------------
Experienced fatal error at 3/3/2023, 12:58:54 PM: TokenError (FATAL): Unable to get trust token caused by iCloudError (FATAL): Authentication failed caused by iCloudError (FATAL): Unable to get iCloud Photos service ready caused by iCloudError (FATAL): Unexpected error while setting up iCloud Photos caused by Request failed with status code 403 (Error Code: 734410aa-a8db-48ce-971f-0f6041f8751a)
-----------------------------------------------------------------------------------------------

Note that I do have Advanced Data Protection turned on.

Logs
Please paste the log file (preferably with LOG_LEVEL=debug), located in .icloud-photos-sync.log, stored in the DATA_DIR.

[2023-03-03T20:58:50.110Z] INFO i-Cloud: Initiating iCloud connection
[2023-03-03T20:58:50.652Z] INFO i-Cloud: Authenticating user
[2023-03-03T20:58:51.323Z] INFO i-Cloud: Authentication successful
[2023-03-03T20:58:51.328Z] INFO i-Cloud: Setting up iCloud connection
[2023-03-03T20:58:53.679Z] INFO i-Cloud: Getting iCloud Photos Service ready
[2023-03-03T20:58:54.627Z] ERROR Error-Handler: TokenError (FATAL): Unable to get trust token caused by iCloudError (FATAL): Authentication failed caused by iCloudError (FATAL): Unable to get iCloud Photos service ready caused by iCloudError (FATAL): Unexpected error while setting up iCloud Photos caused by Request failed with status code 403 (Error Code: 734410aa-a8db-48ce-971f-0f6041f8751a)

Operating environmnent

OS: macOS
Version: 13.2.1
Execution environment: docker

Album diff

Handle active indexing

[1/18/2023, 5:24:17 AM] ERROR Error-Handler: TokenError (FATAL): Unable to get trust token caused by iCloudError (FATAL): Authentication failed caused by iCloudError (FATAL): Unable to get iCloud Photos service ready caused by iCloudError (FATAL): Unknown indexing state (RUNNING) (Error Code: No error code! Please enable crash reporting!)

Error & Crash reporting integration

Album state

Writing asset diff

Asset diff

(Chore - v1 Release) GH Pages review + update

Update & review the documentation on GH pages before release

(Chore - v1 Release) TypeDoc review + update

Update & review the TypeDoc comments before release

Album state (List of Albums)

Applying diff

Sync failed after activating mfa token

I'm trying to setup sync in docker.

Create docker-compose and up / run
Obtain Token
Use script to pass mfa code
Trigger sync

At step 4 I receive the following error:

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Experienced fatal error at 2/10/2023, 7:08:13 AM: SyncError (FATAL): Sync failed caused by iCloudError (FATAL): Authentication failed caused by iCloudError (FATAL): Unexpected HTTP code: 412 caused by Request failed with status code 412 (Error Code: 175f5fd1-47e8-4d8d-a568-95a9cf2d8232)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Log:

[2023-02-10T06:12:32.805Z] INFO i-Cloud: Initiating iCloud connection
[2023-02-10T06:12:32.806Z] DEBUG MFA-Server: Preparing MFA server on port 80
[2023-02-10T06:12:32.808Z] DEBUG i-Cloud-Auth: Trying to load trust token from disk
[2023-02-10T06:12:32.808Z] DEBUG i-Cloud-Auth: Acquired trust token from file
[2023-02-10T06:12:32.897Z] INFO i-Cloud: Authenticating user
[2023-02-10T06:12:34.603Z] ERROR Error-Handler: SyncError (FATAL): Sync failed caused by iCloudError (FATAL): Authentication failed caused by iCloudError (FATAL): Unexpected HTTP code: 412 caused by Request failed with status code 412 (Error Code: 27d405d4-1b61-4c8f-b9c1-7c54108ac068)

Operating environmnent

OS: Arch
Version: 1.0.1
Execution environment: Docker

Writing diff to disk

Asset state ('All Photos')

Full test coverage

Diffing local/remote state

Unknown filetype descriptor: com.apple.m4v-video

Been out of town most of the weekend and didn't get a chance to test all the formats before you posted the changes but on the video side: .M4V .MP4 .3GP .AVI and .MOV are supported in the Photos app.

For Photos, I think this should be a fairly comprehensive list but ill have to test each one to say for sure

JPG (.jpg, .jpeg, .jpe .jif, .jfif, .jfi)
JPEG 2000 (.jp2, .j2k, .jpf, .jpx, .jpm, .mj2)
PNG (.png)
GIF (.gif)
WEBP (.webp)
TIFF (.tiff, .tif)
PSD (.psd)
RAW (.raw, .arw, .cr2, .nrw, .k25. .dng)
BMP (.bmp, .dib)
HEIF (.heif, .heic)
SVG (.svg, .svgz)

_photo-sync_logs-4.txt

Loading local state

Export sync metrics

Export sync metrics (last sync, number of files, ...) as Influx Line Protocol, in order to import it into monitoring systems.

Enable archiving

Add DaemonAppEvents.START

Implement & fire off bt.sendSummedEvent

Working Theory: Rate limit metadata fetching

Describe the bug
Initial sync fails with icloud error. This is the docker :latest and :nightly. Also tried native in an Ubuntu lunar lxc container. Same error in all.

Error code
Experienced fatal error at 2/6/2023, 1:23:27 PM: SyncError (FATAL): Sync failed caused by iCloudError (FATAL): Unable to fetch records for album 'All photos' caused by socket hang up (Error Code: c697db3f-032d-4c54-bdf8-c
b0f08baaa0c)

Logs
...
[2023-02-06T02:22:39.745Z] DEBUG i-Cloud-Photos: Building query for records of album 9612CB9F-BE1F-4384-B6E3-D36F3E48915B at index 6336
[2023-02-06T02:22:39.746Z] DEBUG i-Cloud-Photos: Building query for records of album 9612CB9F-BE1F-4384-B6E3-D36F3E48915B at index 6402
[2023-02-06T02:22:39.749Z] DEBUG i-Cloud-Photos: Building query for records of album 9612CB9F-BE1F-4384-B6E3-D36F3E48915B at index 6468
[2023-02-06T02:22:39.751Z] DEBUG i-Cloud-Photos: Building query for records of album 9612CB9F-BE1F-4384-B6E3-D36F3E48915B at index 6534
[2023-02-06T02:22:39.753Z] DEBUG i-Cloud-Photos: Building query for records of album 9612CB9F-BE1F-4384-B6E3-D36F3E48915B at index 6600
[2023-02-06T02:22:39.755Z] DEBUG i-Cloud-Photos: Building query for records of album 9612CB9F-BE1F-4384-B6E3-D36F3E48915B at index 6666
[2023-02-06T02:22:39.757Z] DEBUG i-Cloud-Photos: Building query for records of album 9612CB9F-BE1F-4384-B6E3-D36F3E48915B at index 6732
[2023-02-06T02:22:39.758Z] DEBUG i-Cloud-Photos: Building query for records of album 9612CB9F-BE1F-4384-B6E3-D36F3E48915B at index 6798
[2023-02-06T02:22:39.760Z] DEBUG i-Cloud-Photos: Building query for records of album 9612CB9F-BE1F-4384-B6E3-D36F3E48915B at index 6864
[2023-02-06T02:22:40.932Z] DEBUG i-Cloud-Photos: Received expected amount (48) of records for album 879D1473-0C27-4037-95CA-D6185CB50D59
[2023-02-06T02:22:41.123Z] DEBUG i-Cloud-Photos: Received expected amount (109) of records for album C171B195-0392-406A-AA38-9BFA5D7F224C
[2023-02-06T02:22:41.135Z] DEBUG i-Cloud-Photos: Expecting 804 records for album 15BBEEA7-3540-4C28-A7DE-17CC6CFC7622, executing 13 queries
[2023-02-06T02:22:41.135Z] DEBUG i-Cloud-Photos: Building query for records of album 15BBEEA7-3540-4C28-A7DE-17CC6CFC7622 at index 0
[2023-02-06T02:22:41.139Z] DEBUG i-Cloud-Photos: Building query for records of album 15BBEEA7-3540-4C28-A7DE-17CC6CFC7622 at index 66
[2023-02-06T02:22:41.142Z] DEBUG i-Cloud-Photos: Building query for records of album 15BBEEA7-3540-4C28-A7DE-17CC6CFC7622 at index 132
[2023-02-06T02:22:41.144Z] DEBUG i-Cloud-Photos: Building query for records of album 15BBEEA7-3540-4C28-A7DE-17CC6CFC7622 at index 198
[2023-02-06T02:22:41.146Z] DEBUG i-Cloud-Photos: Building query for records of album 15BBEEA7-3540-4C28-A7DE-17CC6CFC7622 at index 264
[2023-02-06T02:22:41.148Z] DEBUG i-Cloud-Photos: Building query for records of album 15BBEEA7-3540-4C28-A7DE-17CC6CFC7622 at index 330
[2023-02-06T02:22:41.150Z] DEBUG i-Cloud-Photos: Building query for records of album 15BBEEA7-3540-4C28-A7DE-17CC6CFC7622 at index 396
[2023-02-06T02:22:41.151Z] DEBUG i-Cloud-Photos: Building query for records of album 15BBEEA7-3540-4C28-A7DE-17CC6CFC7622 at index 462
[2023-02-06T02:22:41.153Z] DEBUG i-Cloud-Photos: Building query for records of album 15BBEEA7-3540-4C28-A7DE-17CC6CFC7622 at index 528
[2023-02-06T02:22:41.154Z] DEBUG i-Cloud-Photos: Building query for records of album 15BBEEA7-3540-4C28-A7DE-17CC6CFC7622 at index 594
[2023-02-06T02:22:41.156Z] DEBUG i-Cloud-Photos: Building query for records of album 15BBEEA7-3540-4C28-A7DE-17CC6CFC7622 at index 660
[2023-02-06T02:22:41.158Z] DEBUG i-Cloud-Photos: Building query for records of album 15BBEEA7-3540-4C28-A7DE-17CC6CFC7622 at index 726
[2023-02-06T02:22:41.159Z] DEBUG i-Cloud-Photos: Building query for records of album 15BBEEA7-3540-4C28-A7DE-17CC6CFC7622 at index 792
[2023-02-06T02:22:45.007Z] DEBUG i-Cloud-Photos: Received expected amount (17) of records for album 3A93C39A-0E83-4C3E-8EF3-F2FF494F87D0
[2023-02-06T02:22:46.266Z] DEBUG i-Cloud-Photos: Received expected amount (24) of records for album C91C6650-0852-4A99-83B6-126CE61062E2
[2023-02-06T02:22:46.481Z] DEBUG i-Cloud-Photos: Received expected amount (397) of records for album 8249789F-AA02-4BB5-A06F-199C08E4317D
[2023-02-06T02:22:47.136Z] DEBUG i-Cloud-Photos: Received expected amount (839) of records for album 4863FB82-08A2-493A-A819-279A0F823991
[2023-02-06T02:22:52.161Z] DEBUG i-Cloud-Photos: Received expected amount (804) of records for album 15BBEEA7-3540-4C28-A7DE-17CC6CFC7622
[2023-02-06T02:22:57.198Z] DEBUG i-Cloud-Photos: Received expected amount (1277) of records for album 9e58d879-7583-4387-b98c-b92f099cfcdf
[2023-02-06T02:22:57.947Z] DEBUG i-Cloud-Photos: Received expected amount (765) of records for album 6E7C399D-EF85-49E1-BD98-DF8FFC5ED835
[2023-02-06T02:23:02.588Z] DEBUG i-Cloud-Photos: Expecting 1413 records for album 813B7E6D-BABF-45C8-BC0E-E10450EC3C14, executing 22 queries
[2023-02-06T02:23:02.589Z] DEBUG i-Cloud-Photos: Building query for records of album 813B7E6D-BABF-45C8-BC0E-E10450EC3C14 at index 0
[2023-02-06T02:23:02.595Z] DEBUG i-Cloud-Photos: Building query for records of album 813B7E6D-BABF-45C8-BC0E-E10450EC3C14 at index 66
[2023-02-06T02:23:02.597Z] DEBUG i-Cloud-Photos: Building query for records of album 813B7E6D-BABF-45C8-BC0E-E10450EC3C14 at index 132
[2023-02-06T02:23:02.600Z] DEBUG i-Cloud-Photos: Building query for records of album 813B7E6D-BABF-45C8-BC0E-E10450EC3C14 at index 198
[2023-02-06T02:23:02.602Z] DEBUG i-Cloud-Photos: Building query for records of album 813B7E6D-BABF-45C8-BC0E-E10450EC3C14 at index 264
[2023-02-06T02:23:02.605Z] DEBUG i-Cloud-Photos: Building query for records of album 813B7E6D-BABF-45C8-BC0E-E10450EC3C14 at index 330
[2023-02-06T02:23:02.607Z] DEBUG i-Cloud-Photos: Building query for records of album 813B7E6D-BABF-45C8-BC0E-E10450EC3C14 at index 396
[2023-02-06T02:23:02.609Z] DEBUG i-Cloud-Photos: Building query for records of album 813B7E6D-BABF-45C8-BC0E-E10450EC3C14 at index 462
[2023-02-06T02:23:02.611Z] DEBUG i-Cloud-Photos: Building query for records of album 813B7E6D-BABF-45C8-BC0E-E10450EC3C14 at index 528
[2023-02-06T02:23:02.613Z] DEBUG i-Cloud-Photos: Building query for records of album 813B7E6D-BABF-45C8-BC0E-E10450EC3C14 at index 594
[2023-02-06T02:23:02.615Z] DEBUG i-Cloud-Photos: Building query for records of album 813B7E6D-BABF-45C8-BC0E-E10450EC3C14 at index 660
[2023-02-06T02:23:02.617Z] DEBUG i-Cloud-Photos: Building query for records of album 813B7E6D-BABF-45C8-BC0E-E10450EC3C14 at index 726
[2023-02-06T02:23:02.619Z] DEBUG i-Cloud-Photos: Building query for records of album 813B7E6D-BABF-45C8-BC0E-E10450EC3C14 at index 792
[2023-02-06T02:23:02.620Z] DEBUG i-Cloud-Photos: Building query for records of album 813B7E6D-BABF-45C8-BC0E-E10450EC3C14 at index 858
[2023-02-06T02:23:02.622Z] DEBUG i-Cloud-Photos: Building query for records of album 813B7E6D-BABF-45C8-BC0E-E10450EC3C14 at index 924
[2023-02-06T02:23:02.624Z] DEBUG i-Cloud-Photos: Building query for records of album 813B7E6D-BABF-45C8-BC0E-E10450EC3C14 at index 990
[2023-02-06T02:23:02.626Z] DEBUG i-Cloud-Photos: Building query for records of album 813B7E6D-BABF-45C8-BC0E-E10450EC3C14 at index 1056
[2023-02-06T02:23:02.628Z] DEBUG i-Cloud-Photos: Building query for records of album 813B7E6D-BABF-45C8-BC0E-E10450EC3C14 at index 1122
[2023-02-06T02:23:02.630Z] DEBUG i-Cloud-Photos: Building query for records of album 813B7E6D-BABF-45C8-BC0E-E10450EC3C14 at index 1188
[2023-02-06T02:23:02.632Z] DEBUG i-Cloud-Photos: Building query for records of album 813B7E6D-BABF-45C8-BC0E-E10450EC3C14 at index 1254
[2023-02-06T02:23:02.634Z] DEBUG i-Cloud-Photos: Building query for records of album 813B7E6D-BABF-45C8-BC0E-E10450EC3C14 at index 1320
[2023-02-06T02:23:02.635Z] DEBUG i-Cloud-Photos: Building query for records of album 813B7E6D-BABF-45C8-BC0E-E10450EC3C14 at index 1386
[2023-02-06T02:23:08.317Z] DEBUG i-Cloud-Photos: Received expected amount (1413) of records for album 813B7E6D-BABF-45C8-BC0E-E10450EC3C14
[2023-02-06T02:23:17.163Z] DEBUG i-Cloud-Photos: Expecting 99 records for album 1283B8AB-DA1F-4F0D-99FE-B838F744651E, executing 2 queries
[2023-02-06T02:23:17.164Z] DEBUG i-Cloud-Photos: Building query for records of album 1283B8AB-DA1F-4F0D-99FE-B838F744651E at index 0
[2023-02-06T02:23:17.170Z] DEBUG i-Cloud-Photos: Building query for records of album 1283B8AB-DA1F-4F0D-99FE-B838F744651E at index 66
[2023-02-06T02:23:19.190Z] DEBUG i-Cloud-Photos: Received expected amount (99) of records for album 1283B8AB-DA1F-4F0D-99FE-B838F744651E
[2023-02-06T02:23:21.323Z] DEBUG i-Cloud-Photos: Received expected amount (6877) of records for album 9612CB9F-BE1F-4384-B6E3-D36F3E48915B
[2023-02-06T02:23:27.335Z] ERROR Error-Handler: SyncError (FATAL): Sync failed caused by iCloudError (FATAL): Unable to fetch records for album 'All photos' caused by socket hang up (Error Code: c697db3f-032d-4c54-bdf8-cb
0f08baaa0c)

.icloud-photos-sync.log

Operating environmnent

OS: Ubuntu kinetic
Version: 1.0.1
Execution environment: docker:nightly

steilerdev / icloud-photos-sync Goto Github PK

icloud-photos-sync's People

Contributors

Stargazers

Watchers

Forkers

icloud-photos-sync's Issues

Recommend Projects

Recommend Topics

Recommend Org