Comments (7)
Thanks for the feedback - some points from my POV:
1 - Already discussed :)
2 - Great point, will add this to the docs!
3 - The sync error is expected on long downloads - the data initially loaded is invalidated by Apple after about an hour, which requires a metadata reload. This could be done more gracefully but dying and starting should work and it's simple (see point 5 below). However what should not happen is a locking error, something is wrong in that logic and this should be a separate bug
4 - There are some edge cases in which axios and the download process can die - there are some I'm not catching till now
3, 4 and 5 are somewhat related - because I actually tried to design the program in a way, that it would gracefully recover after a sync error. It should load everything that is on disk and compare it to what's in the cloud and make a decision about what to keep and what to remove based on this - so the logs where you are seeing a full wipe would be helpful, since this should not happen :)
6 - Actually the _All-Photos
folder is only the storage of the assets, after they are downloaded, the tool will also replicate the folder structure of your albums and restore the file names in this view:
This however only happens after an initial asset sync - the reason why I'm not using the actual names, but the IDs created by Apple is that one asset can be linked in multiple folders (so it would be space inefficient to store them in folders which is why I'm linking them) and there can be equally named files in the library (so when I store them all in a central place there might be a collision if I use the original name) - so I'm going for Apple's UUID
from icloud-photos-sync.
I've added documentation around increasing file limits - the other issues are currently hard to investigate for me (especially loosing track because this ticket includes multiple of them).
If you could open a separate issue/discussion for the remaining challenging issues, that would be highly appreciated (please include logs and error codes there).
The locking behaviour is something I'm looking into at the moment.
I'll be closing this for now - but don't hesitate to open a new issue for each of your problems - check out my brand new contributing guidelines in case you are unsure what to do!
from icloud-photos-sync.
This issue should be resolved with version v1.1.1-beta.2, please confirm.
from icloud-photos-sync.
So now that I've finally sync'ed my 300k+ elements, I definitely wanted to comment on this item. Especially the feedback:
Definitely in the enhancement area: would be nice to have the downloaded assets not dropped into a single directory with random character names of the files. Instead, would be nice to have a hierarchy of folders based on picture date with file names what they are in the underlying library similar to how icloudpd works. I was utilizing icloudpd but the project has largely stalled and the PRs for shared library support thus far haven't been successful. As almost all of our assets are in our share library, it lead me to this project. I do appreciate the objectives of a dedicated photo downloader, etc. that you've enumerated and appreciate your fine work. I also appreciate the challenges of hacking through iCloud website api to figure this stuff out.
100% this would be a great feature, even for smaller libraries. I don't really make use of Albums, but rather browse by dates (Year, Month type of folder structure). This is how the Photos organizes itself, and it's pretty comfortable for browsing old memories. The Albums strategy is ok if that's part of your main workflow, but unfortunately doesn't even work for large libraries apparently (all Albums are empty and have a long ID number instead of a name). I am extremely happy that I finally have a backup of the files, but would love to see some automatic organization applied.
But I did just want to add a huge thanks to @steilerDev - this tool downloaded my entire library in just a few days, compared to my Mac with Photos, working on this task for a few months in the native Photos app.
from icloud-photos-sync.
[B]ut unfortunately doesn't even work for large libraries apparently (all Albums are empty and have a long ID number instead of a name)
This sounds like a bug, unless all your items are part of the shared library OR you can't see symlinks through your network share, check out this discussion.
This is how it is supposed to look like:
I'm trying to explain the local file structure in here - I should add some more information around the goals of the design process.
We could 'hide' the folder that is actually holding all the data and create an 'All Photos' meta album that uses date based formatting (and links to the assets). This would add (imho unnecessary) additional logic and complexity, because you should also be able to sort the 'All Photos' album by modified time and browser the pictures in the same order as on the phone (as I'm reading and applying this metadata) - file names would still be ugly, however is AUR0lYIXVYq9SuZQ3OGBqbpEkY9P.arw
really worse than IMG_1181.arw
?
from icloud-photos-sync.
Having a better way to access the library (through a web based application) is something that I've already though about - however this is probably the lowest priority work item at the moment.
Unless someone from the community (with maybe some experience in web technology) wants to take a stab at it, I would not hold my breath for getting this done anytime soon though.
from icloud-photos-sync.
This sounds like a bug, unless all your items are part of the shared library OR you can't see symlinks through your network share, #161.
Ok, that was a good tip - I am indeed running this in a Docker container on Synology. I found out that the file browsing app on the Synology (File Station) doesn't show the links, but they are actually there when browsing in bash or mounting the SMB share. So the album names and links are ok.
However, all my albums are empty due to the
ICLOUD_PHOTOS_COUNT_MISMATCH (WARN): Received unexpected amount of records (expected 55 CPLMaster & 55 CPLAsset records, but got 0 CPLMaster & 0 CPLAsset records for album BA42E172-2EE6-47C9-8344-333F0EAF48AE)
errors (see #231 and common warnings).
Nonetheless, the automatic year/month folder sorting (ie. 2020/05) would be much appreciated. Can certainly also be done with symbolic links, though there's really no issue in that case with duplicate pointers to files as each can be allocated to a single folder.
We could 'hide' the folder that is actually holding all the data and create an 'All Photos' meta album that uses date based formatting (and links to the assets).
This could work, but no need to hide the original folder I suppose. If people want to see it, that's fine.
file names would still be ugly, however is AUR0lYIXVYq9SuZQ3OGBqbpEkY9P.arw really worse than IMG_1181.arw
Not a major issue for me - agree that a nonsense name is a nonsense name. :)
Having a better way to access the library (through a web based application) #120 - however this is probably the lowest priority work item at the moment.
This is not really something needed from my end at least.
from icloud-photos-sync.
Related Issues (20)
- Library Version 2 HOT 2
- Initial Sync stuck at the last ~14.000 assets HOT 27
- Implementing new iCloud authentication process HOT 7
- Use Continuation Marker instead of start rank
- Hard to read logs on Synology HOT 30
- PID locked when attempting sync HOT 10
- [83730a] icloud-photos-sync: iCloudAuthError: Unable to acquire PCS cookies
- [b63792] icloud-photos-sync: SyncError: Sync did not complete successfully within expected amount of tries
- [9b978c] icloud-photos-sync: iCloudPhotosError: Unexpected error while setting up iCloud Photos
- Too many files - how to analyze? HOT 1
- token command not working properly? HOT 5
- exec format error(aarch64) rassberry pi 5 HOT 2
- Reintroduce upstream axios-har-tracker
- Sync Hidden album HOT 5
- AUTH_FAILED: iCloud Authentication failed caused by AUTH_SRP_INIT_FAILED HOT 11
- Can't find album on archive HOT 6
- Errors with ADP/MFA and one-hour ADP limit
- Rotate an image using an ios device create a duplicate photo on sync (original and rotated) HOT 2
- Need assistance building the project (errors when running npm run build) HOT 5
- Installation issue - MacOS Sonoma - npm warn deprecated @[email protected] HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from icloud-photos-sync.