backend: pcloud: support block level sync (upload only) about rclone

[gfelbing]

With the current version, I was finally completed the backup of my 650GB fotos+videos from our travels in decent time 🎉

[ncw]

Great idea :-)

I gave your PR an initial review.

[gfelbing]

Hey @ncw ,

while my initial PR was open for a while, I gave the topic some more thoughts based on your review and my observations during when working with my fork:

As you already mentioned the pcloud API is pretty generic and instead of chunk writer we can simply implement OpenWriterAt, sparing a lot of duplicated logic. Nice!

Additionally, pcloud does show partially uploaded files (and leaves them broken on unfinished uploads), also in the current implementation.
The current implementation tries to fix it by trying to delete on error, but this doesn't always work, e.g. when killing the process ungracefully or when the error is due to persistent connection loss.

So IMHO setting the feature PartialUploads to true would be the cleanest option.
This will also fix some random race conditions: On some uploads, pcloud API seems to have caching issues when altering files via fileops API and query them via file API. With PartialUploads enabled, the final step (renaming the uploaded file) is done via file API and that didn't show caching issues.
Yet this brings one huge drawback to the chunked upload: We have no way to skip already uploaded chunks on partial files anymore, as the retry gets a new random suffix.

proposed solution

In order to fix this, I came up with a solution I'd like to discuss (outlined/implemented in #7949):

1. Implement stable partial suffixes instead of random ones, by hashing the source info.
2. Add a feature flag for backends that support resuming partial uploads, skipping deletion on error
3. Implement OpenWriterAt instead of OpenChunkedWriter for pcloud backend

With 1 and 2, we pcloud can cleanly set PartialUploads to true while keeping the resume functionality.
This could be useful for other backends as well, that also support resuming uploads but have PartialUploads enabled.

What do you think?

[gfelbing]

@ncw , as I removed the resume part from my PR, I'd like to move the discussion about the resume here.

Your comment on the PR:

However I don't like the resume changes. Rclone backends don't resume, and it is very difficult to do and ensure data integrity (how are we 100% sure that the partial file uploaded is the same as the start of the file we are about to upload?).

This could be the responsibility of the backend, couldn't it?
E.g. pcloud offers an API to get the checksum on any part of the file. When uploading a chunk, we can query the checksum first and skip if it fits.
Additionally, as a last line of defense, the copy operation still checks the integrity of the whole file after the resumed upload has finished.

Looking forward to your thoughts on this

[ncw]

I suggest you check out #87 and #4547 for a framework for doing this for lots of backends. This never got merged though so needs a champion to take it over.