Comments (4)
Even better would be to make
rclone sync
or other operations where it is the obvious intent actually do this cleanup automatically. For reference, see #6437 which was closed as completed even though the problem remains.
I tend to agree with you that deleting would be the "correct" behavior when the undecryptable file is on the dest during a sync
. I didn't go that far with the PR for a few reasons:
- Concerns raised here about people intentionally mixing crypt with non-crypt, despite docs advising against this
- Concerns raised here about this catching users by surprise, and deleting their files before they have a chance to investigate the problem
- The
crypt
backend is not aware of when it is acting as a source vs. a dest, so it would actually be pretty tricky to make it delete files only when acting as a dest. It would be easy to make it delete them always, but then that verges on a different kind of incorrect, assync
is not supposed to delete anything from the source. Likewise,copy
is not supposed to delete anything at all, andcrypt
is blind to whether the caller issync
vs.copy
.
I like your idea of a --delete-errored
flag that deletes garbage whenever it's encountered. It wouldn't be terribly difficult to implement that for crypt
.
from rclone.
- Concerns raised here about people intentionally mixing crypt with non-crypt, despite docs advising against this
In my opinion, this is a concern of no importance, at least as far as errored files on the destination. A sync
operation is clearly intended to make the destination match the source, and removing any spurious destination files (including errored ones) is clearly within that intent. If the operation is a sync, and the file/dir doesn't exist on the source, then it should get removed on the destination, full stop. If someone wants to embed non-crypt data within a crypt sync destination, then it's reasonable to expect to have to to add --exclude
rules to deal with that (although I admit I'm not exactly sure how that would/should work).
2. Concerns raised here about this catching users by surprise, and deleting their files before they have a chance to investigate the problem
I also don't think this is a valid concern, as the default for deletions is --delete-after
, so no data will be lost since any potentially errored files that exist on the source would already have been replaced before deletions occur. If the user has chosen a different deletion behavior, then they got what they asked for. And if they want to be super safe, they can always use --max-delete 0
, or they should do the normal (and often recommended) precaution of running with --dry-run
before they actually run their command to make sure it's doing what they expect.
3. The
crypt
backend is not aware of when it is acting as a source vs. a dest, so it would actually be pretty tricky to make it delete files only when acting as a dest. It would be easy to make it delete them always, but then that verges on a different kind of incorrect, assync
is not supposed to delete anything from the source. Likewise,copy
is not supposed to delete anything at all, andcrypt
is blind to whether the caller issync
vs.copy
.
I totally agree that deleting errored files always (regardless of whether they are on the source or destination) is problematic, and I would not support such a solution unless it is protected by a separate and non-default flag to enable it.
However, the difficulty of implementation doesn't change the fact that for a sync
operation performed on a destination with extraneous files (including errored ones), the only proper behavior is that they should be removed.
I like your idea of a
--delete-errored
flag that deletes garbage whenever it's encountered. It wouldn't be terribly difficult to implement that forcrypt
.
I did not intend for my suggestion of --delete-errored
to apply to the source, since that would be unexpected and potentially quite dangerous. As an example, would a file that is unreadable be considered an error? While it isn't necessarily common to have files that are able to be deleted by a user that cannot read them, this is certainly possible (via acl's, SELinux policy, etc), and this could easily cause unexpected data loss if applied to the source.
I would never expect that the source of an operation would be modified under any circumstances unless the clear intent of the operation is to do so (e.g. bisync
).
from rclone.
You make good points here, and it's hard to argue with them because in principle I agree with them.
I think ultimately it is @ncw's decision, and if @ncw thinks it's a good idea, I am happy to work on a PR that treats undecryptable dest files like any other missing-on-src file. I have some ideas about how it could be done -- it would be tricky, but not impossible.
from rclone.
@ncw care to weigh in here?
from rclone.
Related Issues (20)
- can't set config_is_local or config_refresh_token when using rcd config/create endpoint HOT 4
- Fish shell completions error HOT 2
- completion of remote paths fails on spaces HOT 3
- This program can only be run on AMD64 processors with v3 microarchitecture support. HOT 7
- v1.68.0 Docker image build failed HOT 10
- `RCLONE_` prefixed environment variables for multi-valued options not working in v1.68.0 HOT 7
- DigitalOcean setup with env variables is not working anymore HOT 7
- Git-Annex Clone to Dropbox Fails HOT 4
- rclone is ignoring --s3-upload-cutoff 5G, uploads a 4.5G file as multipart HOT 2
- Backblaze with Crypt HOT 2
- Backblaze directory markers PBS datastore HOT 4
- `--rc-addr` argument for rc command not functioning as expected in v1.68.0 HOT 1
- panic during copy HOT 6
- AzureBlobStorage Env_Auth setting forces system managed identity HOT 7
- Unable to copy to FTP Write-Only Directories HOT 5
- vfs: open files disappearing from directory listings when using Proxmox Backup Server HOT 15
- S3 backend Scaleway `failed to refresh cached credentials, no EC2 IMDS role found` HOT 6
- Enable Recursive Search/Filtering for HTTP Serve Mode HOT 4
- no IPv4 port listening for rc in version 1.68.0, version 1.67.0 is fine HOT 1
- Using Rclone mount with iconv module fails to see or create some filenames
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rclone.