Comments (8)
Heads up @rclone/support - the "Support Contract" label was applied to this issue.
from rclone.
We only add permissionIds
if we didn't have any permissions
.
rclone/backend/drive/metadata.go
Lines 379 to 383 in 7b89735
Perhaps this is the problem? I think the reasoning behind this is that decoded permissions will have everything you need in so only show this if not present. This could be a bit cleverer and only show permisisonIds that aren't in permissions.
From the stack overflow post this looks like a bug in google drive (rclone already has quite a few workarounds for them).
If you do an rclone -M -R --disable ListR lsjson source:
on the drive remote then this should be identical to what rclone is doing when it is doing an rclone -M sync/copy
Can you do that a few times and see if you can see flakiness?
I suggest you pipe it through jq
to make it easy to read, then diff
the trials.
I didn't see a bug in the google bug tracker about this.
When metadata is enabled and a given file does not have the permissionIds property set, can we issue a request to get that information for the file in question, thereby working around this apparent API defect?
Possibly but depending on exactly what is happening we may not be able to tell the permissionsIds are missing or not.
from rclone.
I don't have direct access to this environment, but I'll see if I can arrange that test. I corrected my original posting, where I mistakenly said that the permissionIds property was not set in the metadata from rclone. That should have said the permissions property; as far as I know, I don't ever get the permissionIds property in the metadata from rclone, as that wouldn't be helpful for translating to the target backend.
If rclone does not receive a permissions property from Google Drive but does receive a permissionIds property, does it fetch the permissions for those IDs so they can be sent along to the mapper? If so, then I think the problem must be that Google Drive will not always provide at least one of these properties so that the permissions can be available for the mapper. Clarifying my suggestion, I'm asking if we can use the absence of both properties as a signal to proactively attempt to fetch the permissions of an object using the permissions.list
API. This is a hack, but if Google Drive won't provide the permissions in the files.list API, it seems better to take the performance hit of additional overhead than to risk thinking a folder/file doesn't have permissions when it really does.
Regarding --disable ListR
, is that recommended in general for Google Drive? We don't set --fast-list
, but we currently don't specifically disable ListR, so I guess rclone can possibly choose it in our scenarios.
from rclone.
@ncw The HTTP response from the test shows the presence of the permissionIds
property, but the lsjson
output does not have any permissions-related information in the metadata.
HTTP Response
{
"copyRequiresWriterPermission": false,
"md5Checksum": "a11510a3be225916b8e61fe37d0bd81a",
"viewedByMe": true,
"mimeType": "application/pdf",
"parents": ["1WLFh3GPi4yuo14my91XXdhKEl3LVLkwr"],
"webViewLink": "https://drive.google.com/file/d/1mZ2lGOWiTo1fZ9mToOwy7MoELJlhIlH_/view?usp=drivesdk",
"size": "97263",
"id": "1mZ2lGOWiTo1fZ9mToOwy7MoELJlhIlH_",
"name": "REDACTED.pdf",
"starred": false,
"trashed": false,
"explicitlyTrashed": false,
"createdTime": "2024-01-17T20:30:45.000Z",
"modifiedTime": "2024-01-17T20:29:42.252Z",
"viewedByMeTime": "2024-01-17T20:30:16.047Z",
"hasAugmentedPermissions": false,
"permissionIds": ["08143305669083284370", "09148056542529227285", "05730075870274744324", "14428220901927124198", "10333761160097088327", "01188868095341529512", "11263538924981798501", "15302165612694109360", "15968751374380107705", "01848923844518485244", "17043712991773318823", "00993337219959399960", "13488475314984407437", "15844604519322965153", "08826457509796904545", "12361398014948882636", "14049068988452217961", "17543280361655486121", "16152041014387613021", "14207865859295967165", "02698292558398922638", "08631903179092097718", "06109885187853851371", "04455832434476281173", "12302732694699397834", "07136569753069756070", "07122999719496272434", "17295166189381661175", "02012353872150768475", "13316925832582353280", "03558791494311673145", "06079240731634784000", "01693074727832241524", "16323005513084617832", "09884135463593788567", "05699788620167125482", "07088479073237879776", "11978124046481694590", "02068273953380346298"]
}
lsjson
output
[{
"Path": "REDACTED.pdf",
"Name": "REDACTED.pdf",
"Size": 97263,
"MimeType": "application/pdf",
"ModTime": "2024-01-17T20:29:42.252Z",
"IsDir": false,
"Hashes": {
"md5": "a11510a3be225916b8e61fe37d0bd81a"
},
"ID": "1mZ2lGOWiTo1fZ9mToOwy7MoELJlhIlH_",
"Metadata": {
"btime": "2024-01-17T20:30:45.000Z",
"content-type": "application/pdf",
"copy-requires-writer-permission": "false",
"mtime": "2024-01-17T20:29:42.252Z",
"starred": "false",
"viewed-by-me": "true",
"writers-can-share": "false"
}
}
]
from rclone.
From a data perspective, it looks like this is another case of permissions not being propagated because they are not set directly on the file (i.e., `"hasAugmentedPermissions": false). So I think things are being filtered out here:
rclone/backend/drive/metadata.go
Lines 373 to 377 in 7b89735
These being inherited permissions was not my understanding when the issue was raised to me, so I'll pursue it on my end for now. I think it would be helpful to log something (perhaps at the DEBUG level) in this spot so it is clear why permissions are not being propagated.
from rclone.
If rclone does not receive a permissions property from Google Drive but does receive a permissionIds property, does it fetch the permissions for those IDs so they can be sent along to the mapper?
Yes it does.
The logic is
- if rclone gets
permissions
then it ignorespermissionIds
- if there are no
permissions
then rclone fetches the permissions mentioned in thepermissionIds
and returns them as thepermissions
array in the metadata - rclone never puts
permissionIds
in the metadata (sorry about suggesting it did above!)
In the Google Drive API docs it says this about reading permissions
Output only. The full list of permissions for the file. This is only available if the requesting user can share the file. Not populated for items in shared drives.
So in some cases the permissions
won't be available in which case rclone uses the permissionIds
to fetch them if possible.
Regarding
--disable ListR
, is that recommended in general for Google Drive? We don't set--fast-list
, but we currently don't specifically disable ListR, so I guess rclone can possibly choose it in our scenarios.
rclone sync
won't use ListR
by default, but rclone lsjon -R
will. The ListR
code for google drive has a number of workarounds for bugs in the google API so I wanted to make sure they were using the same listing API.
From a data perspective, it looks like this is another case of permissions not being propagated because they are not set directly on the file (i.e., `"hasAugmentedPermissions": false). So I think things are being filtered out here:
That makes sense.
These being inherited permissions was not my understanding when the issue was raised to me, so I'll pursue it on my end for now. I think it would be helpful to log something (perhaps at the DEBUG level) in this spot so it is clear why permissions are not being propagated.
I had a go at that here
v1.67.0-beta.7971.567fa2514.fix-7853-drive-perms on branch fix-7853-drive-perms (uploaded in 15-30 mins)
It will debug
Ignoring %d permissions and %d permissionIds as is shared drive with hasAugmentedPermissions false
from rclone.
@ncw The debug statement works well. Thanks! I think we can consider this issue resolved.
from rclone.
Great!
I've merged the debug to master now which means it will be in the latest beta in 15-30 minutes and released in v1.67
I'll close this for now.
from rclone.
Related Issues (20)
- rclone config does not use --password-command when encrypting configuration
- Implement chunked and parallel Upload in serve WebDAV .
- Public Link for Swift Backend HOT 1
- Explain dual id behavior when lsf google drive HOT 3
- Proton Drive: error 422 due to authenticating using an empty access token? HOT 11
- Suggestion: add a link to the nfsmount command in the documentation for the mount command HOT 2
- docker rclone volume for cloudflare r2 not working
- S3: if a bucket already exists, bucket creation silently fails HOT 5
- Life changing simple feature for headless Oauth signin HOT 1
- Allow passing config password as command parameter HOT 1
- webdav compressed stream not recognized
- `rclone serve` seems to always request a listing
- `sync` need `--download` flag to ensure `--checksum` get hash and avoid falling back to `--size-only` HOT 2
- rclone lsjson does not implement --include filter HOT 1
- DLNA Browse+BrowseMetadata not to spec, angers pedantic client. HOT 3
- Rclone sync behaving badly, forcibly clearing RAM cache HOT 7
- copy the file from onedrive to local it show the error `unauthenticated` HOT 1
- S3(Alibaba): bucket ACL changed after `rclone mkdir` HOT 4
- serve webdav: htpasswd+bcrypt cause poor performance when listing or writing dirs with many subdirs /files HOT 3
- backend: pcloud: support block level sync (upload only) HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rclone.