panic at github.com/rclone/rclone/fs/metadata.go:112: invalid memory address or nil pointer dereference about rclone

[ncw]

I'll fix this in a bit, but if you are desperate for a workaround use --streaming-upload-cutoff 10M where 10M is bigger than the biggest google doc you have.

[ncw]

Thank you for testing.

I've merged this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.67

#7848 is certainly related but we can fix stuff there.

PS Can you let me close the issue - the open issue reminds me that I need to merge the code! Thanks

[github-actions]

Heads up @rclone/support - the "Support Contract" label was applied to this issue.

[ncw]

Rclone crashed here

rclone/fs/metadata.go

Line 112 in 6a0a54a

in.SrcFs = fInfo.Name() + ":" + fInfo.Root()

Which means most likely that fInfo is nil, which is weird. This was created a few lines earlier here

rclone/fs/metadata.go

Line 107 in 6a0a54a

fInfo := o.Fs()

So it looks like the directory entry returned nil here.

I'm having trouble working out how this could happen!

It looks like it is triggered by the --metadata-mapper code when writing info about the source. Was the source google drive or onedrive - I am not sure?

I suspect it is caused by google drive and probably one of the special files like a google doc, but I'm not sure. Are there any non-files in the google drive source?

Thanks

[chscott]

This is a Google Drive (shared drive) to SharePoint (document library) scenario. Is there a way to correlate the failing thread with the activity on that thread leading up to the crash? I don't know if a line like goroutine 32662 means a thread with ID 32662, but even if it does, I don't see that ID mentioned elsewhere in the log, so I'm not sure which file is being processed.

I see these file types in the Metadata mapper sent log records:

video/quicktime
application/pdf
application/vnd.google-apps.document
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
application/vnd.openxmlformats-officedocument.wordprocessingml.document

For the Google docs, SrcFs is memory:, and SrcFsType is object.memoryFs. For example:

{
    "SrcFs": "Source{xrgXa}:",
    "SrcFsType": "drive",
    "DstFs": "Target{qpiyZ}:",
    "DstFsType": "onedrive",
    "Remote": "REDACTED.pdf",
    "Size": 194032414,
    "MimeType": "application/pdf",
    "ModTime": "2023-07-26T11:11:00Z",
    "IsDir": false,
    "ID": "REDACTED",
    "Metadata": {
        "btime": "2023-08-04T15:10:10.334Z",
        "content-type": "application/pdf",
        "copy-requires-writer-permission": "false",
        "mtime": "2023-07-26T11:11:00.000Z",
        "starred": "false",
        "viewed-by-me": "true",
        "writers-can-share": "false"
    }
}

{
    "SrcFs": "memory:",
    "SrcFsType": "object.memoryFs",
    "DstFs": "Target{qpiyZ}:",
    "DstFsType": "onedrive",
    "Remote": "REDACTED.docx",
    "Size": 10606,
    "MimeType": "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
    "ModTime": "2023-12-01T13:25:54.84Z",
    "IsDir": false,
    "Metadata": {
        "btime": "2023-12-01T13:15:46.499Z",
        "content-type": "application/vnd.google-apps.document",
        "copy-requires-writer-permission": "false",
        "mtime": "2023-12-01T13:25:54.840Z",
        "starred": "false",
        "viewed-by-me": "false",
        "writers-can-share": "false"
    }
}

[ncw]

For the Google docs, SrcFs is memory:, and SrcFsType is `object.memoryF

That's odd and probably a clue. Will investigate some more and see if I can reproduce it.

If you set --transfers and --checkers to 1 and that will serialise things and hopefully you'll get to see which file or directory is the problem in the debug. If nothing jumps out, I'll make you a build with more debug in it.

[ncw]

I managed to replicate this with

rclone copy -vvM --streaming-upload-cutoff 0 --metadata-mapper "tee -a /tmp/map.json" drive:GDocs /tmp/drive2 --checkers 1 --transfers 1 --check-first

So it is something to do with Google docs which are larger than the default 100k which is the threshold that we switch from reading into memory to streaming directly.

[ncw]

This should fix it - please give it a go!

v1.67.0-beta.7961.3db983c4d.fix-7845-metadata-panic on branch fix-7845-metadata-panic

[chscott]

I'm not sure if this comment in #7848 means anything for this issue, but the panic no longer occurs, so I'll close on the assumption this specific problem is fixed.