List returns error "cannot allocate memory"? about xattr

[ifelsefi]

Let me know if I can do anything to troubleshoot.

[kuba--]

@fd0 - I'll take a look. Right now we use "golang.org/x/sys/unix" (instead of syscalls), so maybe there is something....

[fd0]

@kuba-- thanks! I find this error very odd, the machine @ifelsefi used seemed to have enough free memory available, and usually Go panics instead of returning an error when no memory could be allocated... I'm a bit puzzled.

For the record, the commit of xattr in question should be v0.2.3 (1d7b7ff).

[kuba--]

@fd0, @ifelsefi - There are 2 places In xattr.List where OOM may happen. One is when we convert path string to byte slice, and the second one is when we allocate memory for a buffer with all attributes. In your case, most likely it happened here:
https://github.com/pkg/xattr/blob/v0.2.3/xattr_linux.go#L37
I don't know if we can do anything with that. Maybe I can add extra check if buf != nil and/or go to the unsafe low level and check memory limits (if I can allocate as much memory as needed).

Can you also check if it's reproducible with the latest?

[davecheney]

I'm pretty sure this error is coming from kernel land; the text is coming from syscall.Error.

…

On 7 October 2018 at 05:26, Kuba Podgórski ***@***.***> wrote: @fd0 <https://github.com/fd0>, @ifelsefi <https://github.com/ifelsefi> - There are 2 places In xattr.List where OOM may happen. One is when we convert path string to byte slice, and the second one is when we allocate memory for a buffer with all attributes. In your case, most likely it happened here: https://github.com/pkg/xattr/blob/v0.2.3/xattr_linux.go#L37 I don't know if we can do anything with that. Maybe I can add extra check if buf != nil and/or go to the *unsafe* low level and check memory limits (if I can allocate as much memory as needed). Can you also check if it's reproducible with the latest? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#35 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAcA4Q_wbOVViOuxXA_rEtKtNecfvmxks5uiPXAgaJpZM4XIVj7> .

[fd0]

@kuba-- thanks for the hints. @ifelsefi is the user in question and he's now running the latest version of restic which uses 0.3.1. We'll see what he reports. If it persists, I'd like him to run restic on just a single file, when that works out, we can e.g. use strace to see what's going on between the program and the kernel.

What I find odd is that in all cases where I had Go programs which were unable to allocate enough memory, they panic instead of returning an error (especially whith make([]byte, n))...

[davecheney]

The error is likely defined here, https://golang.org/src/syscall/zerrors_darwin_arm64.go#L1306

…

On 7 Oct 2018, at 19:19, Alexander Neumann ***@***.***> wrote: @kuba-- thanks for the hints. @ifelsefi is the user in question and he's now running the latest version of restic which uses 0.3.1. We'll see what he reports. If it persists, I'd like him to run restic on just a single file, when that works out, we can e.g. use strace to see what's going on between the program and the kernel. What I find odd is that in all cases where I had Go programs which were unable to allocate enough memory, they panic instead of returning an error (especially whith make([]byte, n))... — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[kuba--]

@davecheney - the issue was encountered on linux, so most likely it's because sysAlloc returned nil:
https://github.com/golang/go/blob/77f9b2728eb08456899e6500328e00ec4829dddf/src/runtime/mem_linux.go#L21

https://github.com/golang/go/blob/4c1c839a3d2270d846f5a6d1cd1287e8560f0265/src/runtime/malloc.go#L1189
or
https://github.com/golang/go/blob/415e948eaea05930b2a16bab6af9e38b24e8414b/src/runtime/mheap.go#L1673

And because it was xattr + rustic it could be any place where errno is ENOMEM, most likely from mmap:
https://github.com/golang/go/blob/77f9b2728eb08456899e6500328e00ec4829dddf/src/runtime/mem_linux.go#L164

[davecheney]

I'm pretty sure when you inspect the type of the error it will be syscall.Errno, https://github.com/pkg/xattr/blob/master/xattr_darwin.go#L93

…

On 7 October 2018 at 20:16, Kuba Podgórski ***@***.***> wrote: @davecheney <https://github.com/davecheney> - the issue was encountered on linux, so most likely it's because sysAlloc returned nil: https://github.com/golang/go/blob/77f9b2728eb08456899e6500328e00 ec4829dddf/src/runtime/mem_linux.go#L21 https://github.com/golang/go/blob/4c1c839a3d2270d846f5a6d1cd1287 e8560f0265/src/runtime/malloc.go#L1189 or https://github.com/golang/go/blob/415e948eaea05930b2a16bab6af9e3 8b24e8414b/src/runtime/mheap.go#L1673 And because it was xattr + rustic it could be any place where errno is ENOMEM, most likely from mmap: https://github.com/golang/go/blob/77f9b2728eb08456899e6500328e00 ec4829dddf/src/runtime/mem_linux.go#L164 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#35 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAcA84vo_5A8gtctKIlbnF4WLleDxb7ks5uicZngaJpZM4XIVj7> .

[kuba--]

I understand it happened on linux with version 0.2.3, so if you got an errno 12 from syscall:
https://github.com/pkg/xattr/blob/v0.2.3/xattr_linux.go#L39
Then there is nothing what we can do - we just propagate the error.

[kuba--]

@fd0 @ifelsefi - shall I close it? Have you had a time to investigate and/or check the latest version?

[fd0]

Ah, yes, @ifelsefi reproduced the issue while running strace:

[pid  4640] listxattr("/[...].png", 0x1049ef0, 0) = -1 ENOMEM (Cannot allocate emory)

This looks like it's just ENOMEM returned from the kernel, so in my opinion this library does the right thing. Sorry for the noise and thanks for the support!