Occasional segfault with --threads about pyflame

[eklitzke]

I have a bunch of these core dumps, and they're always at this offset in this function. Here's another example with a slightly different disassembly:

(gdb) disas
Dump of assembler code for function do_futex_wait:
   0x00007f9c4e419920 <+0>:	push   %r12
   0x00007f9c4e419922 <+2>:	push   %rbp
   0x00007f9c4e419923 <+3>:	mov    %rdi,%rbp
   0x00007f9c4e419926 <+6>:	push   %rbx
   0x00007f9c4e419927 <+7>:	sub    $0x10,%rsp
   0x00007f9c4e41992b <+11>:	mov    0x8(%rdi),%ebx
   0x00007f9c4e41992e <+14>:	callq  0x7f9c4e41a280 <__pthread_enable_asynccancel>
   0x00007f9c4e419933 <+19>:	mov    $0xffffffff,%r9d
   0x00007f9c4e419939 <+25>:	mov    %eax,%r12d
   0x00007f9c4e41993c <+28>:	xor    %r8d,%r8d
   0x00007f9c4e41993f <+31>:	xor    $0x189,%ebx
   0x00007f9c4e419945 <+37>:	xor    %r10d,%r10d
   0x00007f9c4e419948 <+40>:	xor    %edx,%edx
   0x00007f9c4e41994a <+42>:	movslq %ebx,%rsi
   0x00007f9c4e41994d <+45>:	mov    %rbp,%rdi
   0x00007f9c4e419950 <+48>:	mov    $0xca,%eax
   0x00007f9c4e419955 <+53>:	syscall 
   0x00007f9c4e419957 <+55>:	syscall 
=> 0x00007f9c4e419959 <+57>:	add    %al,(%rax)
   0x00007f9c4e41995b <+59>:	add    %al,(%rax)
   0x00007f9c4e41995d <+61>:	add    %al,(%rax)
   0x00007f9c4e41995f <+63>:	mov    %r12d,%edi
   0x00007f9c4e419962 <+66>:	callq  0x7f9c4e41a2e0 <__pthread_disable_asynccancel>
   0x00007f9c4e419967 <+71>:	xor    %eax,%eax
   0x00007f9c4e419969 <+73>:	add    $0x10,%rsp
   0x00007f9c4e41996d <+77>:	pop    %rbx
   0x00007f9c4e41996e <+78>:	pop    %rbp
   0x00007f9c4e41996f <+79>:	pop    %r12
   0x00007f9c4e419971 <+81>:	retq   
   0x00007f9c4e419972 <+82>:	nopw   0x0(%rax,%rax,1)
   0x00007f9c4e419978 <+88>:	mov    %r12d,%edi
   0x00007f9c4e41997b <+91>:	mov    %rax,0x8(%rsp)
   0x00007f9c4e419980 <+96>:	callq  0x7f9c4e41a2e0 <__pthread_disable_asynccancel>
   0x00007f9c4e419985 <+101>:	mov    0x8(%rsp),%rax
   0x00007f9c4e41998a <+106>:	cmp    $0xfffffff5,%eax
   0x00007f9c4e41998d <+109>:	je     0x7f9c4e4199a8 <do_futex_wait+136>
   0x00007f9c4e41998f <+111>:	cmp    $0xfffffffc,%eax
   0x00007f9c4e419992 <+114>:	je     0x7f9c4e4199a8 <do_futex_wait+136>
   0x00007f9c4e419994 <+116>:	cmp    $0xffffff92,%eax
   0x00007f9c4e419997 <+119>:	je     0x7f9c4e4199a8 <do_futex_wait+136>
   0x00007f9c4e419999 <+121>:	lea    0x3420(%rip),%rdi        # 0x7f9c4e41cdc0
   0x00007f9c4e4199a0 <+128>:	callq  0x7f9c4e40f630 <__libc_fatal@plt>
   0x00007f9c4e4199a5 <+133>:	nopl   (%rax)
   0x00007f9c4e4199a8 <+136>:	neg    %eax
   0x00007f9c4e4199aa <+138>:	jmp    0x7f9c4e419969 <do_futex_wait+73>

In this case the pyflame process hit an assertion error which caused it to bail out, I believe before restoring the original code value.

[eklitzke]

I think I understand what is happening now.

When I pyflame the target process, that process could already be blocked in a syscall. In this case, the syscall is futex(2), which blocks. The current code fails in this case. It needs to interrupt the syscall, run the mmap syscall, and then resume the prior syscall.

[eklitzke]

I released v1.3.1, which mitigates this when --threads is not used.

[eklitzke]

This was fixed by #61