janestreet / async_unix Goto Github PK

View Code? Open in Web Editor NEW

33.0 33.0 21.0 2.02 MB

Jane Street Capital's asynchronous execution library (unix)

License: MIT License

Makefile 0.04% OCaml 99.34% C 0.60% Standard ML 0.02%

async_unix's People

Contributors

Stargazers

Watchers

async_unix's Issues

Possible deadlock in Async?

I'm running into what seems like a deadlock in Async somewhere.

Begin big copy/paste of what happens when you attach gdb to the deadlocked process and run info threads and bt all of the threads.

(gdb) info threads
  Id   Target Id         Frame
* 1    Thread 0x7fd1befe8780 (LWP 26796) "my_exe" 0x00007fd1bdcf39f3 in futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0x29b06d8 <caml_master_lock+88>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  2    Thread 0x7fd1b6b60700 (LWP 26799) "close" 0x00007fd1bdcf39f3 in futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0x29b06d8 <caml_master_lock+88>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  3    Thread 0x7fd1b635f700 (LWP 26800) "my_exe" 0x00007fd1bd26213f in __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0,
    timeout=0x7fd1b635ee10) at ../sysdeps/unix/sysv/linux/select.c:41
  4    Thread 0x7fd1b5b5e700 (LWP 26801) "read" 0x00007fd1bdcf7394 in __libc_read (fd=fd@entry=0, buf=buf@entry=0x7fd1ac002ea0, nbytes=nbytes@entry=32768)
    at ../sysdeps/unix/sysv/linux/read.c:27
  5    Thread 0x7fd1b535d700 (LWP 26802) "writev" 0x00007fd1bdcf39f3 in futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0x29b06d8 <caml_master_lock+88>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  6    Thread 0x7fd1b4b3b700 (LWP 26803) "writev" 0x00007fd1bdcf39f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fd1ac002bd8)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  7    Thread 0x7fd1a7fff700 (LWP 26804) "stat" 0x00007fd1bdcf39f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fd1ac0232bc)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  8    Thread 0x7fd1a77fe700 (LWP 26805) "mkstemp" 0x00007fd1bdcf39f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fd19c002bac)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  9    Thread 0x7fd1a691e700 (LWP 26806) "writev" 0x00007fd1bdcf39f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fd1b0002bf8)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  10   Thread 0x7fd1a5a10700 (LWP 26807) "openfile" 0x00007fd1bdcf39f3 in futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0x29b06d8 <caml_master_lock+88>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  11   Thread 0x7fd191eff700 (LWP 26808) "fcntl_getfl" 0x00007fd1bdcf39f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fd19c003018)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  12   Thread 0x7fd1a4b66700 (LWP 26828) "writev" 0x00007fd1bdcf39f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x37e6328)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  13   Thread 0x7fd193fff700 (LWP 26829) "writev" 0x00007fd1bdcf39f3 in futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0x29b06d8 <caml_master_lock+88>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  14   Thread 0x7fd1937fe700 (LWP 6148) "writev" 0x00007fd1bdcf39f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x37e6cc8)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  15   Thread 0x7fd192ffd700 (LWP 12163) "fstat" 0x00007fd1bdcf39f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fd194008be8)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  16   Thread 0x7fd1927fc700 (LWP 12164) "fcntl_getfl" 0x00007fd1bdcf39f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fd178000bb8)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:88

Thread 16 (Thread 0x7fd1927fc700 (LWP 12164)):
#0  0x00007fd1bdcf39f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fd178000bb8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fd178000bf0, cond=0x7fd178000b90) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fd178000b90, mutex=0x7fd178000bf0) at pthread_cond_wait.c:655
#3  0x0000000001c61b71 in caml_condition_wait ()
#4  0x000000000184363c in camlSqueue__wait_not_empty_373 () at condition.ml:18
#5  0x00000000018438d1 in camlSqueue__run_511 () at squeue/src/squeue.ml:97
#6  0x0000000001b62cf1 in camlBase__Exn__protectx_306 () at src/exn.ml:57
#7  0x0000000001840fc6 in camlThread_pool__loop_7572 () at thread_pool/src/thread_pool.ml:437
#8  0x00000000018fce06 in camlCore__Core_thread__f_181 () at src/core_thread.ml:16
#9  0x00000000019038a9 in camlThread__anon_fn$5bthread$2eml$3a37$2c4$2d$2d146$5d_55 () at thread.ml:39
#10 0x0000000001c86f42 in caml_start_program ()
#11 0x0000000001c60d8c in caml_thread_start ()
#12 0x00007fd1bdced6db in start_thread (arg=0x7fd1927fc700) at pthread_create.c:463
#13 0x00007fd1bd26ca3f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 15 (Thread 0x7fd192ffd700 (LWP 12163)):
#0  0x00007fd1bdcf39f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fd194008be8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fd194000bb0, cond=0x7fd194008bc0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fd194008bc0, mutex=0x7fd194000bb0) at pthread_cond_wait.c:655
#3  0x0000000001c61b71 in caml_condition_wait ()
#4  0x000000000184363c in camlSqueue__wait_not_empty_373 () at condition.ml:18
#5  0x00000000018438d1 in camlSqueue__run_511 () at squeue/src/squeue.ml:97
#6  0x0000000001b62cf1 in camlBase__Exn__protectx_306 () at src/exn.ml:57
#7  0x0000000001840fc6 in camlThread_pool__loop_7572 () at thread_pool/src/thread_pool.ml:437
#8  0x00000000018fce06 in camlCore__Core_thread__f_181 () at src/core_thread.ml:16
#9  0x00000000019038a9 in camlThread__anon_fn$5bthread$2eml$3a37$2c4$2d$2d146$5d_55 () at thread.ml:39
#10 0x0000000001c86f42 in caml_start_program ()
#11 0x0000000001c60d8c in caml_thread_start ()
#12 0x00007fd1bdced6db in start_thread (arg=0x7fd192ffd700) at pthread_create.c:463
#13 0x00007fd1bd26ca3f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 14 (Thread 0x7fd1937fe700 (LWP 6148)):
#0  0x00007fd1bdcf39f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x37e6cc8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fd1ac023230, cond=0x37e6ca0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x37e6ca0, mutex=0x7fd1ac023230) at pthread_cond_wait.c:655
#3  0x0000000001c61b71 in caml_condition_wait ()
#4  0x000000000184363c in camlSqueue__wait_not_empty_373 () at condition.ml:18
#5  0x00000000018438d1 in camlSqueue__run_511 () at squeue/src/squeue.ml:97
#6  0x0000000001b62cf1 in camlBase__Exn__protectx_306 () at src/exn.ml:57
#7  0x0000000001840fc6 in camlThread_pool__loop_7572 () at thread_pool/src/thread_pool.ml:437
#8  0x00000000018fce06 in camlCore__Core_thread__f_181 () at src/core_thread.ml:16
#9  0x00000000019038a9 in camlThread__anon_fn$5bthread$2eml$3a37$2c4$2d$2d146$5d_55 () at thread.ml:39
#10 0x0000000001c86f42 in caml_start_program ()
#11 0x0000000001c60d8c in caml_thread_start ()
#12 0x00007fd1bdced6db in start_thread (arg=0x7fd1937fe700) at pthread_create.c:463
#13 0x00007fd1bd26ca3f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 13 (Thread 0x7fd193fff700 (LWP 26829)):
#0  0x00007fd1bdcf39f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x29b06d8 <caml_master_lock+88>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x29b0680 <caml_master_lock>, cond=0x29b06b0 <caml_master_lock+48>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x29b06b0 <caml_master_lock+48>, mutex=0x29b0680 <caml_master_lock>) at pthread_cond_wait.c:655
#3  0x0000000001c60e02 in st_masterlock_acquire.constprop ()
#4  0x0000000001c60e49 in caml_thread_leave_blocking_section ()
#5  0x0000000001c6b756 in caml_leave_blocking_section () at signals.c:167
#6  0x0000000001c588cd in writev_in_blocking_section (count=1, iovecs=<optimized out>, v_iovecs=140538696229184, v_fd=<optimized out>) at bigstring_stubs.c:553
#7  bigstring_writev_stub (v_fd=<optimized out>, v_iovecs=<optimized out>, v_count=<optimized out>) at bigstring_stubs.c:562
#8  0x00000000018d94da in camlCore__Bigstring__writev_1234 () at src/bigstring.ml:205
#9  0x00000000017b53e4 in camlAsync_unix__Syscall__loop_68 () at src/syscall.ml:19
#10 0x0000000001b650d5 in camlBase__Result__try_with_499 () at src/result.ml:179
#11 0x00000000017c9432 in camlAsync_unix__In_thread__doit_113 () at src/in_thread.ml:27
#12 0x0000000001841082 in camlThread_pool__loop_7572 () at thread_pool/src/thread_pool.ml:448
#13 0x00000000018fce06 in camlCore__Core_thread__f_181 () at src/core_thread.ml:16
#14 0x00000000019038a9 in camlThread__anon_fn$5bthread$2eml$3a37$2c4$2d$2d146$5d_55 () at thread.ml:39
#15 0x0000000001c86f42 in caml_start_program ()
#16 0x0000000001c60d8c in caml_thread_start ()
#17 0x00007fd1bdced6db in start_thread (arg=0x7fd193fff700) at pthread_create.c:463
#18 0x00007fd1bd26ca3f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 12 (Thread 0x7fd1a4b66700 (LWP 26828)):
#0  0x00007fd1bdcf39f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x37e6328) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fd1ac01af60, cond=0x37e6300) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x37e6300, mutex=0x7fd1ac01af60) at pthread_cond_wait.c:655
#3  0x0000000001c61b71 in caml_condition_wait ()
---Type <return> to continue, or q <return> to quit---
#4  0x000000000184363c in camlSqueue__wait_not_empty_373 () at condition.ml:18
#5  0x00000000018438d1 in camlSqueue__run_511 () at squeue/src/squeue.ml:97
#6  0x0000000001b62cf1 in camlBase__Exn__protectx_306 () at src/exn.ml:57
#7  0x0000000001840fc6 in camlThread_pool__loop_7572 () at thread_pool/src/thread_pool.ml:437
#8  0x00000000018fce06 in camlCore__Core_thread__f_181 () at src/core_thread.ml:16
#9  0x00000000019038a9 in camlThread__anon_fn$5bthread$2eml$3a37$2c4$2d$2d146$5d_55 () at thread.ml:39
#10 0x0000000001c86f42 in caml_start_program ()
#11 0x0000000001c60d8c in caml_thread_start ()
#12 0x00007fd1bdced6db in start_thread (arg=0x7fd1a4b66700) at pthread_create.c:463
#13 0x00007fd1bd26ca3f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 11 (Thread 0x7fd191eff700 (LWP 26808)):
#0  0x00007fd1bdcf39f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fd19c003018) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fd19c00b150, cond=0x7fd19c002ff0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fd19c002ff0, mutex=0x7fd19c00b150) at pthread_cond_wait.c:655
#3  0x0000000001c61b71 in caml_condition_wait ()
#4  0x000000000184363c in camlSqueue__wait_not_empty_373 () at condition.ml:18
#5  0x00000000018438d1 in camlSqueue__run_511 () at squeue/src/squeue.ml:97
#6  0x0000000001b62cf1 in camlBase__Exn__protectx_306 () at src/exn.ml:57
#7  0x0000000001840fc6 in camlThread_pool__loop_7572 () at thread_pool/src/thread_pool.ml:437
#8  0x00000000018fce06 in camlCore__Core_thread__f_181 () at src/core_thread.ml:16
#9  0x00000000019038a9 in camlThread__anon_fn$5bthread$2eml$3a37$2c4$2d$2d146$5d_55 () at thread.ml:39
#10 0x0000000001c86f42 in caml_start_program ()
#11 0x0000000001c60d8c in caml_thread_start ()
#12 0x00007fd1bdced6db in start_thread (arg=0x7fd191eff700) at pthread_create.c:463
#13 0x00007fd1bd26ca3f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 10 (Thread 0x7fd1a5a10700 (LWP 26807)):
#0  0x00007fd1bdcf39f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x29b06d8 <caml_master_lock+88>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x29b0680 <caml_master_lock>, cond=0x29b06b0 <caml_master_lock+48>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x29b06b0 <caml_master_lock+48>, mutex=0x29b0680 <caml_master_lock>) at pthread_cond_wait.c:655
#3  0x0000000001c60e02 in st_masterlock_acquire.constprop ()
#4  0x0000000001c60e49 in caml_thread_leave_blocking_section ()
#5  0x0000000001c6b756 in caml_leave_blocking_section () at signals.c:167
#6  0x0000000001c61b78 in caml_condition_wait ()
#7  0x000000000184363c in camlSqueue__wait_not_empty_373 () at condition.ml:18
#8  0x00000000018438d1 in camlSqueue__run_511 () at squeue/src/squeue.ml:97
#9  0x0000000001b62cf1 in camlBase__Exn__protectx_306 () at src/exn.ml:57
#10 0x0000000001840fc6 in camlThread_pool__loop_7572 () at thread_pool/src/thread_pool.ml:437
#11 0x00000000018fce06 in camlCore__Core_thread__f_181 () at src/core_thread.ml:16
#12 0x00000000019038a9 in camlThread__anon_fn$5bthread$2eml$3a37$2c4$2d$2d146$5d_55 () at thread.ml:39
#13 0x0000000001c86f42 in caml_start_program ()
#14 0x0000000001c60d8c in caml_thread_start ()
#15 0x00007fd1bdced6db in start_thread (arg=0x7fd1a5a10700) at pthread_create.c:463
#16 0x00007fd1bd26ca3f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 9 (Thread 0x7fd1a691e700 (LWP 26806)):
#0  0x00007fd1bdcf39f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fd1b0002bf8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fd1b0002ba0, cond=0x7fd1b0002bd0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fd1b0002bd0, mutex=0x7fd1b0002ba0) at pthread_cond_wait.c:655
#3  0x0000000001c61b71 in caml_condition_wait ()
#4  0x000000000184363c in camlSqueue__wait_not_empty_373 () at condition.ml:18
#5  0x00000000018438d1 in camlSqueue__run_511 () at squeue/src/squeue.ml:97
#6  0x0000000001b62cf1 in camlBase__Exn__protectx_306 () at src/exn.ml:57
#7  0x0000000001840fc6 in camlThread_pool__loop_7572 () at thread_pool/src/thread_pool.ml:437
#8  0x00000000018fce06 in camlCore__Core_thread__f_181 () at src/core_thread.ml:16
#9  0x00000000019038a9 in camlThread__anon_fn$5bthread$2eml$3a37$2c4$2d$2d146$5d_55 () at thread.ml:39
#10 0x0000000001c86f42 in caml_start_program ()
#11 0x0000000001c60d8c in caml_thread_start ()
#12 0x00007fd1bdced6db in start_thread (arg=0x7fd1a691e700) at pthread_create.c:463
#13 0x00007fd1bd26ca3f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 8 (Thread 0x7fd1a77fe700 (LWP 26805)):
#0  0x00007fd1bdcf39f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fd19c002bac) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fd19c002b50, cond=0x7fd19c002b80) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fd19c002b80, mutex=0x7fd19c002b50) at pthread_cond_wait.c:655
#3  0x0000000001c61b71 in caml_condition_wait ()
#4  0x000000000184363c in camlSqueue__wait_not_empty_373 () at condition.ml:18
#5  0x00000000018438d1 in camlSqueue__run_511 () at squeue/src/squeue.ml:97
#6  0x0000000001b62cf1 in camlBase__Exn__protectx_306 () at src/exn.ml:57
#7  0x0000000001840fc6 in camlThread_pool__loop_7572 () at thread_pool/src/thread_pool.ml:437
#8  0x00000000018fce06 in camlCore__Core_thread__f_181 () at src/core_thread.ml:16
#9  0x00000000019038a9 in camlThread__anon_fn$5bthread$2eml$3a37$2c4$2d$2d146$5d_55 () at thread.ml:39
#10 0x0000000001c86f42 in caml_start_program ()
#11 0x0000000001c60d8c in caml_thread_start ()
#12 0x00007fd1bdced6db in start_thread (arg=0x7fd1a77fe700) at pthread_create.c:463
#13 0x00007fd1bd26ca3f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7fd1a7fff700 (LWP 26804)):
#0  0x00007fd1bdcf39f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fd1ac0232bc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fd1ac023260, cond=0x7fd1ac023290) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fd1ac023290, mutex=0x7fd1ac023260) at pthread_cond_wait.c:655
#3  0x0000000001c61b71 in caml_condition_wait ()
#4  0x000000000184363c in camlSqueue__wait_not_empty_373 () at condition.ml:18
#5  0x00000000018438d1 in camlSqueue__run_511 () at squeue/src/squeue.ml:97
#6  0x0000000001b62cf1 in camlBase__Exn__protectx_306 () at src/exn.ml:57
#7  0x0000000001840fc6 in camlThread_pool__loop_7572 () at thread_pool/src/thread_pool.ml:437
#8  0x00000000018fce06 in camlCore__Core_thread__f_181 () at src/core_thread.ml:16
#9  0x00000000019038a9 in camlThread__anon_fn$5bthread$2eml$3a37$2c4$2d$2d146$5d_55 () at thread.ml:39
#10 0x0000000001c86f42 in caml_start_program ()
#11 0x0000000001c60d8c in caml_thread_start ()
#12 0x00007fd1bdced6db in start_thread (arg=0x7fd1a7fff700) at pthread_create.c:463
#13 0x00007fd1bd26ca3f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7fd1b4b3b700 (LWP 26803)):
#0  0x00007fd1bdcf39f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fd1ac002bd8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fd1ac002b80, cond=0x7fd1ac002bb0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fd1ac002bb0, mutex=0x7fd1ac002b80) at pthread_cond_wait.c:655
#3  0x0000000001c61b71 in caml_condition_wait ()
#4  0x000000000184363c in camlSqueue__wait_not_empty_373 () at condition.ml:18
#5  0x00000000018438d1 in camlSqueue__run_511 () at squeue/src/squeue.ml:97
#6  0x0000000001b62cf1 in camlBase__Exn__protectx_306 () at src/exn.ml:57
#7  0x0000000001840fc6 in camlThread_pool__loop_7572 () at thread_pool/src/thread_pool.ml:437
#8  0x00000000018fce06 in camlCore__Core_thread__f_181 () at src/core_thread.ml:16
#9  0x00000000019038a9 in camlThread__anon_fn$5bthread$2eml$3a37$2c4$2d$2d146$5d_55 () at thread.ml:39
#10 0x0000000001c86f42 in caml_start_program ()
#11 0x0000000001c60d8c in caml_thread_start ()
#12 0x00007fd1bdced6db in start_thread (arg=0x7fd1b4b3b700) at pthread_create.c:463
#13 0x00007fd1bd26ca3f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7fd1b535d700 (LWP 26802)):
#0  0x00007fd1bdcf39f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x29b06d8 <caml_master_lock+88>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x29b0680 <caml_master_lock>, cond=0x29b06b0 <caml_master_lock+48>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x29b06b0 <caml_master_lock+48>, mutex=0x29b0680 <caml_master_lock>) at pthread_cond_wait.c:655
#3  0x0000000001c60e02 in st_masterlock_acquire.constprop ()
#4  0x0000000001c60e49 in caml_thread_leave_blocking_section ()
#5  0x0000000001c6b756 in caml_leave_blocking_section () at signals.c:167
#6  0x0000000001c588cd in writev_in_blocking_section (count=1, iovecs=<optimized out>, v_iovecs=140538696227504, v_fd=<optimized out>) at bigstring_stubs.c:553
#7  bigstring_writev_stub (v_fd=<optimized out>, v_iovecs=<optimized out>, v_count=<optimized out>) at bigstring_stubs.c:562
#8  0x00000000018d94da in camlCore__Bigstring__writev_1234 () at src/bigstring.ml:205
#9  0x00000000017b53e4 in camlAsync_unix__Syscall__loop_68 () at src/syscall.ml:19
#10 0x0000000001b650d5 in camlBase__Result__try_with_499 () at src/result.ml:179
#11 0x00000000017c9432 in camlAsync_unix__In_thread__doit_113 () at src/in_thread.ml:27
#12 0x0000000001841082 in camlThread_pool__loop_7572 () at thread_pool/src/thread_pool.ml:448
#13 0x00000000018fce06 in camlCore__Core_thread__f_181 () at src/core_thread.ml:16
#14 0x00000000019038a9 in camlThread__anon_fn$5bthread$2eml$3a37$2c4$2d$2d146$5d_55 () at thread.ml:39
#15 0x0000000001c86f42 in caml_start_program ()
#16 0x0000000001c60d8c in caml_thread_start ()
#17 0x00007fd1bdced6db in start_thread (arg=0x7fd1b535d700) at pthread_create.c:463
#18 0x00007fd1bd26ca3f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7fd1b5b5e700 (LWP 26801)):
#0  0x00007fd1bdcf7394 in __libc_read (fd=fd@entry=0, buf=buf@entry=0x7fd1ac002ea0, nbytes=nbytes@entry=32768) at ../sysdeps/unix/sysv/linux/read.c:27
#1  0x0000000001c57466 in read (__nbytes=32768, __buf=0x7fd1ac002ea0, __fd=0) at /usr/include/x86_64-linux-gnu/bits/unistd.h:44
#2  bigstring_read_stub (v_min_len=<optimized out>, v_fd=<optimized out>, v_pos=<optimized out>, v_len=<optimized out>, v_bstr=140537421550048) at bigstring_stubs.c:136
#3  0x00000000018d8ad6 in camlCore__Bigstring__read_inner_619 () at src/bigstring.ml:40
#4  0x0000000001807a71 in camlAsync_unix__Reader0__anon_fn$5bsrc$2freader0$2eml$3a299$2c49$2d$2d162$5d_4049 () at src/reader0.ml:300
#5  0x00000000017b53e4 in camlAsync_unix__Syscall__loop_68 () at src/syscall.ml:19
#6  0x0000000001b650d5 in camlBase__Result__try_with_499 () at src/result.ml:179
#7  0x00000000017c9432 in camlAsync_unix__In_thread__doit_113 () at src/in_thread.ml:27
#8  0x0000000001841082 in camlThread_pool__loop_7572 () at thread_pool/src/thread_pool.ml:448
#9  0x00000000018fce06 in camlCore__Core_thread__f_181 () at src/core_thread.ml:16
#10 0x00000000019038a9 in camlThread__anon_fn$5bthread$2eml$3a37$2c4$2d$2d146$5d_55 () at thread.ml:39
#11 0x0000000001c86f42 in caml_start_program ()
#12 0x0000000001c60d8c in caml_thread_start ()
#13 0x00007fd1bdced6db in start_thread (arg=0x7fd1b5b5e700) at pthread_create.c:463
#14 0x00007fd1bd26ca3f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7fd1b635f700 (LWP 26800)):
#0  0x00007fd1bd26213f in __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7fd1b635ee10) at ../sysdeps/unix/sysv/linux/select.c:41
---Type <return> to continue, or q <return> to quit---
#1  0x0000000001c60929 in caml_thread_tick ()
#2  0x00007fd1bdced6db in start_thread (arg=0x7fd1b635f700) at pthread_create.c:463
#3  0x00007fd1bd26ca3f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7fd1b6b60700 (LWP 26799)):
#0  0x00007fd1bdcf39f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x29b06d8 <caml_master_lock+88>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x29b0680 <caml_master_lock>, cond=0x29b06b0 <caml_master_lock+48>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x29b06b0 <caml_master_lock+48>, mutex=0x29b0680 <caml_master_lock>) at pthread_cond_wait.c:655
#3  0x0000000001c60e02 in st_masterlock_acquire.constprop ()
#4  0x0000000001c60e49 in caml_thread_leave_blocking_section ()
#5  0x0000000001c6b756 in caml_leave_blocking_section () at signals.c:167
#6  0x0000000001c61b78 in caml_condition_wait ()
#7  0x000000000184363c in camlSqueue__wait_not_empty_373 () at condition.ml:18
#8  0x00000000018438d1 in camlSqueue__run_511 () at squeue/src/squeue.ml:97
#9  0x0000000001b62cf1 in camlBase__Exn__protectx_306 () at src/exn.ml:57
#10 0x0000000001840fc6 in camlThread_pool__loop_7572 () at thread_pool/src/thread_pool.ml:437
#11 0x00000000018fce06 in camlCore__Core_thread__f_181 () at src/core_thread.ml:16
#12 0x00000000019038a9 in camlThread__anon_fn$5bthread$2eml$3a37$2c4$2d$2d146$5d_55 () at thread.ml:39
#13 0x0000000001c86f42 in caml_start_program ()
#14 0x0000000001c60d8c in caml_thread_start ()
#15 0x00007fd1bdced6db in start_thread (arg=0x7fd1b6b60700) at pthread_create.c:463
#16 0x00007fd1bd26ca3f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7fd1befe8780 (LWP 26796)):
#0  0x00007fd1bdcf39f3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x29b06d8 <caml_master_lock+88>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x29b0680 <caml_master_lock>, cond=0x29b06b0 <caml_master_lock+48>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x29b06b0 <caml_master_lock+48>, mutex=0x29b0680 <caml_master_lock>) at pthread_cond_wait.c:655
#3  0x0000000001c60e02 in st_masterlock_acquire.constprop ()
#4  0x0000000001c60e49 in caml_thread_leave_blocking_section ()
#5  0x0000000001c6b756 in caml_leave_blocking_section () at signals.c:167
#6  0x0000000001c617e3 in caml_thread_yield ()
#7  0x00000000017c79e9 in camlAsync_unix__Raw_scheduler__check_file_descr_watcher_8000 () at thread.ml:26
#8  0x00000000017c7faf in camlAsync_unix__Raw_scheduler__one_iter_8380 () at src/raw_scheduler.ml:865
#9  0x00000000017c832d in camlAsync_unix__Raw_scheduler__loop_8428 () at src/raw_scheduler.ml:881
#10 0x00000000017c80e9 in camlAsync_unix__Raw_scheduler__be_the_scheduler_inner_8415 () at src/raw_scheduler.ml:885
#11 0x00000000015badaa in camlMy_foo__anon_fn$5basync_command$2fsrc$2fasync_command$2eml$3a46$2c21$2d$2d1183$5d_59271 () at async_command/src/async_command.ml:75
#12 0x00000000015d1769 in camlMy_foo__anon_fn$5bsrc$2fcommand$2eml$3a2931$2c33$2d$2d659$5d_19757 () at src/command.ml:2937
#13 0x00000000015cd195 in camlMy_foo__run_3502 () at src/exn.ml:102
#14 0x00000000010a1b58 in camlDune__exe__my_exe__entry () at my_exe.ml:2
#15 0x0000000001090a09 in caml_program ()
#16 0x0000000001c86f42 in caml_start_program ()
#17 0x0000000001c6a3c5 in caml_startup_common (argv=0x7ffc6c461788, pooling=<optimized out>, pooling@entry=0) at startup_nat.c:160
#18 0x0000000001c6a41b in caml_startup_exn (argv=<optimized out>) at startup_nat.c:165
#19 caml_startup (argv=<optimized out>) at startup_nat.c:170
#20 0x0000000001c6a442 in caml_main (argv=<optimized out>) at startup_nat.c:177
#21 0x000000000108e1dc in main (argc=<optimized out>, argv=<optimized out>) at main.c:44

This is compiled against jane v0.13.0, OCaml 4.08 flambda.

It's in a long-running process and the deadlock happens pretty unpredictably so I'm a bit slow to iterate the debugging.

The most interesting part is the program is long-running but watches stdin for input using a Reader.read_line and when the deadlock happens, the only thread that's not blocked on waiting for the OCaml runtime lock is that Reader.read_line. Instead, it's blocked in libc read, and hitting Enter actually resumes everything.

Working backwards, that libc read appears to be Bigstring.read, suggesting it's both blocked in a libc read and holding the runtime lock. Though, looking at the code, it's not obvious how it could actually acquire the lock but still block in read. This code seems pretty tight:

CAMLprim value bigstring_read_stub(
  value v_min_len, value v_fd, value v_pos, value v_len, value v_bstr)
{
  CAMLparam1(v_bstr);
    size_t min_len = Long_val(v_min_len);
    int fd = Int_val(v_fd);
    size_t init_len = Long_val(v_len);
    size_t len = init_len;
    ssize_t n_read;
    char *bstr_start = get_bstr(v_bstr, v_pos);
    char *bstr = bstr_start;
    char *bstr_min = bstr_start + min_len;
    caml_enter_blocking_section();
      do {
        do {
          n_read = read(fd, bstr, len);
        } while (n_read == -1 && errno == EINTR);
        if (n_read <= 0) {
          value v_n_good = Val_long(bstr - bstr_start);
          caml_leave_blocking_section();
          if (n_read == 0) {
            if (init_len == 0) CAMLreturn(Val_long(0));
            else raise_eof_io_error(v_n_good);
          }
          else raise_unix_io_error(v_n_good, "read", Nothing);
        } else {
          bstr += n_read;
          len -= n_read;
        }
      } while (bstr < bstr_min);
    caml_leave_blocking_section();
  CAMLreturn(Val_long(bstr - bstr_start));
}

As said above, hitting Enter will actually resume all of the threads that are blocked waiting on the OCaml runtime/masterlock waits. Confirmed it by looking at the call stacks again in gdb.

Speculating wildly: You could imagine contriving a situation where the inner while loop exits due to some arcane errno condition, the blocking section is left (acquiring the runtime lock), but the raises somehow don't terminate the C function and it instead nexts the outer loop. Then the read is re-entered while holding the runtime lock, and the only way out is to press Enter (assuming the same thread can call leave blocking section twice). Though that would be really contrived.

I don't mean to overly implicate what must be a very well worn and tested Bigstring.read, just seems like the only lead so far.

Anyway, to make this even more strange, these deadlocks don't appear at all on Ubuntu 16.04 but do appear on Ubuntu 18.04.

Thanks for any insight you can offer.

Exceed the maximum number of file descriptor

There must be a bug in my program (filedescriptor leaking), however the error I obtained in this case seems quite uninformative. Perhaps the problem should be caught before, with a gentler error message.

((exn
  ("key's index out of range" (8192 8192 (Should_be_between_0_and 8191))
   /home/oci/.opam/4.02.3/build/core_kernel.113.33.01/_build/src/bounded_int_table.ml:176:4))
 (backtrace
  ("Raised at file \"src/error.ml\", line 7, characters 20-30"
   "Called from file \"src/bounded_int_table.ml\", line 225, characters 8-23"
   "Called from file \"src/fd_by_descr.ml\", line 34, characters 8-47"
   "Called from file \"src/raw_scheduler.ml\", line 146, characters 2-38"
   "Called from file \"src/deferred1.ml\", line 14, characters 63-68"
   "Called from file \"src/job_queue.ml\", line 160, characters 6-47" ""))

Signal.handle terminating

On the new Async working on 4.03.0, my code which was previously using Signal.handle terminating to do some effects is not working anymore, i.e. the code supposed to run on interruption and other termination case is not run anymore.

Writer.save fails on empty contents

I tried creating an empty file using Writer.save, but it fails. I can work around it by using any of several other methods, but curious why save doesn't work with empty contents. Is it intentional or is this a bug?

utop # Writer.save "foo.txt" ~contents:"";;
Exception:                                                                                                                              (src/monitor.ml.Error_
 ((exn                                                                                                                                  
   ("writer error"
    ((src/monitor.ml.Error_
      ((exn (Unix.Unix_error "Invalid argument" writev "")) (backtrace (""))
       (monitor
        (((name (id 118)) (here ()) (id 118) (has_seen_error true)
          (is_detached true) (kill_index 0))
         ((name block_on_async) (here ()) (id 115) (has_seen_error false)
          (is_detached true) (kill_index 0))))))
     ((id 9)
      (fd
       ((file_descr 46) (info foo.txtPlkeyr) (kind File)
        (supports_nonblock false) (have_set_nonblock false)
        (state (Open Empty))
        (watching ((read Not_watching) (write Not_watching)))
        (watching_has_changed false) (num_active_syscalls 1)
        (close_finished Empty)))
      (monitor
       (((name (id 117)) (here ()) (id 117) (has_seen_error true)
         (is_detached false) (kill_index 0))
        ((name block_on_async) (here ()) (id 115) (has_seen_error false)
         (is_detached true) (kill_index 0))))
      (inner_monitor
       (((name (id 118)) (here ()) (id 118) (has_seen_error true)
         (is_detached true) (kill_index 0))
        ((name block_on_async) (here ()) (id 115) (has_seen_error false)
         (is_detached true) (kill_index 0))))
      (background_writer_state Stopped_permanently) (syscall Per_cycle)
      (bytes_received 0) (bytes_written 0) (scheduled <opaque>)
      (scheduled_bytes 0) (buf <opaque>) (scheduled_back 0) (back 0)
      (flushes <opaque>) (close_state Open) (close_finished Empty)
      (close_started Empty) (producers_to_flush_at_close ())
      (flush_at_shutdown_elt (<opaque>)) (check_buffer_age ())
      (consumer_left Empty) (raise_when_consumer_leaves true)
      (open_flags (Full (Ok (rdwr))))))))
  (monitor
   (((name (id 117)) (here ()) (id 117) (has_seen_error true)
     (is_detached false) (kill_index 0))
    ((name block_on_async) (here ()) (id 115) (has_seen_error false)
     (is_detached true) (kill_index 0)))))).

Helper threads never finalize

It appears that when using In_thread.Helper_thread, in spite of what is indicated by the docs:

create uses a thread from Async's thread pool, reserving that thread for exclusive use by the helper thread until the helper thread is no longer used (specifically, finalized and is finished with all its work), at which point the thread is made available for general use by the pool.

helper threads are in fact never finalized and never made available for general use by the pool.

The reason for this seems to be that Thread_pool.Thread.t's type state, when in Helper mode, references the Helper_thread.t (and thus the Helper_thread cannot be finalized), and the only way that a Thread can leave Helper mode is through maybe_finish_helper_thread, which only does something if the state of the helper thread is Finishing, which is only the case if finished_with_helper_thread is called, which only happens when the In_thread.Helper_thread is finalized (which it can't be because it's in Helper mode, a catch-22).

If In_thread.Helper_thread exposed a call to finished_with_helper_thread, then the user could at least manually free up the thread. Or you can work around it by using Thread_pool directly instead of using In_thread, but then you have to reimplement In_thread.run, a non-trivial function. Otherwise, there is probably a way around the catch so that it gets properly gc'd but I don't have a solution right off.

Unix_syscalls.readdir returns a string and raises on end-of-dir

True to the Unix.readdir interface and readdir(3) before it, Unix_syscalls.readdir returns but a string and raises End_of_file when the directory handle is exhausted. To use it correctly, the programmer must write something like:

try_with ~extract_exn:true (fun () -> Unix_syscalls.readdir dh)
>>= function
| Ok dirent -> next (dirent::listing)
| Error End_of_file -> return listing
| Error exn -> raise exn

This is particularly bad because ~extract_exn:true must be passed to try_with or the end-of-directory condition is effectively un-catchable as the wrapped exception that is raised is not exposed in an interface (as far as I could tell). This seems like fairly poor ergonomics that should either be corrected or very clearly documented as legacy.

A final minor issue is the uncertainty about the | Error exn -> raise exn clause which, from an LWT programmer's point of view, looks wrong because the exception is not injected back into the monad and instead a try .. catch is necessary inside bind or similar to make sure that these truly exceptional cases are handled correctly and passed upward mostly un-modified. Of course, some metadata is probably lost here but I'll let you be the judge of how horrible that is.

Reading Deferreds From Multi-Process Pipe Causes Segmentation Fault

We ran across a problem where we were sending a Deferred over a TCP socket-backed Pipe between two processes. The write worked OK, but the read on the other end of the pipe caused a Segmentation Fault.

I'm 90% convinced this should never actually work. Sending a Deferred and expecting the other end to know how to resolve it seems too much to ask. But it shouldn't seg fault either.

I can construct a simple test case, if needed, but it might take a few days.

Ignore SIGPIPE by default?

Right now SIGPIPE exceptions can leak out and kill the whole server process in Cohttp_async. There's no real reason to not ignore SIGPIPE these days, since the next read/write will result in an error in the return code anyway.

Can it be blocked globally when linking Async?

("unhandled exception"
 ((lib/monitor.ml.Error_
   ((exn (Unix.Unix_error "Broken pipe" writev_assume_fd_is_nonblocking ""))
    (backtrace
     ("Raised by primitive operation at file \"lib/bigstring.ml\", line 275, characters 2-56"
      "Called from file \"lib/result.ml\", line 101, characters 10-14"
      "Re-raised at file \"lib/result.ml\", line 108, characters 23-26"
      "Called from file \"lib/raw_fd.ml\", line 261, characters 11-25"
      "Re-raised at file \"lib/writer0.ml\", line 578, characters 49-52"
      "Called from file \"lib/scheduler.ml\", line 139, characters 6-17"
      "Called from file \"lib/jobs.ml\", line 65, characters 8-13" ""))
    (monitor
     (((name
        (writer
         (3
          ((file_descr 6)
           (info
            (socket
             ((listening_on
               ((type_
                 ((family
                   ((family PF_INET) (address_of_sockaddr_exn <fun>)
                    (sexp_of_address <fun>)))
                  (socket_type SOCK_STREAM)))
                (fd
                 ((file_descr 3)
                  (info
                   (replaced
                    (listening
                     (previously_was
                      (replaced
                       ((socket (bound_on (Inet (0.0.0.0 8080))))
                        (previously_was
                         (socket
                          ((family
                            ((family PF_INET) (address_of_sockaddr_exn <fun>)
                             (sexp_of_address <fun>)))
                           (socket_type SOCK_STREAM))))))))))
                  (kind (Socket Passive)) (supports_nonblock true)
                  (have_set_nonblock true) (state (Close_requested <fun>))
                  (watching ((read Stop_requested) (write Not_watching)))
                  (watching_has_changed true) (num_active_syscalls 1)
                  (close_finished Empty)))))
              (client (Inet (127.0.0.1 50021))))))
           (kind (Socket Active)) (supports_nonblock true)
           (have_set_nonblock true) (state Closed)
           (watching ((read Not_watching) (write Not_watching)))
           (watching_has_changed false) (num_active_syscalls 0)
           (close_finished (Full ()))))))
       (here ()) (id 52) (has_seen_error true) (someone_is_listening true)
       (kill_index 0))
      ((name main) (here ()) (id 1) (has_seen_error true)
       (someone_is_listening false) (kill_index 0))))))
  (Pid 79172)))

Unix.File_kind.t doesnt have sexp exposed

The definition in lib/unix_syscalls.ml has a with sexp, but the signature doesnt:

module File_kind : sig
  type t = [ `File | `Directory | `Char | `Block | `Link | `Fifo | `Socket ]
end

Could this be exposed so that it's easier to print File_kinds?

Extracting exception raised from inner monitor in writer

In writer0.ml line 439, exceptions raised in the inner monitor are wrapped into Error. Is there a way to extract the original exception or handle this exception in another way? Error seems to be there only to provide human-readable messages.

I would like to ignore broken pipes exceptions like this:

(src/monitor.ml.Error_
  ((exn
     ("writer error"
       ((src/monitor.ml.Error_
          ((exn (Unix.Unix_error "Broken pipe" writev "")) (backtrace (""))
            (monitor
              (((name (id 39)) (here ()) (id 39) (has_seen_error true)
[…]

Assert failure in Writer0.add_iovec

Unfortunately our logs format this in an annoying way, but we're getting errors like this relatively often and I'm not sure what's going on or how we should track this down:

  | Sep 16 05:02:02.670 | ((pid 8) (thread_id 2)))))
  | Sep 16 05:02:02.670 | ("Caught by monitor (id 2)"))
  | Sep 16 05:02:02.670 | (open_flags (Full (Ok (wronly)))) (line_ending Unix))))
  | Sep 16 05:02:02.670 | (consumer_left Empty) (raise_when_consumer_leaves true)
  | Sep 16 05:02:02.670 | (bytes_seen 60899842) (too_old Empty))))
  | Sep 16 05:02:02.670 | (bytes_received_queue ()) (times_received_queue ())
  | Sep 16 05:02:02.670 | (bytes_received_at_now_minus_maximum_age 0)
  | Sep 16 05:02:02.670 | (((writer <opaque>) (maximum_age 2m)
  | Sep 16 05:02:02.670 | (check_buffer_age
  | Sep 16 05:02:02.670 | (num_producers_to_flush_at_close 0) (flush_at_shutdown_elt (<opaque>))
  | Sep 16 05:02:02.670 | (close_state Open) (close_finished Empty) (close_started Empty)
  | Sep 16 05:02:02.670 | (scheduled_bytes 272) (scheduled_back 272) (back 1073)
  | Sep 16 05:02:02.670 | (syscall Per_cycle) (bytes_received 61206728) (bytes_written 61205655)
  | Sep 16 05:02:02.670 | (background_writer_state Running) (background_writer_stopped Empty)
  | Sep 16 05:02:02.670 | (is_detached false))))
  | Sep 16 05:02:02.670 | ((name main) (here ()) (id 1) (has_seen_error false)
  | Sep 16 05:02:02.670 | (is_detached true))
  | Sep 16 05:02:02.670 | (((name (id 3)) (here ()) (id 3) (has_seen_error true)
  | Sep 16 05:02:02.670 | (inner_monitor
  | Sep 16 05:02:02.670 | (is_detached false))))
  | Sep 16 05:02:02.670 | ((name main) (here ()) (id 1) (has_seen_error false)
  | Sep 16 05:02:02.670 | (is_detached false))
  | Sep 16 05:02:02.670 | (((name (id 2)) (here ()) (id 2) (has_seen_error false)
  | Sep 16 05:02:02.670 | (monitor
  | Sep 16 05:02:02.670 | (close_finished Empty)))
  | Sep 16 05:02:02.670 | (watching_has_changed false) (num_active_syscalls 0)
  | Sep 16 05:02:02.670 | (watching ((read Not_watching) (write Not_watching)))
  | Sep 16 05:02:02.670 | (have_set_nonblock false) (state (Open Empty))
  | Sep 16 05:02:02.670 | ((file_descr 1) (info <stdout>) (kind Fifo) (supports_nonblock false)
  | Sep 16 05:02:02.670 | (fd
  | Sep 16 05:02:02.670 | ((id 0)
  | Sep 16 05:02:02.670 | (writer
  | Sep 16 05:02:02.670 | ("Writer error from inner_monitor" "Assert_failure src/writer0.ml:834:2"
  | Sep 16 05:02:02.670 | ((monitor.ml.Error
  | Sep 16 05:02:02.670 | ("unhandled exception"
  | Sep 16 05:02:02.670 | "unhandled exception in Async scheduler"
  | Sep 16 05:02:02.670 | (((pid 8) (thread_id 0)) "2019-09-16 09:02:02.668802059Z"

The assert on line 834 is https://github.com/janestreet/async_unix/blob/v0.11.0/src/writer0.ml#L834

let add_iovec t kind (iovec : _ IOVec.t) ~count_bytes_as_received =
  assert (t.scheduled_back = t.back);
  if count_bytes_as_received then (got_bytes t iovec.len);
  if not (is_stopped_permanently t)
  then (
    t.scheduled_bytes <- t.scheduled_bytes + iovec.len;
    Deque.enqueue_back t.scheduled (iovec, kind));
  assert (t.scheduled_back = t.back); (* <--- here *)

We're using v0.11 but this function seems unchanged since then. I could try moving to a later version if you think that would help, although the base64 stuff is going to make that really difficult for us :(

Any ideas what's wrong? Since it's an assert I'm guessing this is supposed to be impossible, but we could also be doing something wrong.

A way to set thread-global tags?

In some cases, we want to tag every log message with things like a process ID, a request ID, request path, user name, etc. I was hoping I could do this by setting thread-global tags with Scheduler.with_local, then looking them up inside my custom Output. Unfortunately, it seems like outputs run in a different Async context so they can’t find the tags.

I can think of two ways Async.Log could support our use-case:

Pass the async context or thread-local universal map to log outputs
Add a way to mutate messages as soon as they're logged, independent of Output

(1) solves exactly our use-case, but (2) is probably simpler, more performant, and also more general-purpose.

As an example, it would be nice if I could do something like:

open Async

let tags_key = Univ_map.Key.create ~name:"tags" [%sexp_of: (string * string) list]

let () =
  Log.add_rewriter (fun msg ->
    match Scheduler.find_local tag_key with
    | None -> msg
    | Some tags ->
      Log.Message.add_tags message tags);
  Scheduler.with_local tags_key ["example", "tag"] ~f:(fun () ->
    Log.info "Some message") (* will log "Some message" with tags ["example, "tag"] *)

See https://discuss.ocaml.org/t/thread-global-tags-in-async-log/2739/3

blocking_infer_using_stat doesn't work with Unix domain sockets

Hey!

Code to reproduce:

open Core
open Async

let () =
  let doit () =
    let fd = Unix.Socket.create Unix.Socket.Type.unix
             |> Unix.Socket.fd
             |> Unix.Fd.file_descr_exn in
    Unix.Fd.Kind.infer_using_stat fd |> ignore
  in
  doit () |> ignore;
  Scheduler.go () |> never_returns

Result:

(((pid 57699) (thread_id 0)) "2018-08-15 13:29:29.288067Z"
 "unhandled exception in Async scheduler"
 ("unhandled exception"
  ((monitor.ml.Error
    (Unix.Unix_error "Protocol not available" getsockopt
     "((fd 3) (opt SO_ACCEPTCONN))")
    ("Raised at file \"src/import0.ml\" (inlined), line 351, characters 22-32"
     "Called from file \"src/result.ml\", line 168, characters 17-26"
     "Called from file \"src/deferred1.ml\", line 20, characters 40-45"
     "Called from file \"src/job_queue.ml\", line 159, characters 6-47"
     "Caught by monitor main"))
   ((pid 57699) (thread_id 1)))))

OS: macOS 10.13.4
Ocaml: 4.05.0

The problem is here https://github.com/janestreet/async_unix/blob/master/src/fd.ml#L31.

It means that we can't infer Unix.Fd from Unix domain sockets. Nodejs, for example, uses Unix domain sockets for stdio with child processes by default (so things like force Reader.stdin fails).

Code to reproduce with nodejs:

ocaml:

open Core
open Async

let () =
  force Reader.stdin |> ignore

JS:

const cp = require('child_process');

let res = cp.spawnSync("/path/to/binary");

console.log(res.stderr.toString());

file_exists should take ?follow_symlinks arg

file_exists and files_exists_exn should support the ?follow_symlinks optional arg as in Core. The fix is to define these functions using wrap_is instead of wrap1.

How to use Thread_safe_pipe?

I'm trying to upgrade to v0.12 but I can't figure out how to access Thead_safe_pipe. There doesn't seem to be an Async_unix.Thread_safe_pipe, and Thread_safe_pipe isn't a global. I tried linking a library named thread_safe_pipe but that doesn't exist.

Is this not released or am I doing something wrong?

Bind address wrongly set to ipv4 in case of ipv6

It is not possible to connect to an ipv6 address using Where_to_connect when specifying no bind address. It errors out with Address family not supported by protocol. This seems to happen because create_bind_any always uses ipv4:

async_unix/src/unix_syscalls.ml

Line 650 in cc9687d

let create_bind_any ~port = `Inet (Inet_addr.of_string "0.0.0.0", port)

It should instead detect when ipv6 is used and change the address appropriately.

Sequentialized DNS lookup implementation

At line 237 of unix_syscalls.mli, seems that DNS lookup has been sequentialized due to some thread safety issue.

I talked to Yaron in the IRC and he said

The winbind issue looks like an old one to me. I think we don’t rely on winbind anymore, but I’m not sure. In any case, we can probably make the degree of parallelism tuneable.

Is it possible to make some improvement on it?

Thanks

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.