sustrik / libdill

Structured concurrency in C

libdill: Structured Concurrency for C

Documentation

For the documentation check the project website:

http://libdill.org

License

Libdill is licensed under MIT/X11 license.

libdill's Issues

dill_prologue checks for dill_allocstack failure incorrectly

I built and ran ./perf/go 1 and it segfaulted.

It looks like this is because dill_prologue subtracts from the result of dill_allocstack, and then checks for NULL, when it should be the other way around.

As for why I ran out of memory, I think it is because perf/go.c is not closing the handles returned from go(worker()), so none of the stacks get reused.

reading strips from tiff file

I have this method[1] that can read individual strips from a tiff file.
[1] http://www.libtiff.org/man/TIFFReadEncodedStrip.3t.html

Is there a way to use libdill to read multiple strips at the same time?
Not sure having more cpu cores will help this to achieve concurrency

[Bug] Stack unwinding fails in assembly context switching

The stack unwinding in the assembly implementation is broken; stack traces lose information usually past the last dill_wait call.

compile warning in macOS - inerr is defined in struct crlf_sock as 1 bit of unsigned int but the code use -1 which is invalid

crlf.c:193:43: warning: implicit truncation from 'int' to bitfield changes value from -1 to 1
      [-Wbitfield-constant-conversion]
        if(dill_slow(rc < 0)) {obj->inerr = -1; return -1;}
                                          ^ ~~

GC mechanism like Go's defer

After coroutine exited, mechanisms like this are very useful for clean resouce like coroutine handle etc.
Have Any plan for this?

Compilation error on MacOSX 10.11, libdill0.9-beta

I've been using libdill-0.5 for sometime with no problem, then I tried to switch libdill-0.9-beta to give it a try. However, my project started complaining about Conflicting types for 'hcreate'. These are the conflicting types. I'm using XCode to compile my project on MacOSX 10.11 w/ clang

Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1 Apple LLVM version 7.0.2 (clang-700.1.81) Target: x86_64-apple-darwin14.5.0 Thread model: posix

Previous declaration (as shown by Xcode) in /usr/include/search.h:
__BEGIN_DECLS int hcreate(size_t);

Conflicting declaration from libdill.h:
DILL_EXPORT int hcreate(struct hvfs *vfs);

Any ideas?

[Experiment] Coroutine Closures

Implemented coroutine closures using GCC's nested-functions and Clang's -fblocks.

raedwulf@351e8ff

Pros:

1. Works brilliantly with GCC.
2. It is possible to force the closure call not to be inlined at the call-site!
3. Performance is comparable, if you now force inlining on the previously non-inlined methods.
4. The compiler is a lot more predictable with closures and you can now declare the coroutine contents within go() without breaking your code horribly:

int cr1 = go(
         int i;
         for(i = 0; i != 3; ++i) {
             sum += 7;
             int rc = yield();
             assert(rc == 0);
         }
     );

Cons:

1. Totally non-standard, but then statement expressions are not standard either.
2. Clang needs compiler flag -fblocks and needs https://github.com/mackyle/blocksruntime if building on non-OSX.
3. Clang does not support accessing arrays within the closure.
4. Performance of coroutine creation and context switching roughly halves loses 20-30% when inlining of the functions in closures isn't performed on both GCC and Clang when compared with the non-closure method.

Undocumented special meaning of passing 0 to deadline

I noticed in chan.c you assume that 0 in deadline returns immediately.
This is incorrect because the parameter is deadline and 0 should be treated as a wrap-around time-value especially since we use CLOCK_MONOTONIC on linux which could use any arbitrary starting point.

In practice, bugs resulting from this are unlikely. We should, however, either remove this behaviour and tell the user to use now(), or document this behaviour and change the implementation now() to start at timestamp 0 (record application start time at initialisation and subtract).

now() should be documented that it does not return a time value that is valid across instances of libdill cross-process. Although it can still be used it within multiple contexts of the same process.
Maybe a compromise is to provide a easy-to-use unix-time conversion function (at millisecond range).

[Bug] Segfault in 3th go_mem() call

When using the go_mem() function to start coroutines, it consistently fails with a segmentation fault after the 2nd time:

#include <libdill.h>
#include <assert.h>
#include <errno.h>
#include <string.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

#define errno_assert(x) \
    do {\
        if(!(x)) {\
            fprintf(stderr, "%s [%d] (%s:%d)\n", strerror(errno),\
                (int)errno, __FILE__, __LINE__);\
            fflush(stderr);\
            abort();\
        }\
    } while(0)

coroutine void fun() {
    msleep(now() + 1000);
}

// Resource cleanup omitted for brevity
int main(int argc, char *argv[]) {
    int cnt = 0;
    while(1) {
        void *stk = malloc(16384);
        errno_assert(stk != NULL);

        int cr = go_mem(fun(), stk, 16384);
        printf("%d\n", ++cnt);
        errno_assert(cr >= 0);
    }
}

Output:

> ./go_mem_fail 
1
2
fish: “./go_mem_fail” terminated by signal SIGSEGV (Address boundary error)

GDB:

1
2

Program received signal SIGSEGV, Segmentation fault.
0x0000000000401679 in dill_prologue (jb=0x7fffffffdcc0, ptr=0x7fffffffdcc8, len=16384, 
    file=0x404b1c "go_mem_fail.c", line=29) at cr.c:198
198         cr->vfs.query = dill_cr_query;

Plattfom: Linux, x86_64, static libdill build (git 5e736e5), debug enabled

example crash

1. make check
2. valgrind
   --gen-suppressions=no
   --tool=memcheck --leak-check=full --show-leak-kinds=all
   --free-fill=0x0
   tests/.libs/example

==21902== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==21902== Access not within mapped region at address 0x100000007
[1] 21902 segmentation fault (core dumped) valgrind --gen-suppressions=no --tool=memcheck --leak-check=full

[Feature] __builtin_setjmp and __builtin_longjmp support

I believe it is possible to have support for __builtin_setjmp and __builtin_longjmp which are compiler-specific optimised implementations of the setjmp/longjmp which do not depend on glibc. However, the downside of those are that you should use the same compiler with the the application and the library (but GCC and clang are compatible with each other in this instance).

It's just an idea - from my previous tests, it's slower than assembly, but faster than glibc-based ones.

It does not support having setjmp and longjmp in the same function so dill_wait needs to be separated into two where the second one marked with __attribute__((noinline)).

This is a candidate for another compile-time flag and it might fill the gap (performance + might work with stack protection) between -DDILL_ARCH_FALLBACK and the assembly version.

Note: Intel Cilk support in GCC uses this.

Failed to from source

I got the below when I attempted to install from source. And when I checked the configure.ac I found
LT_INIT. I can't figure out why It could not build

Libtool

LT_INIT

Finish the configuration phase.

AC_CONFIG_MACRO_DIR([m4])

AC_OUTPUT([Makefile libdill.pc])
cp confdefs.h config.h

em@Lenovo-G40-30:/home/projects/libdill$ sudo aclocal
em@Lenovo-G40-30:/home/projects/libdill$ sudo autoconf
em@Lenovo-G40-30:/home/projects/libdill$ sudo ./autogen.sh
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal --force -I m4
autoreconf: configure.ac: tracing
autoreconf: configure.ac: not using Libtool
autoreconf: running: /usr/bin/autoconf --force
autoreconf: configure.ac: not using Autoheader
autoreconf: running: automake --add-missing --copy --force-missing
Makefile.am:32: error: Libtool library used but 'LIBTOOL' is undefined
Makefile.am:32: The usual way to define 'LIBTOOL' is to add 'LT_INIT'
Makefile.am:32: to 'configure.ac' and run 'aclocal' and 'autoconf' again.
Makefile.am:32: If 'LT_INIT' is in 'configure.ac', make sure
Makefile.am:32: its definition is in aclocal's search path.
autoreconf: automake failed with exit status: 1

Test Suite Errors

When compiling under Arch, I get the following error output:

==> Starting check()...
make  tests/example tests/go tests/fd tests/cls tests/handle tests/chan tests/choose tests/sleep tests/signals tests/overload tests/proc tests/proc1 tests/proc2 tests/proc3 tests/proc4 tests/proc5
make[1]: Entering directory '/home/halosghost/.build/libdill/src/libdill-0.5-beta'
  CC       tests/example.o
  CCLD     tests/example
  CC       tests/go.o
  CCLD     tests/go
  CC       tests/fd.o
  CCLD     tests/fd
  CC       tests/cls.o
  CCLD     tests/cls
  CC       tests/handle.o
  CCLD     tests/handle
  CC       tests/chan.o
  CCLD     tests/chan
  CC       tests/choose.o
  CCLD     tests/choose
  CC       tests/sleep.o
  CCLD     tests/sleep
  CC       tests/signals.o
  CCLD     tests/signals
  CC       tests/overload.o
  CCLD     tests/overload
  CC       tests/proc.o
  CCLD     tests/proc
  CC       tests/proc1.o
  CCLD     tests/proc1
  CC       tests/proc2.o
  CCLD     tests/proc2
  CC       tests/proc3.o
  CCLD     tests/proc3
  CC       tests/proc4.o
  CCLD     tests/proc4
  CC       tests/proc5.o
  CCLD     tests/proc5
make[1]: Leaving directory '/home/halosghost/.build/libdill/src/libdill-0.5-beta'
make  check-TESTS
make[1]: Entering directory '/home/halosghost/.build/libdill/src/libdill-0.5-beta'
make[2]: Entering directory '/home/halosghost/.build/libdill/src/libdill-0.5-beta'
./test-driver: line 107: 22194 Segmentation fault      (core dumped) "$@" > $log_file 2>&1
FAIL: tests/example
./test-driver: line 107: 22223 Segmentation fault      (core dumped) "$@" > $log_file 2>&1
FAIL: tests/go
./test-driver: line 107: 22259 Segmentation fault      (core dumped) "$@" > $log_file 2>&1
FAIL: tests/fd
./test-driver: line 107: 22294 Segmentation fault      (core dumped) "$@" > $log_file 2>&1
FAIL: tests/cls
PASS: tests/handle
./test-driver: line 107: 22358 Segmentation fault      (core dumped) "$@" > $log_file 2>&1
FAIL: tests/chan
./test-driver: line 107: 22390 Segmentation fault      (core dumped) "$@" > $log_file 2>&1
FAIL: tests/choose
./test-driver: line 107: 22424 Segmentation fault      (core dumped) "$@" > $log_file 2>&1
FAIL: tests/sleep
./test-driver: line 107: 22459 Segmentation fault      (core dumped) "$@" > $log_file 2>&1
FAIL: tests/signals
./test-driver: line 107: 22493 Segmentation fault      (core dumped) "$@" > $log_file 2>&1
FAIL: tests/overload
./test-driver: line 107: 22525 Segmentation fault      (core dumped) "$@" > $log_file 2>&1
FAIL: tests/proc
./test-driver: line 107: 22559 Segmentation fault      (core dumped) "$@" > $log_file 2>&1
FAIL: tests/proc1
./test-driver: line 107: 22595 Segmentation fault      (core dumped) "$@" > $log_file 2>&1
FAIL: tests/proc2
./test-driver: line 107: 22629 Segmentation fault      (core dumped) "$@" > $log_file 2>&1
FAIL: tests/proc3
PASS: tests/proc4
PASS: tests/proc5
============================================================================
Testsuite summary for libdill Unknown
============================================================================
# TOTAL: 16
# PASS:  3
# SKIP:  0
# XFAIL: 0
# FAIL:  13
# XPASS: 0
# ERROR: 0
============================================================================
See ./test-suite.log
Please report to [email protected]
============================================================================
make[2]: *** [Makefile:1211: test-suite.log] Error 1
make[2]: Leaving directory '/home/halosghost/.build/libdill/src/libdill-0.5-beta'
make[1]: *** [Makefile:1319: check-TESTS] Error 2
make[1]: Leaving directory '/home/halosghost/.build/libdill/src/libdill-0.5-beta'
make: *** [Makefile:1631: check-am] Error 2

The file at test-suite.log is here (though it's not terribly helpful as far as I can tell).

Curious C++ argument type conversion crash

Here's a segmentation fault in the simplest C++ program:

Source

main.cpp:

extern "C" {
#include <libdill.h>
}

void test(int);

int main(int ac, char **) {
    bool x = ac;
    go(test(x));
}

test.cpp:

void test(int);
void test(int) { };

Compile:

c++ -g -pthread -static -O1 -I./include -L./lib main.cpp test.cpp -ldill -o test

Result

> gdb ./test
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 800c0f400 (LWP 101037/test)]
0x000000000040048b in LJMPRET0 () at main.cpp:9
9	    go(test(x));
(gdb) 

Once you switch from bool to int in the main.cpp file, everything works as intended.

I wonder if it can be fixed somehow at the library or at some sort of a shim level that can be made convenient for C++.

Warnings when compiling with clang

The following warnings appear when compiling with clang:

In file included from pollset.c:38:
./kqueue.inc:95:9: warning: implicit declaration of function 'close' is invalid in C99 [-Wimplicit-function-declaration]
        close(dill_kfd);
        ^
./kqueue.inc:173:14: warning: initializing 'int *' with an expression of type 'uint32_t *' (aka 'unsigned int *') converts between pointers to integer types with different sign [-Wpointer-sign]
        int *pidx = &dill_changelist;
             ^      ~~~~~~~~~~~~~~~~
./kqueue.inc:177:18: warning: assigning to 'int *' from 'uint32_t *' (aka 'unsigned int *') converts between pointers to integer types with different sign [-Wpointer-sign]
            pidx = &dill_fdinfos[*pidx - 1].next;
                 ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./kqueue.inc:202:39: warning: | has lower precedence than ==; == will be evaluated first [-Wparentheses]
        if(!dill_list_empty(&fdi->in) | fdi == dill_parentinfo) {
                                      ^~~~~~~~~~~~~~~~~~~~~~~~
./kqueue.inc:202:39: note: place parentheses around the '==' expression to silence this warning
        if(!dill_list_empty(&fdi->in) | fdi == dill_parentinfo) {
                                      ^
                                        (                     )
./kqueue.inc:202:39: note: place parentheses around the | expression to evaluate it first
        if(!dill_list_empty(&fdi->in) | fdi == dill_parentinfo) {
                                      ^
           (                               )

$ clang --version
Apple LLVM version 7.3.0 (clang-703.0.31)
Target: x86_64-apple-darwin15.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

[Feature] Generating man pages

The documentation for libdill is quite comprehensive on the website and structured in a manpage sort of format. ronn can generate manpages from Markdown - perhaps some investigation should be made to see if it is possible to integrate optional manpage building support if ronn is installed. This would be for offline manpage use on the application developer's system.

$ ./configure --enable-docs # checks if ronn is installed
$ make docs # build documentation only
$ make # only build documentation if ronn is enabled in configure

As the documentation is in a separate branch, perhaps having documentation distributed separately is a viable alternative as well instead of complicating libdill's build system.

libdill failures on FreeBSD with gcc

With gcc:
./configure 1 failure
./configure --disable-threads == same failure
./configure --disable-threads CFLAGS="-DDILL_ARCH_FALLBACK -fno-stack-protector :

make  check-TESTS
Segmentation fault (core dumped)
FAIL: tests/example
Segmentation fault (core dumped)
FAIL: tests/go
Segmentation fault (core dumped)
FAIL: tests/fd
PASS: tests/handle
Segmentation fault (core dumped)
FAIL: tests/chan
Segmentation fault (core dumped)
FAIL: tests/choose
Segmentation fault (core dumped)
FAIL: tests/sleep
Segmentation fault (core dumped)
FAIL: tests/signals
Segmentation fault (core dumped)
FAIL: tests/overload
PASS: tests/rbtree

With clang
./configure all OK
./configure --disable-threads all OK
./configure --disable-threads CFLAGS="-DDILL_ARCH_FALLBACK -fno-stack-protector all ok

build error on ubuntu 12.04

Following the build instruction on libdill.org, I ran into following build error on ubuntu 12.04. I was trying to build 0.6-beta.

cat libdill-0.6-beta/make.log
  CC       libdill_la-cr.lo
cr.c: In function 'dill_wait':
cr.c:356:9: error: incompatible type for argument 1 of '__sigsetjmp'
/usr/include/setjmp.h:60:12: note: expected 'struct __jmp_buf_tag *' but argument is of type 'struct __jmp_buf_tag'
cr.c:369:13: error: incompatible type for argument 1 of 'siglongjmp'
/usr/include/setjmp.h:110:13: note: expected 'struct __jmp_buf_tag *' but argument is of type 'struct __jmp_buf_tag'
make: *** [libdill_la-cr.lo] Error 1

This is my local configuration

[ubuntu libdill]$ uname -a
Linux tdwong-ubuntu 3.2.0-107-generic-pae #148-Ubuntu SMP Mon Jul 18 20:44:42 UTC 2016 i686 i686 i386 GNU/Linux

[ubuntu libdill]$ gcc --version
gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3
Copyright (C) 2011 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Without looking into the source code details, I tried to modify the code to overcome the errors and this is what I have done in libdill.h. Everything builds okay and the sample hello.c program runs fine. But, I am not sure this is really a good fix.

[ubuntu libdill-0.6-beta]$ diff -rupN libdill.h.orig libdill.h
--- libdill.h.orig      2016-10-12 15:12:53.993017613 -0700
+++ libdill.h   2016-10-12 14:38:18.506710255 -0700
@@ -151,8 +151,10 @@ DILL_EXPORT void dill_proc_epilogue(void
         : : "a" (ctx) : "rdx" \
     )
 #else
-#define dill_setjmp(ctx) sigsetjmp(*ctx, 0)
-#define dill_longjmp(ctx) siglongjmp(*ctx, 1)
+//#define dill_setjmp(ctx) sigsetjmp(*ctx, 0)
+//#define dill_longjmp(ctx) siglongjmp(*ctx, 1)
+#define dill_setjmp(ctx) sigsetjmp(ctx, 0)
+#define dill_longjmp(ctx) siglongjmp(ctx, 1)
 #endif

 /* Statement expressions are a gcc-ism but they are also supported by clang.

[Discussion] Channel priorities and choices

Two options here.

Reproduced from libdill.h for discussion:

#define CHSEND 1
#define CHRECV 2

struct chclause {
    int op;
    int ch;
    void *val;
    size_t len;
    char reserved[64];
};

DILL_EXPORT int channel(size_t itemsz, size_t bufsz);
DILL_EXPORT int chsend(int ch, const void *val, size_t len, int64_t deadline);
DILL_EXPORT int chrecv(int ch, void *val, size_t len, int64_t deadline);
DILL_EXPORT int chdone(int ch);
DILL_EXPORT int choose(struct chclause *clauses, int nclauses,
    int64_t deadline);

Proposed API for message priorities (separate functions):

DILL_EXPORT int pchannel(size_t itemsz, size_t bufsz, int priorities);
DILL_EXPORT int pchsend(int ch, int prio, const void *val, size_t len, int64_t deadline);
DILL_EXPORT int pchrecv(int ch, int *prio, void *val, size_t len, int64_t deadline);
DILL_EXPORT int pchdone(int ch);

Second proposed API.
Replace existing API, but then requires additional parameters which will break existing API.

#define CHSEND 1
#define CHRECV 2

struct chclause {
    int op;
    int ch;
    int prio; /* CHSEND reads prio from clauses and CHRECV writes to prio in clauses. */
    void *val;
    size_t len;
    char reserved[64];
};

DILL_EXPORT int channel(size_t itemsz, size_t bufsz, int priorities);
DILL_EXPORT int chsend(int ch, int prio, const void *val, size_t len, int64_t deadline);
DILL_EXPORT int chrecv(int ch, int *prio, void *val, size_t len, int64_t deadline);
DILL_EXPORT int chdone(int ch);
DILL_EXPORT int choose(struct chclause *clauses, int nclauses,
    int64_t deadline);

Any thoughts?

Compatibility documentation

A page for the documentation:

https://gist.github.com/82a7cb6d3c6ecff92541b2501d070fe6

Aspects of it are relevant to #36 and #37

Slow signals test with --with-census option

This is significantly slower. The test blocks for couple of seconds.

When fdin/fdout times out fd is not removed from the pollset

#include <assert.h>
#include <errno.h>
#include <stdio.h>

#include "libdill.h"

int main() {
    int pp[2];
    int rc = pipe(pp);
    assert(rc == 0);
    rc = fdin(pp[0], now() + 100);
    assert(rc == -1 && errno == ETIMEDOUT);
    ssize_t sz = write(pp[1], "ABC", 3);
    assert(sz == 3);
    rc = msleep(now() + 1000);
    assert(rc == 0);
    return 0;
}

Result:

Assert failed: !dill_qlist_empty(&ctx->ready) (cr.c:389)

[Discussion] Best-practice Documentation

Creating this issue as a reminder.

As there's been a lot of discussion relating to the design of applications using libdill, I think we should, eventually, have some FAQ page containing a distilled version on how to use libdill.

[Request] Broadcast Message to Multiple Channels

Hello, and thanks for reading this (again). The semester has been really busy and ironically due to my studies I have not been able to... well.. continue my studies. I've got a bit of time coming up after finals and the project that initially prompted this request has started to pear it's head over the horizon so I think I'll pose this again.

My initial request is as follows. I've taken the liberty to adjust the language used to be more suitable for use with this project:

This is more of a feature request to see if such a thing is possible.

I was thinking of writing an IRC server or some such system with libdill (changed from libmill) and I was looking through the docs. It doesn't seem like this sort of system is supported without managing an array of channels (one for each listener).

This feature was also brought up in a mailing list that I found when googling: http://www.freelists.org/post/libmill/broadcast-effect

And the answer that was proposed seemed more like an anti-pattern in this case: http://www.freelists.org/post/libmill/broadcast-effect,1

Is it possible to support

coroutine void on_c(int channel)
{
  char *message;
  int err = chrecv(channel, &message, sizeof(char*), -1);
  if (err) message = "Error handling message";
  printf("%s\n", message);
}


void main()
{
  char *message = "Hello World";
  // Create a broadcast channel 
  int ch = bchannel(sizeof(int), 0);
  // Pass this to many coroutines
  go(on_c(ch));
  go(on_c(ch));
  go(on_c(ch));
  // Broadcast this to listeners. 
  chbroadcast(ch, &message, sizeof(char*), -1);
}

Where the output is:

Hello World
Hello World
Hello World

This would, in my opinion, create some interesting design patterns for message/event driven systems in C.

Do you think something like this would be detrimental to this library? How would someone go about implementing this into the code base if you aren't interested in doing this? Would a feature like this be merged/mainlined or are you uninterested in supporting a feature like this within this platform?

I'm very excited about the prospects of being able to design a structured concurrency system that can take advantage of an API like this.

Handle Example

Could you give me an example of an use case in which one would need to create his own handler type, please?

Reference:
http://libdill.org/handle.html

memory leak

valgrind log:

==14375==
==14375== HEAP SUMMARY:
==14375== in use at exit: 24,746,000 bytes in 2 blocks
==14375== total heap usage: 6 allocs, 4 frees, 24,750,208 bytes allocated
==14375==
==14375== 266,240 bytes in 1 blocks are possibly lost in loss record 1 of 2
==14375== at 0x4C2D110: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14375== by 0x4C2D227: posix_memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14375== by 0x4E3AC0C: dill_allocstack (stack.c:90)
==14375== by 0x4E3918B: dill_prologue (cr.c:108)
==14375== by 0x40120C: main (main.c:97)
==14375==
==14375== 24,479,760 bytes in 1 blocks are still reachable in loss record 2 of 2
==14375== at 0x4C2CC70: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14375== by 0x4E3A4BD: dill_poller_init (epoll.inc:65)
==14375== by 0x4E3A687: dill_fdwait_ (poller.c:59)
==14375== by 0x504260D: tcpaccept (tcp.c:182)
==14375== by 0x40120C: main (main.c:104)
==14375==
==14375== LEAK SUMMARY:
==14375== definitely lost: 0 bytes in 0 blocks
==14375== indirectly lost: 0 bytes in 0 blocks
==14375== possibly lost: 266,240 bytes in 1 blocks
==14375== still reachable: 24,479,760 bytes in 1 blocks
==14375== suppressed: 0 bytes in 0 blocks

Should we release memory when exit progress?
That will make valgrind happy.

Stream Channels?

I wanted to get a feel on what your thoughts were on "streaming" channels (as opposed to message-based).

Let's work with an example: let's say I wanted to handle HTTP responses.

The natural division would seem to be have two coroutines: one to receive data from a TCP socket and stuff it into a channel (tcp_handler), and another to read from the channel and parse the HTTP responses (http_handler).

Method 1

tcp_handler reads packets from TCP, and then writes the whole packet to the channel. (It could write a a pointer, or the whole buffer -- which is more idiomatic here? Currently chan doesn't work across proc() anyways...). This coroutine is simple.

http_handler gets awkward. While parsing the headers, it wants to read line-delimited data. It needs to maintain it's own buffer between channel reads, and then stitch the buffers together. It all works, but it seems like you've reintroduced a state machine and were unable to take advantage of the fact that it's a coroutine.

Method 2

tcp_handler reads packets from TCP. It then buffers the data and delimits it by newlines. Each newline is written separately into the channel.

http_handler now can parse the header very cleanly. However, there's a bug with parsing the response body: it's not required to end with a newline. The parser will for a newline/more data. So this just doesn't work.

Possibly tcp_handler could know more about how HTTP works, or http_handler could send some type of "mode" information back to tcp_handler, but both of these seem too hacky.

Method 3

tcp_handler reads packets from TCP, and writes the data to a channel which acts as a stream.

http_handler can now read with whatever granularity it wants from the channel.

Ideally, it would be great if the API supported peaking into the channel's buffer before removing it -- this would allow http_handler to find newlines without having to read 1 byte at a time. (Though, this does make the API more complex)

---

So, thoughts? Does Method 3 have enough merit, or is HTTP just a special-case for being both newline and length delimited? Is there something I'm missing? Otherwise, if it were just newline delimited, I think Method 2 would be best.

Does libdill support linux x86 32bit?

Hi Sustrik and Raedwulf! I try to run "Hello world" libdill's example on Linux 32bit (Ubuntu 14.04) and got error message: " Assert failed: !dill_slist_item_inlist(&cr->ready) (cr.c:412)", but it is fine on Ubuntu 14.04 64bit. So Libmill doesn't support linux 32bit yet, does't it? Thank you!

macports as problem with .cfi ops

On darwin with macports as being installed (at /opt/local/bin/as) and the default gcc being the macports gcc, the assembler doesn't know about the .cfi ops.

  CC       libdill_la-cr.lo
  CC       libdill_la-now.lo
  CC       libdill_la-pollset.lo
  CC       libdill_la-rbtree.lo
cr.c:383:Unknown pseudo-op: .cfi_def_cfa
cr.c:383:Rest of line ignored. 1st junk character valued 37 (%).
cr.c:384:Unknown pseudo-op: .cfi_offset
cr.c:384:Rest of line ignored. 1st junk character valued 37 (%).
cr.c:385:Unknown pseudo-op: .cfi_offset
...

The fix is to use CC=/usr/bin/cc ./configure which bypasses the broken macports assembler.
This probably needs a configure probe.

Windows support?

Is it within the scope and desire of this project to support modern Windows platforms?

[Cleanup] Possible redundant condition

https://github.com/sustrik/libdill/blob/master/cr.c#L387

    /* Store the context of the current coroutine, if any. */
    if(ctx->r) {

I think this condition is redundant - because ctx->r should always be set when it enters this function.

@sustrik can you double check if this is the case?

"coroutine" switch strange, does this work correctly

here is example code,

example.c

`#include <stdio.h>

#include "assert.h"
#include "../libdill.h"

int secs = 0;

coroutine void worker(const char *text) {
while(1) {
printf("[%3d]----------------%s-------------\n", ++secs, text);
yield();
sleep(1);
}
}

int main() {
int cr1 = go(worker("0 -- text"));
errno_assert(cr1 >= 0);
int cr2 = go(worker("1 -- text"));
errno_assert(cr2 >= 0);
int rc = msleep(now() + 100);
errno_assert(rc == 0);
rc = hclose(cr1);
errno_assert(rc == 0);
rc = hclose(cr2);
errno_assert(rc == 0);
return 0;
}
`

I compile with " gcc ./example.c -o example -ldill "
centos 6.2 32bit

here is the log

`
[hash@centos tests]$ gcc ./example.c -o example -ldill
[hash@centos tests]$ ./example
[ 1]----------------0 -- text-------------

[ 2]----------------1 -- text-------------
[ 3]----------------0 -- text-------------
[ 4]----------------1 -- text-------------
[ 5]----------------0 -- text-------------
[ 6]----------------1 -- text-------------
[ 7]----------------0 -- text-------------
[ 8]----------------1 -- text-------------
[ 9]----------------0 -- text-------------
[ 10]----------------1 -- text-------------
[ 11]----------------0 -- text-------------
[ 12]----------------1 -- text-------------
[ 13]----------------0 -- text-------------
.
.
.
[105]----------------0 -- text-------------
[106]----------------1 -- text-------------
[107]----------------0 -- text-------------
[108]----------------0 -- text-------------
[109]----------------0 -- text-------------
[110]----------------0 -- text-------------
[111]----------------0 -- text-------------
[112]----------------0 -- text-------------
[113]----------------0 -- text-------------
[114]----------------0 -- text-------------
[115]----------------0 -- text-------------
[116]----------------0 -- text-------------
[117]----------------0 -- text-------------
[118]----------------0 -- text-------------
[119]----------------0 -- text-------------
[120]----------------0 -- text-------------
[121]----------------0 -- text-------------
[122]----------------0 -- text-------------
[123]----------------0 -- text-------------
[124]----------------0 -- text-------------
[125]----------------0 -- text-------------
[126]----------------0 -- text-------------
[127]----------------0 -- text-------------
[128]----------------0 -- text-------------
[129]----------------0 -- text-------------
[130]----------------0 -- text-------------
[131]----------------0 -- text-------------
[132]----------------0 -- text-------------
[133]----------------0 -- text-------------
[134]----------------0 -- text-------------
`

after 100 times switch, the "1-text" coroutine never show again.

libdill implementation of python >=3.5 asyncio event loop?

Basically wondering if it might be possible or even make sense to repeat with libdill what the MagicStack folks did with libuv?

All the features required by such an event loop may be beyond the scope of libdill alone.

Anyway, for funzies I threw up a place holder repo over at ubergarm/dillloop. If nothing else, it may be amusing to get Python bindings for coroutines and channels.

Cheers!

changes to get this to compile on Win with mingw64

Work in progress, please join in and help if you'd like. I have this compiling on Win now. have not tried running a single thing yet though (I am cross-compiling so it's at least a little work to switch over to a win box...).

Attached is the diff file (had to rename to txt to be able to upload).

In addition, there was a Make command that was missing the "-lWs2_32" so rather than change the Makefile I just did that compilation step by hand:

/bin/sh ./libtool --silent --tag=CC   --mode=link x86_64-w64-mingw32-gcc -std=gnu99 -fvisibility=hidden -DDILL_EXPORTS -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0 -g -O2 -no-undefined -version-info 3:0:0  -o libdill.la -rpath /opt/mingw64/lib libdill_la-chan.lo libdill_la-cr.lo libdill_la-fd.lo libdill_la-handle.lo libdill_la-libdill.lo libdill_la-list.lo libdill_la-pollset.lo libdill_la-proc.lo libdill_la-slist.lo libdill_la-stack.lo -lWs2_32

Will get around to testing at some point. May move to libmill first. Comments welcome

libdill_win.txt

Something wrong in libdill Tutorial - http://libdill.org/tutorial.html

Step 4: Deadlines

File descriptors can be a scarce resource. If a client connects to greetserver and lets the dialogue hang without entering the name, one file descriptor on the server side is, for all practical purposes, wasted.

To deal with the problem we are going to timeout the whole client/server dialogue. If it takes more than 10 seconds, server will kill the connection straight away.

One thing to note is that libdill uses deadlines rather than more conventional timeouts. In other words, you specify the time instant when you want the operation to finish rather than maximum time it should take to run. To construct deadlines easily, libdill provides now() function. The deadline is expressed in milliseconds so you can create a deadline 10 seconds in the future like this:

int64_t deadline = now() + 10000;
Further, you have to modify all the potentially blocking function calls in the program to take the deadline parameter. For example:

int rc = msend(s, "What's your name?", 17, deadline);
if(rc != 0) goto cleanup;
Note that errno is set to ETIMEDOUT in case the deadline is reached. However, we treat all the errors in the same way (by closing the connection) and thus we don't have to do any specific provisions for the deadline case.

---

If a client connects to greetserver and lets the dialogue hang without entering the name,

I think it should set deadline here, in mrecv not msend in the tutorial.

sz = **mrecv**(s, inbuf, sizeof(inbuf), **deadline**);
if(sz < 0)       goto cleanup;

Crashes and freezes in a simple setup

Crashes and freezes in a simple test

1. Compile the attached program. Assuming both libdill.a and libdsock.a are in ./lib, this will do: cc -pthread -I./include -g -O0 -o test test.c -L./lib -ldill -ldsock

#include <stdio.h>
#include <assert.h>

#include <dsock.h>
#include <libdill.h>

static coroutine void echo(int fd) {
    for(;;) {
        char buf[64];
        int r = brecv(fd, buf, sizeof(buf), now() + 1000);
        if(r < 0) break;
        r = bsend(fd, buf, 64, now() + 1000);
        if(r < 0) break;
    }
    hclose(fd);
    printf("Closed fd %d\n", fd);
};

static coroutine void acceptor(int port) {

    ipaddr local;
    int rc = ipaddr_local(&local, NULL, port, 0);
    assert(rc == 0);
    int lsock = tcp_listen(&local, 10);
    assert(lsock != -1);

    while(1) {
        int fd = tcp_accept(lsock, NULL, -1);
        if(fd >= 0) {
            printf("Accepting fd %d\n", fd);
            go(echo(fd));
        }
    }
}

static coroutine void ticker() {
    while(1) {
        printf(".");
        fflush(stdout);
        msleep(now() + 1000);
    }
}

int main() {

    go(ticker());
    go(acceptor(8081));

    msleep(-1);
}

2. Run tcpkali with the following options:

tcpkali 127.1:8081  -m foobarbaz -r10 -c50 -d

Option	Meaning
127.1:8081	Create connections to port 8081 on localhost
-m foobarbaz	send foobarbaz messages
-r10	Ten messages per second
-c50	Maintain 50 simultaneous connections
-d	Print messages as they are sent and received

Observe one of the following results

• On a macOS, ticker() won't even print the second dot.
• Segmentation fault on both Linux and macOS, here:

0x0000000000403fc3 in dill_rbtree_first (self=self@entry=0x7ffff7fed758)
    at rbtree.c:145
145	    while(!dill_rbtree_isnil(self, x->left)) x = x->left;
(gdb) 

• Program appears to be frozen. top will show 100% utilization on both Linux and macOS. The code goes into an infinite loop because rbtree is self-circular:

* thread #1: tid = 0x3d31, 0x00000001001d55fa libdsock.5.dylib`dill_rbtree_insert(self=0x00000001004027e8, val=837351, x=0x000000010184b588) + 170 at rbtree.c:80, queue = 'com.apple.main-thread', stop reason = step in
    frame #0: 0x00000001001d55fa libdsock.5.dylib`dill_rbtree_insert(self=0x00000001004027e8, val=837351, x=0x000000010184b588) + 170 at rbtree.c:80
   77  	    x->left=x->right = &self->nil;
   78  	    struct dill_rbtree_item *y = &self->root;
   79  	    struct dill_rbtree_item *z = self->root.left;
-> 80  	    while(!dill_rbtree_isnil(self, z)) {
   81  	        y = z;
   82  	        if(z->val > x->val) z = z->left;
   83  	        else z = z->right;
(lldb) p z->right
(dill_rbtree_item *) $7 = 0x000000010109c548
(lldb) p z->right->z_right
error: no member named 'z_right' in 'dill_rbtree_item'
(lldb) p z->right->right
(dill_rbtree_item *) $8 = 0x0000000100871e88
(lldb) p z->right->right->right
(dill_rbtree_item *) $9 = 0x000000010109c548
(lldb) p z->right->right->right->right
(dill_rbtree_item *) $10 = 0x0000000100871e88
(lldb) p z->right->right->right->right->right
(dill_rbtree_item *) $11 = 0x000000010109c548
(lldb) 

Libdill performance analysis (and muddled thoughts)

Forward

I've been working on this for 3-4 days, so this text might be slightly scrambled as I've been reworking/research/implementing different parts of libdill. I initially looked into implemented segmented stacks but have come up with a faster, more refined version of simply stack switching.

Intro

What started with some minor tweaks to the libdill code base, ended up being a significant optimisation effort within libdill's core coroutine code. My initial non-intrusive changes are available here:
https://github.com/raedwulf/libdill/tree/optimized-limit

Martin, this can be merged if you want, although it might be best once I've actually completed the rest of my changes.

Some micro-bench results

On a i7 4770k:

• Libdill (current master) performs a context switch in 16ns (for some reason my values earlier in the year were 22ns, but perhaps compiler changes/other codebase changes improved it)
• Libdill (optimised-limit) performs a context switch in 13ns
• Libdill (current master) performs a creation/termination of coroutine in 44ns
• Libdill (optimised-limit) performs a creation/termination of coroutine in 25ns

More noticeable on a old Core 2 Duo:

• Libdill (current master) performs a context switch in 46ns
• Libdill (optimised-limit) performs a context switch in 37ns
• Libdill (current master) performs a creation/termination of coroutine in 131ns
• Libdill (optimised-limit) performs a creation/termination of coroutine in 100ns

This is probably the limit of optimisation though inlining (I shaved off one more nanosecond with an intrusive change for my i7 4770k but it isn't worth the effort/code clarity to keep).

As a comparison, I fired up go and scraped out some direct translations of our perf programs:
https://gist.github.com/raedwulf/e6bbb76a3becb5fa14eb27e670260b1b
https://gist.github.com/raedwulf/e11f7bf6a25eabad3ed3e439f14b262d
https://gist.github.com/raedwulf/4a2c6c5b6209c0dd839fb95ec46378aa

This is far better than on Go (on i7 4770k): context switches take 331ns and creation/termination: 237ns
gccgo is even worse: 825 ns context switching and 3652 ns creation/termination ... OUCH

As I have very little experience with Go, check those gists in case I've actually missed something which causes such poor performance!

The Problem(s)

Signal Handling

There is no clear way to deal with signals within libdill. The signal mask belongs to the original context of the process and if that is changed within a coroutine, it applies everywhere. Furthermore, the signal handler for a trapped signal will be executed in context of a coroutine. This may be an issue for future more complex applications.

This may also be the reason why libmill/dill performs so well vs Go! To preserve the signal masks across different coroutines, the syscall SYS_rt_sigprocmask needs to be made. This kills performance - to try yourself, comment out the libdill's assembly routines and use sigsetjmp(ctx, 1).

To resolve this problem, I initially thought that optimising the context switching mechanism so that the signal mask is only set when the mask is different between coroutines (using cached values).
In an implementation, that would involve wrapping each of the sigproc* commands with dill equivalents (like proc for fork).

The pathological case would be thrashing between two or more coroutines that have different signal masks. Since, if anywhere, the signal handler should probably be done in the same context as the scheduler and control always returns there, we still have a problem of thrashing.

The best tradeoff between semantics and performance I've come up with is to have signals handled in two different ways. Signals that need to be handled immediately (e.g. SIGSEGV, SIGKILL, etc...) are handled in all contexts (thus the signal mask is the same in the scheduler and coroutines) and signals that are used to communicate behaviour (SIGINT, SIGUSR1, SIGUSR2 etc.) that are handled in dill_wait using epoll_pwait or kqueue's EVFILT_SIGNAL.

From there, it's a design choice whether to simulate a read-write based interface for signals, implementing a signal handler callback within the coroutine and/or emulate sigpending, sigwait, and sigtimedwait.

Coroutine Limits In Performance

What happens right now

However, in digging, one of the limitations of libmill/dill is that the system runs out of mmaps on 32k coroutines - whispers is a perfect example. For networking servers with many connections, this can be an issue, although you can adjust the number of mmaps using:

sysctl -w vm.max_map_count=65530 # default

However, the performance of libdill degrades to that below of Go and increases with each addition coroutine. In my tests, libdill was averaging at around 4000-4500 ns per "whisper" on the whisper benchmark and Go was speeding up, till it averaged around 2200 ns per whisper @ 100000 coroutines.

Now, disabling the guard page (and thus no mmap through mprotect), using DILL_NOGUARD, improves the performance of libdill into the same order of magnitude of Go (actually slightly faster). However, we do lose the ability to check for stack overflows.

Memory overhead is not as bad the allocated size suggests as the majority of the memory in a stack is not resident until it is touched. This translates to being a single page of memory (either 4096 or 8192 bytes). On my system with 32gb RAM, I can handle around 6 million resident coroutines in the whisper benchmark before it gives up and starts paging like crazy. In a real application, I suspect this would be closer to 4 - 5 million in-memory.

What can happen?

I've currently updated my unpushed code base with an extended go() interface that allows optional parameters (through some standards compliant C macro magic).
In particular, you can now specify,

if(stk(dstkbest | dstkmalloc | dstkmalign))
    return -1;
go(worker(), dstkguard, 8192);

which means, it'll choose the best memory allocation mechanism between malloc & posix_memalign, use page guards, and allocate a 8192 byte stack.

Omitting parameters will give defaults (currently set to dstkbest and 256kb stack) so that the original libdill interface is possible.

/* Hints - these don't end up in the final result of the mode, but they choose
   out of the following */
#define dstkbest     0x0000 /* Use this if you don't link with another library */
#define dstkcompat   0x0001 /* Use this if you do link with another library */
/* Memory Allocation methods are selected in two ways:
   - If a hint, dstkbest or dstkcompat, are chosen, then choose the best out of
     those selected here.
   - If no hints is chosen, then these flags are mutually exclusive. stk() will
     error with -1 and errno=ENOTSUP */
#define dstkdefault  0x0000
#define dstkmalloc   0x0010 /* Fallback */
#define dstkmalign   0x0020
#define dstkmmap     0x0040
/* Memory Allocation attributes only work with specifically chosen allocation
   methods.  dstkhuge is a hint, and will only apply if dstkmmap is chosen. */
#define dstkseg      0x0100 /* Only works with -fsplit-stack */
#define dstkhuge     0x0200 /* Huge pages, only works with dstkmmap */
#define dstkpool     0x0400
/* Runtime choices to be passed to go */
#define dstkfixed    0x0400 /* Fixed sized stacks, works with everything */
#define dstkguard    0x0800 /* Only works with dstkmmap and dstkmalign */

The amount of additional code required is minimal, as most of this is already implemented. The main code added is to expose the choice out at the API level.

The features I currently want to add:

• Huge page support simply requires OS support and a flag to mmap (and documentation on how to do it. So implementing it is trivial, although the smallest huge page size is typically 2Mb.
• Memory pool support
• Segmented stack support (actually I've changed my mind, stack switching is better!)

Memory Pool Support

Given the possibility that millions of coroutines are possible, it would be nice allocate smaller sized stacks if it is known that the stack won't exceed a certain size. For example, if the call stack within the coroutine is reasonably flat - i.e. it only travels through libdill and dsock in it's processing. Some analysis would need to be done here, but I would assume this is reasonable and thus stacks could be as small as 1024 bytes.

This alone would be useful and my implementation is around 90% there for supporting this.
Passing (dstkfixed | dstkhuge | dstkmmap) would give you a fixed sized allocated coroutine using a

Segmented stack support (controversial)

The Problem

The ideal situation is:

• Coroutines can be made arbitrarily tiny (almost zero overhead!).
• Coroutines have some stack protection (e.g. page guards).
• Coroutines have very little allocation/deallocation overhead.
• Coroutines have no limits except memory limits.

What we currently have is;

• Coroutines can be made small - down to roughly a single page size of resident memory (4k or 8k).
• Coroutines can have stack protection with mprotect but will exhaust OS limits unless they are changed. However, this increases allocation overhead as well.
• Without stack protection, there is very little allocation/deallocation overhead.

Segmented stacks have problems. There's no denying that. But segmented stacks also have solutions.

Quick Primer

Stacks are usually big contiguous blocks of memory pre-allocated up front. This means they're fixed and overflowing can be easily triggered.
Segmented stacks are non-contiguous. A small stack is initially allocated and when an overflow is detected, another block is then allocated.

The original gcc -fsplit-stack design was presented was presented here.

The Problems Go and Rust have found

1. The hot-split problem or stack-thrashing performance problem

Imagine a function call happening when the stack is almost full. The call will force the stack to grow and a new segment will be allocated. When that function returns, the segment will be freed and the stack will shrink again. Reference

1. Code without segmented stack support requires a large stack to be allocated. This defeats the purpose of segmented stacks. Reference
2. Rust has stricter memory rules, and I think they found issues with segmented stacks preventing them optimising memory usage.

And how we can handle them

In Rust and Go, it's assumed that any function can be a coroutine and can call any other function and can be interrupted at any time. This makes segmented stacks problematic.

However, in libdill, we can make a very important assumption that when calling a function that does not use stack splitting support, we don't have context switch nor callbacks (unless you have some code that links with libdill that calls the scheduler) which would have a lifetime that extends after a libdill context switch. This fact means we can make coroutines split their stack any time to a single/global scratch buffer/stack. This scratch buffer/stack could actually be the application stack as it would not be used at that time.

Implementation Issues

Using -fsplit-stack

Both GCC >4.6 and LLVM > 3.4 has segmented stacks implemented with -fsplit-stack. However, to get them working, the interface provided by libgcc is woefully painful to use.
One could override __morestack and __morestack_allocate_stack_space to support libdill, and it would very likely work pretty well. However, -fsplit-stack is a global flag which applies to all functions compiled in a *.c file. This makes it cumbersome in the sense you have to explicitly mark all other functions as __attribute__((no_split_stack)) and leave the coroutines unmarked, ironically.

There's ways around it, like submitting a patch or creating a plugin for GCC/LLVM to have a flag like -fsplit-stack-explicit but the cost/benefit of developing this approach is currently on the fence.

Recommended solution (stack switching and not stack segmenting!)

However, given that this is an optimisation for coroutines, a neater solution would be to set a small size for coroutines and then wrap all functions that are likely to trigger a stack overflow with a "stack switcher". Is it safe? As safe as our coroutines, because it does a perfect tiny subset of what our context switching does; it keeps all the registers but changes the stack into a scratch buffer. Thus is is super cheap - much much cheaper than context switching.

In fact, I've just realised that I've already written a generic macro that switches the stackpointer... so I actually do support this in my current code base already. I'll wrap this up with some of my memory pool optimisation and show off this technique in a test.

The TLDR example:

coroutine helloworld() {
    /* I only use small buffers here */
   int buffer[128]; int world; int hello;
   .... /* do interesting stuff like prepare and loop buffers */
   scratch(bsend(s, buffer, sizeof(buffer), -1)); /* do something with dsock */
   scratch(externalcall("president"));  /* call some external library which uses lots of stack space */
   /* tidyup */
}

So for a constrained memory coroutine, the scratch macro gives the control for the application writer to optimise stack usage without running into any problems (except still allowing them to shoot themselves in the foot). The one advantage of -fsplit-stack is that it does this automatically at a runtime cost, making sure that it is a lot harder for the application writer to shoot themselves in the foot, but it moves the fight against the compilers instead.

Coroutine Local Storage Example

Could you give me an example of an use case in which one would need to use the coroutine local storage, please?

Reference:
http://libdill.org/cls.html

[Discussion] Proc and Thread Model

I split this out from the original discussion.

P1         |P2                       proc (Unsupported on Windows)
-----+-----+-----+-----
T1   |T2   |T3   |T4                 thread (Windows & *nix)
--+--+--+--+--+--+--+--
C1|C2|C3|C4|C5|C6|C7|C8              go(routines) (libdill)

Currently in libdill, the API hints at and I think does support nested proc. Going for the spawn-like approach would remove that hierarchy.

I think this could be reflected in the API:

/* function that parses information from the parameters to determine whether the execution of the application is a child, returns -1 for failure, 0 for non-child, 1 for child */
int procinit();
/* spawn a new process of this program */
int procspawn(const char *argv[]);

procspawn sets an environment variable to communicate an named-IPC to communicate between the main process and the child processes (this approach allows minimal conflict with the command line parsing by the user and this is also supported on Win32)

The application pattern would be:

int main(int argc, char **argv) {
    int rc;
    if(!(rc = procinit()) {
        /* main application */
        int h = procspawn(argv); /* spawn a process */
        /* sleep. do stuff */
        hclose(h);
    } else {
        /* child */
        if(dill_slow(rc == -1)) {
            if(errno == ENOPERM) fprintf(stderr, "cannot access process IPC\n");
            else fprintf(stderr, "unknown error\n");
            return 1;
        }
        /* do stuff in child process */
    }
}

On an unrelated note, I noticed that libdill has a function msleep. Is the naming a throwback to libmill?

testsuite core dumps on FreeBSD

Hi, I have been tested libdill on FreeBSD (to make a port out of it)

Running the testsuite everything is ok but the "fd" example which dies with a:
"Assert failed: !dill_slist_empty(&ctx->ready) (cr.c:405)"

I'm testing on FreeBSD current the compiler being clang 3.9.0 (note that I have tried enabling and disabling SSP)

Assert when DILL_POLL is used

Assert failed: !dill_slist_empty(&dill_ready) (cr.c:472)

Can be reproduced using keeplive test in dsock.

Heisenbug in 'threads' test

Run:

while true; do ./tests/threads; done

Error:

Assert failed: !cr->ready.next (cr.c:385)

Backtraces:

Program terminated with signal SIGABRT, Aborted.
#0  0x00007f1ea59aec37 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007f1ea59aec37 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f1ea59b2028 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x000000000040309d in dill_docancel (cr=0x7f1ea63677b0, id=1, err=110) at cr.c:385
#3  0x0000000000402fc5 in dill_trigger (cl=0x7ffc1fe55980, err=110) at cr.c:398
#4  0x0000000000402e4b in dill_wait () at cr.c:360
#5  0x0000000000402aa9 in dill_epilogue () at cr.c:252
#6  0x0000000000401702 in main () at tests/threads.c:64
(gdb) info threads
  Id   Target Id         Frame 
  2    Thread 0x7f1ea5977700 (LWP 9526) 0x00007f1ea5a6c8d7 in mprotect () from /lib/x86_64-linux-gnu/libc.so.6
* 1    Thread 0x7f1ea6367900 (LWP 9525) 0x00007f1ea59aec37 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) thread 2
[Switching to thread 2 (Thread 0x7f1ea5977700 (LWP 9526))]
#0  0x00007f1ea5a6c8d7 in mprotect () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007f1ea5a6c8d7 in mprotect () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x0000000000405bc5 in dill_ctx_stack_term (ctx=0x7f1ea59776d0) at stack.c:73
#2  0x0000000000406674 in dill_ctx_term (ptr=0x7f1ea5977550) at ctx.c:95
#3  0x00007f1ea5f4cf82 in __nptl_deallocate_tsd () from /lib/x86_64-linux-gnu/libpthread.so.0
#4  0x00007f1ea5f4d197 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5  0x00007f1ea5a7237d in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) q

Add dill_fdwait for bi-directional event monitoring

Libdill has no API to wait on a file descriptor for either FDW_IN or FDW_OUT and return the the events that apply.

An argument has been made that each event should be managed in it's own co-routine but I believe this increases the complexity of implementing bi-directional protocols that are not FIFO such as WebSockets / HTTP/2 and hampers integration with existing protocol libraries.

Take for example an implementation of nghttp2 on libmill:

    while (nghttp2_session_want_read(connection.session) ||
           nghttp2_session_want_write(connection.session)) {
        int events = mill_fdwait_(fd, FDW_IN | FDW_OUT, -1, NULL);
        
        if ((events & FDW_IN) && nghttp2_session_want_read(connection.session)) {
            if (nghttp2_session_recv(connection.session) < 0) {
                die("Connection error");
            }
        }
        
        if ((events & FDW_OUT) && nghttp2_session_want_write(connection.session)) {
            if (nghttp2_session_send(connection.session) < 0) {
                die("Connection error");
            }
        }
        
        if (events & FDW_ERR) {
            die("Connection error");
        }
    }

Is there an easy way to accomplish something similar in libdill?

Relationship to libmill

Since I don't really know where else to ask, I'll just do it here:

How do you see this library in relation to libmill?
I know the FAQ has a section about it, but do you consider libdill a replacement that should be preferred over libmill?

What are your future plans? To prefer and focus development on libdill?
Develop both actively?

[Bug] hdup SEGFAULTs after the 255th duplication of the same handle

The error originally occurred while handling TCP connections with dsock, but can be reduced to the following snippet:

void dup_test() {
    uint64_t count = 0;
    uint64_t runs = 1024;

    int ch = chmake(1);
    assert(ch >= 0);

    while (count < runs) {
        printf("\r%lu", count); fflush(stdout);
        int h = hdup(ch);
        assert(h >= 0);
        ++count;
        h = hclose(h);
        assert(h >= 0);
    }
}

Output with ASAN enabled:

255ASAN:SIGSEGV
=================================================================
==12351==ERROR: AddressSanitizer: SEGV on unknown address 0x607017000011 (pc 0x0000004054b3 bp 0x7ffe03ca6580 sp 0x7ffe03ca6550 T0)
    #0 0x4054b2 in hdup [...]/libdill/handle.c:107
    #1 0x402ef2 in dup_test [...]/foo.c:142
    #2 0x4022ca in main [...]/foo.c:151
    #3 0x7f5b3f8ae82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #4 0x402318 in _start ([...]/seg+0x402318)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV [...]/libdill/handle.c:107 hdup
==12351==ABORTING

Process finished with exit code 1

Library compiled with ./configure --disable-shared --enable-debug

Use without install libdill

Hi all,

maybe the question is quite naive, however I wasn't able to come up with any solution.

In a project I am developing I am considering to use libdill, however I need it to don't be installed in the machine.
I need libdill inside a folder, then compile it and link it to my library, without installing it to my host machine.

Are there any way to achieve this?

Best,

[Discussion] Moving out of beta

I've done some API cleanup and I think we are ready to move out of beta.

What may be missing:

• custom stack allocators (can be done using go_s in separate libs as needed)
• buffered queues (can be provided in external libs)
• census should report at the end of process rather than at the end of thread (not critical)

Thoughts? Objections?

Mailing List

http://libdill.org/documentation.html

The documentation mentions the mailing list, but I can't find a link anywhere. Is there a mailing list yet?
By the way, Libdill is awesome. I was waiting for a release for a long time.

Congratulations!

About coroutine cancelation

Hi,sustrik.
I think "gocancel" is a very useful API.
Why did you cancel it?

Do channels work across procs?

If I send a channel from the main coroutine to a proc, could I retrieve values sent to the channel in the proc from the main coroutine?