support mac os about shared-memory-disruptor

[davedoesdev]

@z-hao-wang that looks a lot neater and is completely different from the squid code, thanks.

[davedoesdev]

OK I limited the tests to number of CPUs and merged #6
@z-hao-wang thanks again

[davedoesdev]

Hi @z-hao-wang
I don't have access to a Mac to test. Can you run the tests on a Mac?

[davedoesdev]

I'm trying to get it working on Travis CI OSX runners

[z-hao-wang]

Looks like it throw an error when doing npm ci

../src/disruptor.cc:380:5: error: static_assert failed due to requirement 'std::is_same<decltype(errmsg), char *>::value' "strerror_r must return char*"
    static_assert(std::is_same<decltype(errmsg), char*>::value,
    ^             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../src/disruptor.cc:383:43: error: incompatible operand types ('int' and 'std::__1::string' (aka 'basic_string<char, char_traits<char>, allocator<char> >'))
        std::string(msg) + ": " + (errmsg ? errmsg : std::to_string(errnum)));

my env is python 2.7.16, OS 10.14.6
node 12.14
npm 6.13.4

[z-hao-wang]

I made some changes (z-hao-wang@0cd33c6) to let it compile in mac OS. but it failed the tests. The changes I made still passes tests in linux (ubuntu 18).

grunt test
Running "mochaTest:src" (mochaTest) task


  multi-process many-to-many (producers: 1, consumers: 1, elements to write: 1)
    ✓ should transfer data (128ms)

  multi-process many-to-many (producers: 1, consumers: 1, elements to write: 2)
    1) should transfer data


  1 passing (138ms)
  1 failing

  1) multi-process many-to-many (producers: 1, consumers: 1, elements to write: 2)
       should transfer data:
     Error: Failed to open shared memory object: 22
    at new Disruptor2 (lib/disruptor.js:16:9)
    at Context.<anonymous> (test/multi_process.js:15:10)
    at processImmediate (internal/timers.js:439:21)
    at process.topLevelDomainCallback (domain.js:130:23)
  

I think there are something broken when reading data from shared memory. But I don't have an idea on where to look into. Any help would be appreciated.

[davedoesdev]

@z-hao-wang thanks. Can you pull master and try that please? I've made similar changes but get EINVAL from shm_open

[z-hao-wang]

Thanks @davedoesdev for the quick fix.
I just pulled master a110085 but still getting same error in testing. Error: Failed to open shared memory object: 22

[davedoesdev]

@z-hao-wang you should be getting Error: Failed to open shared memory object: Invalid argument
as per https://travis-ci.org/davedoesdev/shared-memory-disruptor/jobs/630463623

[davedoesdev]

For some reason shm_open is failing. I had thought it was the Travis CI environment but maybe not. Are you using a physical machine?

[z-hao-wang]

yes, I am testing on physical macbook pro. yes I am getting exact same error Error: Failed to open shared memory object: Invalid argument

[davedoesdev]

I have to leave this for a day or so now but feel free to play with the arguments to see if one of them is causing EINVAL. I'm wondering if it's O_TRUNC (http://lists.squid-cache.org/pipermail/squid-dev/2016-January/004832.html and https://stackoverflow.com/questions/25502229/ftruncate-not-working-on-posix-shared-memory-in-mac-os-x?lq=1). However, the shared memory shouldn't already exist so I don't see why that's an issue.
Maybe the name shouldn't start with a / on OSX, I don't know.

[z-hao-wang]

Thanks for the help. made PR to fix this #6

[davedoesdev]

@z-hao-wang thanks. The logic looks fine but I'm going to check the latest squid sources to see what code they ended up merging. I might check Chromium too to see if they do something similar.

[davedoesdev]

Squid code is here: https://github.com/squid-cache/squid/blob/75aadeb9cc1128bb50adf8fc629d3957e9a88f2f/src/ipc/mem/Segment.cc#L89

They seem to have rewritten from the original patch to make it neater.

[z-hao-wang]

yeah, the squid code is definitely cleaner. it looks like we can just use that path for both linux and mac os.
I'm happy to do some clean ups to make it similar like the squid code.

[davedoesdev]

@z-hao-wang note that squid is GPL so I think we need to implement our own version or find another project which has implemented similar workarounds.

[z-hao-wang]

@davedoesdev I'm not directly copy pasting their code. besides, our logic is slightly different anyway.
just updated the PR #6.

[davedoesdev]

I'll wait for the tests to complete then merge. Thanks for your help, appreciate it.

[davedoesdev]

Looks like the Travis OSX runners are slow so the test timed out.
I'll take a look later today or tomorrow but the code seems to work.

[davedoesdev]

@z-hao-wang do the tests pass for you on your Mac? How long do they take?
They seem to be taking way too long on the Travis OSX runners, almost like something's wrong.

[z-hao-wang]

Tests (grunt test) passed on my mac, took 53s. I have 4 core 16GB RAM.
Maybe Travis is just too slow for this. Or the test runs better with multi core cpu?

[davedoesdev]

That's consistent with my Linux laptop. Something's a bit wrong with Travis.

[z-hao-wang]

I'm guessing Travis is single core cpu, and this test spin up 6 child processes.
multi-process many-to-many (producers: 4, consumers: 2, elements to write: 10000)
It's pretty intensive, this may cause the cpu to have 100% usage.

[davedoesdev]

I think it has two vCPUs: https://docs.travis-ci.com/user/reference/overview/
But CircleCI (which I use for Linux tests) also has two vCPUs. Maybe Linux multitasks better or something.

[davedoesdev]

Published as version 3.0.3