erleans / erleans Goto Github PK

Env:

• Erlang/OTP 20 [erts-9.2.1] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe] [kernel-poll:false]
• mac os high sierra

===> Verifying dependencies...
===> Compiling erleans
===> Running Common Test suites...
%%% dist_lifecycle_SUITE:
2018-07-24 19:12:36.337 [info] pid=<0.1976.0> module=erleans_grain function=do_for_ref line=178 start=#{id => <<"grain1">>,implementing_module => test_grain,placement => prefer_local,provider => {ets_provider,ets}}
2018-07-24 19:12:36.337 [info] pid=<0.1976.0> module=erleans_grain_sup function=start_child line=33 node='[email protected]' grain=#{id => <<"grain1">>,implementing_module => test_grain,placement => prefer_local,provider => {ets_provider,ets}}
2018-07-24 19:12:36.360 [info] pid=<0.1985.0> module=lasp_core function=enforce_once line=123 Threshold not met: {state_awset,{[{<0.1981.0>,[{<<131,100,0,17,101,114,108,101,97,110,115,64,49,50,55,46,48,46,48,46,49>>,1}]}],{[{<<131,100,0,17,101,114,108,101,97,110,115,64,49,50,55,46,48,46,48,46,49>>,1}],[]}}} {strict,{cardinality,1}}
2018-07-24 19:12:36.377 [debug] pid=<0.1944.0> module=partisan_peer_service_server function=handle_info line=84 connection socket {connection,#Port<0.54951>,gen_tcp,inet,false} has been remotely closed
2018-07-24 19:12:36.384 [debug] pid=<0.1645.0> module=partisan_default_peer_service_manager function=do_gossip line=742 Sending state with updated membership: ['[email protected]','[email protected]']
2018-07-24 19:12:36.384 [debug] pid=<0.1645.0> module=partisan_util function=maybe_connect_listen_addr line=110 Node #{channels => [undefined],listen_addrs => [#{ip => {127,0,0,1},port => 10201}],name => '[email protected]',parallelism => 1} connected, pid: <0.1992.0>
2018-07-24 19:12:36.385 [debug] pid=<0.1645.0> module=partisan_default_peer_service_manager function=handle_info line=478 Node #{channels => [undefined],listen_addrs => [#{ip => {127,0,0,1},port => 10201}],name => '[email protected]',parallelism => 1} connected!
2018-07-24 19:12:36.397 [debug] pid=<0.1948.0> module=partisan_peer_service_server function=handle_info line=84 connection socket {connection,#Port<0.54954>,gen_tcp,inet,false} has been remotely closed
2018-07-24 19:12:36.485 [debug] pid=<0.1645.0> module=partisan_default_peer_service_manager function=do_gossip line=742 Sending state with updated membership: ['[email protected]','[email protected]']
2018-07-24 19:12:36.498 [debug] pid=<0.1950.0> module=partisan_peer_service_server function=handle_info line=84 connection socket {connection,#Port<0.55070>,gen_tcp,inet,false} has been remotely closed
2018-07-24 19:12:36.586 [debug] pid=<0.1645.0> module=partisan_default_peer_service_manager function=do_gossip line=742 Sending state with updated membership: ['[email protected]','[email protected]']
2018-07-24 19:12:36.600 [debug] pid=<0.1952.0> module=partisan_peer_service_server function=handle_info line=84 connection socket {connection,#Port<0.55571>,gen_tcp,inet,false} has been remotely closed
2018-07-24 19:12:36.687 [debug] pid=<0.1645.0> module=partisan_default_peer_service_manager function=do_gossip line=742 Sending state with updated membership: ['[email protected]','[email protected]']
2018-07-24 19:12:36.701 [debug] pid=<0.1954.0> module=partisan_peer_service_server function=handle_info line=84 connection socket {connection,#Port<0.55738>,gen_tcp,inet,false} has been remotely closed
2018-07-24 19:12:36.788 [debug] pid=<0.1645.0> module=partisan_default_peer_service_manager function=do_gossip line=742 Sending state with updated membership: ['[email protected]','[email protected]']
2018-07-24 19:12:36.802 [debug] pid=<0.1956.0> module=partisan_peer_service_server function=handle_info line=84 connection socket {connection,#Port<0.55819>,gen_tcp,inet,false} has been remotely closed
2018-07-24 19:12:36.888 [debug] pid=<0.1645.0> module=partisan_default_peer_service_manager function=do_gossip line=742 Sending state with updated membership: ['[email protected]','[email protected]']
2018-07-24 19:12:36.904 [debug] pid=<0.1958.0> module=partisan_peer_service_server function=handle_info line=84 connection socket {connection,#Port<0.55820>,gen_tcp,inet,false} has been remotely closed

%%% dist_lifecycle_SUITE ==> manual_start_stop: FAILED
%%% dist_lifecycle_SUITE ==>
Failure/Error: ?assertEqual({ok,'[email protected]'}, test_grain : node ( Grain1 ))
  expected: {ok,'[email protected]'}
       got: {ok,'[email protected]'}
      line: 108
.
%%% grain_lifecycle_SUITE: .......
%%% grain_streams_SUITE: ...
%%% grain_timer_SUITE: ....
%%% stateless_grain_SUITE: .
2018-07-24 19:12:46.725 [debug] pid=<0.3241.0> module=lager_handler_watcher line=119 Lager installed handler lager_console_backend into lager_event
EXPERIMENTAL: Writing retry specification at /Users/starbelly/devel/erlang/erleans/_build/test/logs/retry.spec
              call rebar3 ct with '--retry' to re-run failing cases.
Failed 1 tests. Passed 16 tests.
Results written to "/Users/starbelly/devel/erlang/erleans/_build/test/logs/index.html".

The finalize_and_stop function expects {save_state, State} or {ok, State} returned from deactivate, so let's specify that. I have a fix at #71 .

the erlans provider seems to have a optional callback post_init that isn't defined so makes dialyzer cry.

The specs of the read and read_by_hash callbacks in the erleans_provider behaviour are wrong, the third parameter is supposed to be an ID or hash value, not the grain reference. I have a fix for it in #73

Processes can subscribe to grains to receive notifications for grain specific events. If a grain supports observers a group is created through lasp_pg that observers are added to and to which notifications are sent.

Currently erleans doesn't do any of the cleanup it should when the application shuts down. It should attempt to remove all lasp_pg registered grains from the registry and then leave the cluster through partisan after gossiping out the change.

Timers are basically the same as erlang:start_timer. They are only active during activation they are created during.

It would be wonderful to have a cuttlefish schema?

A vonnegut should be included, at least as an example.

We've been using lease_time to mean the time a grain will wait without new requests before deactivating. This was chosen for the functionality at gofactory before the creation of erleans, but it maps to what Orleans calls "deactivation age limit", https://dotnet.github.io/orleans/Documentation/Advanced-Concepts/Activation-Garbage-Collection.html

At first I wasn't bothered by the name difference. But this issue on orleans has changed that dotnet/orleans#2428

Not simply because it uses lease time for something else but because it hit me that is more related to the common use of 'lease' in distributed systems -- lock leases, master leases, etc. While not a completely separate sort of use, similar stronger consistency in that Orleans issue may come up in Erleans where lease makes more sense (where it gets renewed), it may be worth renaming.

I'm not really a fan of "age limit" though, that doesn't convey that it is related to the most recent activity on the grain, as opposed to how long ago the grain was activated.

inactivity_timeout? since_last_request_timeout?

Starting erleans fails with (probably?) some unset variable:

2> application:ensure_all_started(erleans).
%%% ...
22:01:35.306 [error] gen_server erleans_stream_manager terminated with reason: no match of right hand value undefined in erleans_config:get/1 line 29

A persistent storage provider for postgres that is simple should be included by default, with an optional dependency on pgsql.

Pooling is an interesting question... sbroker is already included as a dependency so providing a default pooling strategy for connections may be a good idea. But we would want it to be just as easy to not use it, since many have their own.

For the default postgres provider the grain state should simply be stored as a blob.

The required metadata for grain persistence will need to be finalized so the columns can be defined.

Reminders are timers that are associated with a grain, meaning if a grain is not active but a reminder for that grain ticks the grain is activated at that time and the reminder is delivered.

Implementation will require working similar to streams and their use of "vnodes".

In the erleans_grain module I see calls like this: otel:current_span_ctx(). It looks like calling an earlier version of the OpenTelemetry API. Is it supposed to work or is it just a non-finished idea? If it supposed to work - how? What kind of metrics are collected?

verify_etag

erlans_grain (236) calls:

             {CbState2, ETag1} = verify_etag(CbModule, Id, Provider, ETag, CbState1),

if the grain either has no provider the variable Provider is bound to undefined, however verify_etag calls Provider:...() even if no provider is provided.

When the grain module was not loaded before erleans:get_grain/2 is called the return value is giving false results (probably due to checking for exported values).

3> Grain1 = erleans:get_grain(howl_grain, <<"grain1">>).
#{id => <<"grain1">>,
  implementing_module => howl_grain,
  placement => random}
4> howl_grain:module_info().
5> f().
6> Grain1 = erleans:get_grain(howl_grain, <<"grain1">>).
#{id => <<"grain1">>,
  implementing_module => howl_grain,
  placement => prefer_local,
  provider => ets_provider}

Consider supporting a function like erleans_providers:add_providers(Name, Module) that does not do the init_provider setup that calls start_child on the providers supervisor.

This would mean some providers aren't children of the supervisor though but maybe could just link to them in erleans_providers?

The idea could also be taken further and have all providers work this way. Instead of blocking during startup until all providers are started, let them start concurrently with the rest of the system, or even optionally start dynamically only when referenced by a grain.

Only concern with not setting up providers in the init/1 of erleans_providers is grain usage could happen before they complete and result in an error for the user. But user code should be able to handle temporary lose of providers... so maybe not an issue to care about?

hash_ring is missing as a dependency in the app.src file of (something)

See: lasp-lang/partisan#42

Currently the only way to have state in a stateful grain that isn't persisted when returning save from a callback is to keep both states in a map:

#{persistent => PState,
  ephemeral  => EState}

This may be a bit clunky. A tuple like {PState, EState} is an option, but prone to mistakes. Expanding the return options to support like, {save_reply, ok, EState, PState}, is also error prone, while the addition of the 4-tuple option also complicates the callback handler in erleans_grain, so {save_reply, ok, {e, EState}, {p, PState} would help against being error prone but still complicates the handlers.

Anyway, just some thoughts.

Which stream agents are responsible for polling from which streams is handled by partitioning up the streams based on a consistent hash of the stream reference and the corresponding node this maps to through a hash ring that is calculated based on the known members of the cluster.

The guarantee we must provide is "at least once delivery", periods where multiple nodes calculate that they are responsible for the same vnodes (and thus the same streams) is acceptable while cluster becomes consistent.

Currently this is done by simply monitoring for down/up events of nodes through disterl and the hash_ring library to build a ring based on the current view.

Much of riak_core is unneeded for our functionality since we mainly need to know what vnodes we are responsible for on a node and when it changes. The actual stream metadata is persisted through a provider, so I'm not sure we need handoff. Maybe we can utilize parts of riak_core for optimizations, but it certainly isn't necessary.

So taking parts of riak_core and building on top of partisan could be useful.

I'd like to discuss here what is necessary to support this in a way we are happy with (I am not happy with how it currently recalculates on net_kernel events and has no coordination between nodes that may be useful).

Note: Ideas separate from using the hash ring are also welcome!

Not sure yet how to reproduce this issue but I do see it in our environments that it tries to create a grain and fail with this. May be a race that allows it to start creating a new grain while another is already going that beats it to saving the grain, which then, rightly, causes the second to fail -- but I'm not sure that is actually possible, so have to investigate to be sure.

I'm undecided on this but it might be useful if a user can simply have a Repo setup that gets used by Erleans instead of what happens today -- a dynamic repo is started based on erleans application configuration and is linked to the Erleans providers simple-one-for-one supervisor.

It likely requires changes to the init_providers code in erleans_providers. Or to have a way to dynamically add providers, which is another issue.

the contract for all/1 is defined as {ok, [...]}:
https://github.com/GOFactory/erleans/blob/master/src/erleans_provider.erl#L29

however this expects a return of [...]:
https://github.com/GOFactory/erleans/blob/master/src/erleans_stream_manager.erl#L146

Instead of using monitor_nodes from net_kernel the erleans_stream_manager needs to hook into the partisan events for information on when nodes join or leave.