Ephemeral State and `erleans_grain` behaviour callbacks about erleans

[jadeallenx]

Yes that's what I mean. An opaque data structure that's handled by a module API will help protect against people crafting them by hand.

[tsloughter]

Kicking this around again. Some examples:

{reply, ok, State#{counter => Counter+1}, [{save, Counter+1}}.

{reply, ok, State#{counter => Counter+1}, [save_state]}.

{save, State#{counter => Counter+1}, [{reply, From, ok}]}.

{save_reply, ok, State#{counter => Counter+1}}.

NewState = {EState, PState} = erleans_state:update(persistent, OldState, Counter+1),
{reply, ok, NewState, [save]}.

I definitely think we should also support, similar to gen_statem, the triggering of new events in the return, so like [{info, increment}] to run handle_info(increment, ....

The save_state option would mean the return State variable is persisted. The save tuple means persist this variable. While the last one with just save is special since it keeps the persistent and ephemeral state together, so you only have to say save and erleans knows which to save.

[jadeallenx]

Perhaps make state opaque as a data structure and use tagged tuples to implement it - maybe it's hidden behind a module interface?

OldState = {{e, <<"bar">>}, {p, <<"hoge">>}},
NewState = {EState, PState} = state:update(ephemeral, OldState, <<"foo">>),
{save_reply, ok, NewState}

[tsloughter]

Ah, you mean to guard against the mis-ordering of e and p since it is handled by state:update?

[cmeiklejohn]

I'm with @mrallen1.

[evanmcc]

I also agree. both get and put should be via the module.

[tsloughter]

What about the case with no ephemeral state, seems we'd still want to have the same interface used no matter if the grain has only persistent state or not? And my only issue with putting update behind a module/function is not being able to pattern match in the function header of handle_call.

[jadeallenx]

There'd have to be some kind of underlying convention about how to represent all possible valid state combinations: persistent and ephemeral, just one of them, none of them (stateless). I think it might be possible to add some macros in a header file which let you do pattern matching while still keeping the implementation opaque.

case erleans_state:get(persistent, Foo) of
   ?PSTATE(#{foo := bar}) -> do_something();
   _ -> ok
end

Maybe that's too ugly? I'm not sure. But it should be possible. Maybe it could be a parse transformation if the macro is too hideous to contemplate.

[tsloughter]

Another idea, move saving to an optional list of actions to perform after returning. The state returned is then treated the same as gen_server:

handle_call({set, x, X}, _From, State) ->
  State1 = maps:put(x, X, State),
  {reply, ok, State1, [{save, State1}]}.

or

handle_call({set, x, X}, _From, State=#state{whatever = Persistent}) ->
  Persistent1 = maps:put(x, X, Persistent),
  {reply, ok, State#state{whatever=Persistent1}, [{save, Peristent1}]}.

The issue with this one I guess is having to include the right variable twice... hm. Yea, may have to stick with Mark's idea.

[michalmuskala]

Thanks for pining me on IRC @tsloughter. It took me some time to finally get to this.

I think going for a more gen_statem-like API could be beneficial. This could allow fairly easily solving couple rather hard issues. Two main ones are addressed in this PR to OTP erlang/otp#1429.

While, I don't think asynchronous initialization will be a big problem with erleans (given the initialization can happen at any time, anywhere), checkpointing the state (especially the persistent one) during operations will be an issue.
I can easily imagine an operation, where I'd like to persist the state in the middle, but do something before replying to the sender, possibly mutating the state even further. One example could be an event-sourced system where we want to persist events before applying them, that in turn may produce even more events to persist.

Adopting a gen_statem-like interface would also allow easily extending the return tuple with additional operations. The gen_server interface is rather rigid in that regard (the number of possible combinations becomes ridiculous).

As to state representation itself. I don't have much experience designing interfaces for Erlang. If I were to design the interface for this in Elixir, I'd probably go for something similar to how Plug.Conn works. A top-level map with two sub-maps - one for the persistent state, the other for the ephemeral one (conn has those for "assigns" - user state and "private" - library state). Elixir makes it easy to access nested maps with e.g. conn.assigns.foo or with pattern matching - in Erlang it's a bit more verbose. What's also important is that the Plug.Conn struct is usually treated as a semi-opaque data structure. It's generally modified only using library functions, but it's often read using pattern matching or with . syntax. Formally it's not opaque and you're "allowed" to modify it directly, but that is rarely done in practice - usually only in libraries or library-like modules, extremely rarely mixed with logic.