callback interop about js-csp

[ubolonton]

Hi,

Sorry for the late reply.

I thought about callbacks/promises interop a bit, but I'm still not sure what would be the best way to go about it.

For converting callbacks/promises into channels, I think there are at least 2 possible approaches:

• 2 separate channels for result/error.
• A single channel that will deliver a [result, error] array (or an object with result/error key), and maybe an additional special take_that_may_fail operation that could throw an error, accordingly.

Related to the second approach, we can also allow processes to yield promises (similar to Twisted's inlineCallbacks), in which case it's probably better to add an intermediate promise-based API layer above the callback-based implementation, and re-implement the process runner on top of it (or use/copy one from a promise library, like https://github.com/ubolonton/twisted-csp/blob/master/csp/defer.py).

Any thought?

[jlongster]

For converting callbacks/promises into channels, I think there are at least 2 possible approaches

Approach #2 is what I'm doing. This is what I have in my utils.js:

function invokeCallback(func /*, args... */) {
  var args = Array.prototype.slice.call(arguments, 1);
  args.unshift(null);
  return invokeCallbackM.apply(this, args);
}

function invokeCallbackM(ctx, func /*, args... */) {
  var args = Array.prototype.slice.call(arguments, 2);
  var c = chan();
  args.push(function(err, res) {
    go(function*() {
      yield put(c, err || res);
    });
  });
  func.apply(ctx, args);
  return c;
}

I really like it. I haven't implemented error handling, but I think it would be good to have separate take that throws if an Error is passed. We need a good story for error handling anyway, so that's useful for all of our channels. Hopefully it will be named something short, but I can't think of anything better than maybeTake? On the other hand, the user could rename the take operations to map take to maybeTake by default when required:

var { maybeTake: take } = require('js-csp');

That way in app code that you know you always want to throw on error, it's easy. But that's probably not a good idea, as it makes the semantics confusing. Another option would be have a different kind of goroutine that throws whenever an error is received on a take instruction within it, named goAndError or something. The reason is that the semantic I want are certainly per-goroutine. In my app-level code, I always want it to throw on error. But in my utils code that implements specific compositions, I don't want it to.

in which case it's probably better to add an intermediate promise-based API layer above the callback-based implementation, and re-implement the process runner on top of it

So you would have to use a separate kind of goroutine? The thing I like about channels is that it's trivial to convert it anything else (callback, promise, stream) to a channel. I was planning on also implementing takePromise which simply converts the promise to a channel first. Easy. Though I guess how we handle errors does have some bearing on this.

[ubolonton]

If I understand it correctly, when an error is thrown in a go block (in core.async), it propagates to the bottom of the call stack. In browser this means it is silenced/logged into the console. In node this means crashing the process. When that happens, the "return" channel associated with that go block will be closed. The problem with this is that we cannot distinguish between this case, and the case where the last expression in the go block returns nil.

It is trivial in core.async to write a <? operation that is similar to <!, but throws if the value is an error. But then we will probably need a corresponding alts? operation. maybeTake in js-csp would have the same problem.

Another option would be have a different kind of goroutine that throws whenever an error is received on a take instruction within it, named goAndError or something. The reason is that the semantic I want are certainly per-goroutine. In my app-level code, I always want it to throw on error. But in my utils code that implements specific compositions, I don't want it to.

I'm thinking about something very similar:

• When a goroutine throws an error:
  • If it has a return channel, the error will be put on that channel.
  • If it does not, the error is let to propagate to the bottom of the call stack (similar to core.async).
• When taking from a channel, any object that is an instance of Error will be thrown.

This means that error chaining between goroutines can be achieved simply by having one goroutine taking from the return channel of the another goroutine. Only top-level goroutines (those without a return channel) need to handle the errors (before the system's handler kicks in). Non-top-level goroutines can be composed without thinking about handling errors. A problem with this is that even when an error was propagated through several goroutines, its stack trace will not show intermediate goroutines.

So you would have to use a separate kind of goroutine

No, I mean goroutine will be built on top of promises instead of raw callbacks. In Twisted this would integrate very well since everything is built on promises (deferreds). For example we can just yield http_request_as_deferred() for one-off task, without having to wrap it in a channel. Maybe it's not that great in Javascript where we have a mix of promises and callbacks everywhere.

[ubolonton]

Maybe we can ask the core.async guys what they think about this.

[jlongster]

If I understand it correctly, when an error is thrown in a go block (in core.async), it propagates to the bottom of the call stack. In browser this means it is silenced/logged into the console. In node this means crashing the process. When that happens, the "return" channel associated with that go block will be closed. The problem with this is that we cannot distinguish between this case, and the case where the last expression in the go block returns nil.

I don't think that's true; go blocks in Clojure(Script) never silence errors. At least, I'm not sure what you mean by "propagate to the bottom of the call stack". They have the same behavior as they do in any normal code. It's just that the call stack is lost (and that you can't "call out" to another process and catch errors with try/catch).

It's a good idea to automatically close the channel though if an errors is thrown. (not sure how that relates to the problem of distinguishing that and "nil"? I've been thinking we should change csp.CLOSED to a unique value and not null, also...)

It is trivial in core.async to write a <? operation that is similar to <!, but throws if the value is an error. But then we will probably need a corresponding alts? operation. maybeTake in js-csp would have the same problem.

Hm, good point. I'd like to work with js-csp a lot more before we go adding too much into it. I'll get a better feel for error handling that way. In my fork I did add takem which stands for "take maybe" and it throws on error. I suppose it wouldn't be hard just to have altsm. I'd like to keep it as simple as possible though.

Another option would be have a different kind of goroutine that throws whenever an error is received on a take instruction within it, named goAndError or something. The reason is that the semantic I want are certainly per-goroutine. In my app-level code, I always want it to throw on error. But in my utils code that implements specific compositions, I don't want it to.

I'm thinking about something very similar:

When a goroutine throws an error:
    If it has a return channel, the error will be put on that channel.
    If it does not, the error is let to propagate to the bottom of the call stack (similar to core.async).
When taking from a channel, any object that is an instance of Error will be thrown.

This means that error chaining between goroutines can be achieved simply by having one goroutine taking from the return channel of the another goroutine. Only top-level goroutines (those without a return channel) need to handle the errors (before the system's handler kicks in). Non-top-level goroutines can be composed without thinking about handling errors. A problem with this is that even when an error was propagated through several goroutines, its stack trace will not show intermediate goroutines.

So, in my fork I made it so that goroutines always return channels, like in core.async. It's just too common to want that. You're right though that if we force the user to tell us that we could automatically handle errors a little easier.

So you would have to use a separate kind of goroutine

No, I mean goroutine will be built on top of promises instead of raw callbacks. In Twisted this would integrate very well since everything is built on promises (deferreds). For example we can just yield http_request_as_deferred() for one-off task, without having to wrap it in a channel. Maybe it's not that great in Javascript where we have a mix of promises and callbacks everywhere.

So, personally I don't want to see promises anywhere inside js-csp. The reason is that I love the fact that a TypeError or ReferenceError inside a goroutine acts like normal (it actually throws) by default instead of in promises where they get gobbled up if you forget to end the promise chain. I'd be severely disappointed if we started using promises.

Like you said, JS has a huge mix of async handling (callbacks, promises, FRP, etc etc). I really want to keep things simple and explicit for now, and make it clear when you are dealing with what (take works with channels, everything else must be wrapped). I hate looking at a yield and not knowing what it's actually yielding, even if it magically works, since everything kind of has different semantics.

I'd like to keep things as they are for now. We need more real-world experience with this in JS before we can know what the right changes to make are. So far I've enjoyed passing errors along channels and using takem, and I don't feel like I need anything else yet, but time will tell. Passing channels along manually also allows me to wrap them with my current stack, which is interesting. (it's also interesting that 90% of the channels I use don't even use errors, for various reasons, like UI input). I will be building a lot of software with this, so I will get some good experience.

As far as I know that's where core.async is too.

[jlongster]

Also, we should discuss how this project is going to move forward. I made a fork with some changes but I'd like to have one canonical project. If you are interested in still maintain this, let's talk. Otherwise I'm happy to take it over. Either way can you send me an email at longster at gmail?

[pedroteixeira]

I've been using https://github.com/odf/ceci-channels - perhaps it might be of interest to have a look how similar things were solved - there a go block returns a promise, and registering with promise.fail would be the way of receiving the error thrown by the routine.

[getify]

FWIW, asynquence's go-CSP emulation returns promises from take(..), takem(..) and put(..) (if and only if they've been blocked), but since you immediately yield that promise back out, the runner(..) mechanism automatically takes care of blocking the goroutine until such promise resolves.

As for callback vs promise and synchronizing that approach, the takeAsync(..), takemAsync(..), and putAsync(..) methods all take an optional last param as the callback function, and if provided, it's called, but if omitted, those methods just return a promise (well, an asynquence sequence, technically, but kinda the same thing).

The main goal with asynquence is to bring promises, generators, and coroutines/goroutines into parity with each other so that you can mix-n-match them as you see fit. So, it's an important characteristic that sensible promise vending happens in such places.

[jlongster]

@pedroteixeira good point, will take a look at it.

@getify returning promises from those functions (in our case) doesn't add much since we already have our own dispatcher. But I think it's nice that both our libraries exist. If people want to mix promises with this, they can use your library, but personally I'd like to see this one be more explicit and only work with channels. It also gives us full control over the dispatcher behavior (buffering, etc). I also don't like how promises handle errors... but this is a huge topic :) We'll take is slowly.

[getify]

@jlongster... but what about takeAsync(..) and such? those can/should clearly return promises if no callback is provided, right, because those aren't blocking with your scheduler, no?

[ThomasDeutsch]

About error handling with CSP, i have found this article some time ago.
Maybe @srikumarks can help moving things forward.

[jlongster]

@jlongster... but what about takeAsync(..) and such? those can/should clearly return promises if no callback is provided, right, because those aren't blocking with your scheduler, no?

I just don't see much value in that. I'd rather not load in a big promise dependency. You really only use the async functions when you want to quickly put something on a channel outside of a go block (like creating a channel interface for DOM stuff), and you don't really even use the callback.

[jlongster]

About error handling with CSP, i have found this article some time ago.
Maybe @srikumarks can help moving things forward.

I found the dangling catch statement very strange, but maybe there are things we can learn from it. I don't think we necessarily even have a problem right now, and this issue could be closed. I'm going to write a lot of software with this and I'd like to only add stuff that is based in real experience using it (and so far, the Clojure community is doing fine with this style of passing errors errors through channels).

[ubolonton]

I think that building based on promise could be valuable, but only after the whole js world settles on promise. With the current state of mixed promise/callback, building on the lowest-level thing (callback) would be better.

We can come back to this issue later when this has been used more and the trade-offs become clearer.

[getify]

You really only use the async functions when you want to quickly put something on a channel outside of a go block

In just a few days since making the go-CSP emulation API, I've already found several useful places where asyncTakem(..) returning a promise, and that promise being error-rejected if an error is sent along the channel, has improved my code.

The whole reason I've switched to generators is to get sync-looking async code. generators+promises give that, and the very last thing in the world I'd want to do is go back to callbacks. They're the bad thing most of us are trying to get away from.

I just don't see much value in that.

I know you don't see much value in promises. I read your reasons in your posts, and I frankly couldn't disagree more.

But a huge chunk of the rest of the JS community does believe in them. IMO it's a silly stance to take that you'd build an "async" library that you want widespread uptake in that's willfully ignorant of promises.

...shrugs...

@ubolonton perhaps you're referring to a different JS world than I am in, but there's a huge contingent of people who have fully switched to the promises mindset.

All new DOM API's are being built as promise-based, ES7 is already de facto accepted the async / await syntax which codifies the marriage between generators and promises into direct syntactic support, there are dozens of hugely popular promise libraries with tens of thousands of libs/frameworks dependent on them... and on and on...

Anyway, couldn't disagree more with a somewhat callous "promises are just a fad still and I don't see much value". But heh, this is your lib, your choice. :/

I'd rather not load in a big promise dependency.

Promises are standardized and included in ES6, which is where generators come in too. They come together, so it's not like one is more a burden than the other. For generators, you have to use a big server-side transpiler tool to target older browsers, but for promises, you can use a super tiny polyfill like the one I built: native-promise-only. At 1.3kb, that's not exactly a "big promise dependency" IMO.

But again, your lib, your choice.

I guess when people find CSP and also want to integrate with promises (since all their surrounding code is likely to be that), hopefully they'll find asynquence more useful.

Sorry to have added noise to the thread. Cheers.

[jlongster]

The whole reason I've switched to generators is to get sync-looking async code. generators+promises give that, and the very last thing in the world I'd want to do is go back to callbacks. They're the bad thing most of us are trying to get away from.

Of course, I agree about callbacks. With js-csp you never use them. The putAsync and takeAsync only exist for the .001% of times that you aren't in a go block but just want to quickly put an item on a channel, usually this happens when you are converting an existing interface into channels (streams->channels). In user code you never use those. You are always inside a go block.

Anyway, couldn't disagree more with a somewhat callous "promises are just a fad still and I don't see much value". But heh, this is your lib, your choice. :/

Never said they were a fad, I don't think they are going away any time soon. JS has adopted them through and through. But I find it extremely confusing to think about promises, generators, and now channels. I just want one single abstraction, and it turns out that you can do promise-style code with channels very easily, and I personally like how it deals with errors better. So channels at the core, and provide functions to interface with promises/callbacks (yield takePromise(foo())) for using libraries. I find it a lot clearer about what's going on.

You're right about the dependency not being very big (and at some point they will be native). It's more of a mental thing; users have to ask "is this a promise? when do I do what?" Providing a single abstraction make it easier to explain.

The idea for this library (my vision at least) is that you wouldn't use promises much in your code. You would use channels. You'd only have to touch promises when interfacing with libs and such, and that's very easy. So we don't want you just using channels randomly here and there, use them everywhere (and now with transducers, you get any sort of transformation you would normally use anywhere else too!)

I do see your points, and I'm actually very happy that you've integrated CSP into asyncquence because you're right, if people want to mix abstractions they now have a choice! I will happily forward them to your library if they want that. Many people will find that useful because maybe their codebase already uses promises everywhere. I hope to show that this single abstraction is more powerful, though, and can be used a lot more widely.

[getify]

But I find it extremely confusing to think about promises, generators, and now channels. I just want one single abstraction

I have now explored in depth promises, generators, CSP channels, and reactive streams. I find relative strengths of each, and find that some places one fits better than others. The fact that CSP can handle the same task as one of the other abstractions doesn't necessarily convince me that it should. For example, the then(..).then(..).then(..).. chain in promises is sequential flow control, but I prefer (when possible) the sequential code inside a generator for that. But promises are much better for "in parallel" things IMO than say alts(..) channel stuff (thus far).

What I like more than "single abstraction" is "natural abstraction", meaning different problems have different natural ways to be expressed. When I find myself trying to think about a single-use channel as a stand-in for promise, for instance, it twists my brain more, because I already understand (and appreciate) the way promises are expressed as wrappers for future-values with immutable characteristics, etc.

I guess it comes down to "pick the right tool for the job" vs. "but i'm really good with a hammer". Pros/cons on each side.

---

👍 for choice.

[srikumarks]

(Responding late on this discussion. I'll try to convey the motivation behind the error mechanism in srikumarks/cspjs - i.e. the "dangling catch" et al.)

Obviously, I concur with @jlongster that CSP is a better model .. and I have a rather specific notion of "better". Being from an electronics background, I can think of "processes" roughly like integrated circuits and "channels" like pins and have a composable async abstraction to work with (has worked for me thus far). I do think working with values (promises) is just dual to this view - i.e. much like "verb at end" and "verb at start" grammars - and not any less in power ... just that they have a rather crippled expressivity in JS, unlike Mozart/Oz.

If you see folks adopting generators for async code, they often say they do it to get sequential-looking code, the only thing that breaks that sequentiality is the traditional try-catch error model borrowed into generators. I tried to code up what I felt would be a better approach to doing this ... for sync as well as async.

With srikumarks/cspjs, the "dangling catch" block traps all errors that emanate from that point on "down" as you're reading code, and the "dangling finally" block/statement declares a cleanup of some resource that got created "above" it. Down = future, Above = past, as the code executes.

The "dangling finally" actually has some subtleties that result in less error. An example -

task example {
    var f = fs.openSync("blah1");
    finally { f.close(); }
    await makeBlah2(f);
    f = fs.openSync("blah2");
    finally { f.close(); }
    await doSomething(f);
}

The above will do what you mean - close both files at the end - and not try to close "blah2" twice. i.e. the "finally" makes no assumptions about future activities. I find this relieves a lot of mental burden for me given JS's var lifting rules and my own stupidity. Similarly, "catch" can attempt recovery only based on future state.