deprecate domains about node

[RobertWHurst]

This is an interesting discussion. I want to add my own input and see what you guys think. I think it's important to keep things manageable inside node core, and I can imagine domains are a real pain to implement, but I don't see an alternative to them. Firstly I think I'm in alignment with the 3 points that @DaSpawn laid out above. Additionally I think try catches in every callback, or using promises and generators doesn't seem like a sane solution, in fact I believe it's just pushing the complexity into the hands of the engineers who use node. The whole point of node as I understand it is to make creating async servers less painful. Ryan achieved that goal by forcing the program to run in an event loop. One of the consequences of the event loop model is that you loose the call stack, including try catches, upon each tick. I think that is an acceptable trade off as the expressiveness gained when dealing with concurrent logic is very powerful. That all said, domains make the negatives of this trade off a lot less painful. Cordoning off subsystems within a process is a good way to manage uncaught exceptions. Those of us who author web servers or frameworks should be able to handle internal errors independent of the rest of the code running within the same process. With only one global catch all for uncaught exceptions you loose the ability for these systems to recover independent of the the rest. The separation of concerns is lost. I think errors within http handlers is but one example of this.

I'm looking forward to how you guys improve this solution, but I personally insist that domains or something similar need to exist in some form. No one ever writes perfect code, and everyone has triggered a runtime error at some point in their development experience using node. The expectation that engineers must write perfect code is unreasonable. To my knowledge no other modern language expects that of users. I feel domains, or something similar solve this issue. Without them error handling is seriously undermined.

[bayov]

I haven't read the whole thread, so sorry if I'm repeating things.

I want to say that I've recently been using domains to capture all calls to console.log, console.warn, etc. It is not a simple matter of monkey-patching these methods, since I also need to separate each call to its source (which is a unique context defined by my application).

So, I'm not using domains for handling errors at all. I don't even register an error event listener on my domains. I'm using domains as a way of execution context management.

This feature is necessary for my application to work, and so I hope that any replacement API to Domain will be able to cover my use-case...

BTW, I would be glad to know if there's a better way to do what I'm trying to do, other than abusing domains.

EDIT
I just wanted to add that it's not possible for me to improve my application to keep track of context manually, as my application keeps track of third-party user-code.

---

N.B.
I've also encountered the issues with native Promises. In my case, I fixed it by monkey-patching Promise (even though it might have a big performance hit):

function deferResult(fn, ...args) {
    let hasError = false;
    let error;
    let result;

    try {
        result = fn(...args);
    } catch (err) {
        hasError = true;
        error = err;
    }

    return () => {
        if (hasError) { throw error; }
        return result;
    };
}

const originalThen = Promise.prototype.then;
Promise.prototype.then = function (onFulfilled, onRejected) {
    const activeDomain = domain.active;

    let onF = onFulfilled;
    if (activeDomain && typeof onFulfilled === 'function') {
        onF = function (value) {
            activeDomain.enter();
            const deferredResult = deferResult(onFulfilled, value);
            activeDomain.exit();
            return deferredResult();
        };
    }

    let onR = onRejected;
    if (activeDomain && typeof onRejected === 'function') {
        onR = function (reason) {
            activeDomain.enter();
            const deferredResult = deferResult(onRejected, reason);
            activeDomain.exit();
            return deferredResult();
        };
    }

    Reflect.apply(originalThen, this, [onF, onR]);
};

[evanlucas]

@ORESoftware I've moderated your previous comment. I found it to be inappropriate for this issue tracker. We intend on keeping this issue tracker professional and would appreciate if everyone who posts here does the same.

[othiym23]

+1 sad 🎺

@Marak

I never understood why domains were added to core.

There's a place for unified sync & async error handling for Node, and due to the way that state propagates through core, there at least need to be hooks exposed by core to facilitate this. uncaughtException is too gross-grained, not everybody uses (or likes) promises, try/catch + stream error listeners + callback convention is complicated to teach and learn and harder to master, etc. Probably the best effort at this so far is @piscisaureus and co's zones, but that solution has limitations that are going to be tough to resolve outside core.

I'm hopeful that @trevnorris's AsyncWrap can provide the foundation for a better / lower-impact way of building something useful along these lines in the future, and I consider domains a very useful failure that helped point to what AsyncWrap / AsyncListener should do.

[naholyr]

👍 @scottnonnenberg
Never encountered any issue with domains. Never did some complex gymnastic with them anyway, but the API is very simple and straightforward: domain.create().run(foo), grab any error in an isolated "error" event. Good for me.

I don't get how we can deprecate something that covers a use case that cannot be covered otherwise, without providing clear (and ready) alternatives.

[addaleax]

@ORESoftware We have a Code of Conduct that asks you to be respectful of other people’s viewpoints and refrain from using language that “could reasonably be considered inappropriate in a professional setting”.

If you don’t feel comfortable with that, it might be easier to take a step back; Node’s project leadership is standing behind the choice to apply that CoC to our spaces.

[roberttaylortech]

Going to do my best to link together a few different ideas, issues and and some notes. I definitely don't have a full handle on all the moving parts, but I have plenty of experience trying to patch around the issues and resolve my own countless hours of pain. :)

• I think # 1 is that want to assert what I think is an important need and consideration as domains appear destined for deprecation.... something I've leaned on domains for (even though it's presented me issues as well): asynchronous context ala continuations local storage (CLS) via domain.active.
• Lots of stuff just doesn't seem to play well with continuations local storage
  • Such as: petkaantonov/bluebird#583 which is where I came upon the idea to divert to domain.active
  • And this as well: othiym23/node-continuation-local-storage#29

My initial issue wasn't related to Bluebird it was with session middleware - totally awful for an entire web app to fall down from that. So we switched to domain.active based on that note from the Bluebird issue and thought we were in good shape. However, next we ran up against: nodejs/node-v0.x-archive#8648 where anything using native Promises similarly lost context. We went ahead and dealt with out first major issue with native Promises that appeared in co with a little patch (tj/co#239). Seemed to work well - though maybe instead of that patch we could have switched to Bluebird for our co-routine execution and been solved? (Though with all these issues floating around, it's not exactly clear that would work)

All seemed well -- then we realized that in fact session middleware was losing context with domain.active just the same as it was with CLS ( is session middleware also using native Promises and suffering the same fate?) . We ended up tinkering for hours and believe we finally found a way to have the rest of the execution re-bound to the domain -- see my comment on this issue: othiym23/node-continuation-local-storage#29

I think its also noteworthy on the side (for those who haven't read the code of CLS) that CLS appears to globally monkey patch native promises (just to make aware another place things can fall apart.

Aside from tying together a bunch of related needs/issues, again # 1 is that whether it be CLS or domain.active or similar, it seems much needed (and definitely very much for product I am working on) that something dependable for asynchronous context emerge. Lots of moving parts, I know!

[emilsedgh]

Heartbreaking and surprising!

Domains are very useful for me. I would like to describe my usecase and hear your ideas about how it could be handled otherwise.

I'm maintaing an API and I want each endpoint to be atomic. That means, if an endpoint does multiple things and for any reason, fails one of them, I want all the changes to be reverted.

The way I handle this situation is using domains. For each request, a new domain is created. A new database connection is assigned to the domain.
If the request fails, the database connection ROLLBACK's and an error is sent to the user.

This is super useful and I would like to hear how other people handle this scenario.

The following gist is a code dump directly from our project but you get the idea:
https://gist.github.com/emilsedgh/a01a04b81846d7ae09dc

[juliangruber]

afaik no breaking changes will happen until v1

[jonathanong]

just labeling something as deprecated is not a breaking change

[juliangruber]

gotcha, 👍 from me, domains have only meant pain to me personally

[rvagg]

I'm on board, I bet a bunch of TC members would be too, marking as tc-agenda

[indutny]

+1

[Marak]

I never understood why domains were added to core.

It seemed wrong at the time, and still seems equally as wrong.

For backwards compatibility, will it be possible to put this functionality into an npm module as a custom add-on? Does this already exist?

[Fishrock123]

+1 domains are confusing and I hear nothing but incompatibilities and edge cases

[bnoordhuis]

It would have to go through the TC but I don't think there is much opposition to deprecating domains. Who volunteers to follow up with a pull request?

[tellnes]

@bnoordhuis A deprecation warning if you call domain.create()? Or when you require it since it has side effects (acording to code comments).

[bnoordhuis]

@tellnes I'll leave that to your own judgment. Doing it at require() time would get the message out most effectively but it might end up being extremely annoying.

If you do do it at require() time, you should also update lib/repl.js and make it load the domain module lazily.

[tellnes]

lib/repl.js needs to be changed anyway because it is calling domain.create().

[rlidwka]

Is there a suitable replacement that could be used? Repl is a valid use-case for example.

[tellnes]

@rlidwka Yes, the new tracing module in core. Please see my implementation in #75.

[jodytate]

+1 what @Marak said and +1 @Fishrock123 said, so +2

[jmcbee]

What is TC? Only TechCrunch is going in my mind.

[SomeoneWeird]

See Technical Committee

[benjamingr]

+1 domains have only brought me pain and suffering in my code and were not good enough in practice. I'd love to see them removed in v1 and deprecated today.

[scottnonnenberg]

If domain isn't the right approach, I'd love to know the standard recommendation for clean error handling in http servers. At minimum we need to be able to:

1. Return an error as a response to the original request that generated it
2. Shut down gracefully (no socket hang-ups), allowing time to finish other in-process async operations

Both of these are already implemented via domain in my library thehelp-cluster. As far as I can tell the only other way to do number 1 is via comprehensive Promises or try/catch wrapping. Number 2 is easier; it and other process-level stuff can can be done via process.uncaughtException. Is there something I'm missing?

[trevnorris]

I'm not sure I'd recommend doing a full deprecation until proper exception handling is added to AsyncWrap. The latest patch introduced just what was needed to get v0.12 out the door. Still need to add JS support for timers/nextTick, change the exception flow control to use the custom error callback and acknowledge the fact we won't be able to touch exceptions that occur in Promises.

[rvagg]

TC today agreed to a "soft deprecation" of domains, i.e. a documentation change rather than a full deprecation for now. Someone will accept #15 but it'll effectively be the last major addition to domains from here. Concerns were mainly around the lack of a solid alternative to domains for now but mostly the TC is +1 on moving to an eventual removal of domains, which would have a proper deprecation in the path somewhere.
See TC minutes in the docs directory for more information, merged in #144

[benjamingr]

+1 awesome news

[rvagg]

action being taken in #141, please comment there or open a new issue if you have anything further to add.

[scottnonnenberg]

@benjamingr I would love to hear more about your negative experiences with domains. As @rvagg said on #141 - "If you do a google search for "node.js domains" the results are almost all positive."

[algesten]

@scottnonnenberg @naholyr
The problem isn't the user experience of domains, as you say the API is simple and users are generally happy with it, but the consequences domains have in maintenance and complications for the node.js core code – that seemingly simple API impacts all of the code base in a too negative way.

Domains in its current form must go. That's certain. What the alternative is, no one knows.

I don't get how we can deprecate something that covers a use case that cannot be covered otherwise, without providing clear (and ready) alternatives.

If we flip that the other way around. We know domains must go, and will have to be removed at some point soon-ish. Would you prefer there be no indication of this?

[naholyr]

If we flip that the other way around. We know domains must go, and will have to be removed at some point soon-ish. Would you prefer there be no indication of this?

Well it's more acceptable with the upper explanation.

Anyway, I really feel like it should go only when there is an immediate gain then, or at lease a clear alternative. Like "look, as we removed domains, we could easily implement this long-time feature, go get it :D", or even better like "we removed domains, here is the equivalent code for your usual use case".
Or users feel the loss without any counterpart, and it will greatly impact release adoption.

[algesten]

@naholyr

Yes, this is exactly how the TC was talking about it and why it was called a "soft deprecation". At this point it's only a heads up to users that this API will be changing. However everyone is entirely clear on that domains fill an important role and requires some (maintainable) alternative.

[terinjokes]

I'm also running into the use cases where domains or CLS would prove useful, but after 11 months of "soft" deprecation, I'm we've still not been given any heads up on what to replace the use cases with.

Is this a problem still being worked on?

[Fishrock123]

@terinjokes Not really, it's pretty hard and most of us already have our hands full.

[terinjokes]

@Fishrock123 no problem, mostly wanted to check in.

[bjouhier]

@terinjokes
We (Sage) have some experience in this area, as we have been programming node.js with CLS and structured exceptions for almost 5 years.

We use CLS in our product to propagate our security and localization context.

We use structured exception handling because it gives us robust code: code where exceptions are always caught and handled in context (try/catch) and where we can guarantee (and easily verify) that resources are always released and program invariants always restored (try/finally). To be contrasted with code where exceptions end up in a global (context-less) uncaught exception handler or crash the whole process.

We do all our parallelization with futures (aka thunks). This blends very smoothly with structured exceptions. The important rule is that every execution thread should either be executed in fire and forget mode or joined with a parent thread (of course these threads are green). In the fire and forget case, every child thread is responsible for its own error handling; this happens naturally in the callback plug that terminates the child. In the joined case, any exception thrown by a child thread is propagated to its parent thread at the join point; it can then be handled in context in the parent.

None of this is really hard to implement as long as you accept to reify continuations in the language. This is what streamline.js does with its _ and what ES7 is going to do with async/await. CLS was not hard to implement in streamline.js because the code is preprocessed. With ES7 async/await it could be implemented by monkey patching promises (less obvious because promises have more surface API).

If you don't reify continuations (in the language or through special programming discipline), you will be struggling. There is no free lunch.

My 2 cents.

[nmccready]

Is continuation-local-storage not a viable alternative? It was listed here and seems to work quite well.

[mfuhrmeister]

@nmccready

As @souluniversal says... unfortunately NO!!
I could not find any suitable solution for the issues mentioned before.

EDIT: Zone seems to be (even though under development) promising!?

[Fishrock123]

@mfuhrmeister Zone looks like it was last updated a year ago, so it might be abandoned.

---

Most people haven given up on replacing/using domains at this point. Generally it's an unreasonable amount of work. Anything like this ties so deeply into everything it is almost impossible to maintain well. Also coming up with the correct API and error handling heuristics is extremely difficult.

Perhaps there will be an alternative in some future, but don't count on it. Please do not use domains unless you absolutely need them.

[cybrown]

Are domains going to be removed before any other solution like continuation local storage is available ?

[trevnorris]

No

[dman777]

I wish they did not do away with Domains, or at least deprecate it with a like replacement. I prefer Domains myself, the api was very simple and I liked the error reporting use of it.

[bjouhier]

@emilsedgh
Here is how we do it:

helper.withConnection = function(_, body) {
  var connection = pool.allocConnection(_, transacted);
  try {
    body(_, connection);
    if (transacted) connection.commit(_);
  } catch (ex) {
    logException(ex);
    if (transacted) connection.rollback(_);
  } finally {
    pool.releaseConnection(connection);
  }
}

// everywhere else
helper.withConnection(_, function(_, connection) {
  // do the job
});

We are writing our code in sync style with streamline.js but you could use other solutions: fibers, suspend, co, genny, ES7 async/await (with babel).

Domains is an experimental API. It gives you dynamic scopes but it has fundamental problems with dynamic nested sequences: enter D1, enter SUBD2, exit D1, exit SUBD2.

The combination of sync-style code and classical SEH (structured exception handling) is a simple and robust solution. Scopes are static and easy to analyze.

[emilsedgh]

@bjouhier Problem with that approach is that you will have to carry the connection around.

Models are usually something like this:

User.getById(user_id, cb);

I do not want to make something like this

User.getById(user_id, connection, cb);

How can I achieve this using any other solution?

My models are currently abstract from routes and express. They are not aware of requests at all.
All they do is to pick up connections from process.domain.db

This also allows me to implement other transaction policies.
For example in my cli scripts where they use the same models, there is no concept of request.
Therefore, my cli scripts provide a database connection on process.domain.db without the notion of transactions.

Domains allow me to have this whole thing neatly abstracted away.

I don't want to keep talking about my very specific use case. Just trying to point out that Domains are useful to some of us.

[bjouhier]

Problem with that approach is that you will have to carry the connection around.

@emilsedgh Not really. Streamline.js provides CLS (Continuation Local Storage). See https://github.com/Sage/streamline-runtime/blob/master/index.md. We don't use it for db connections but we use it extensively for locale and security context.

fibers also provides CLS, via Fiber.current.

I don't know about other libraries (suspend, co, genny).

[aaaristo]

It looks to me like ES6 generators have the potential to solve both error handling and context propagation issues. In fact one could just use try/catch for error handling and use the root function of the call stack as the scope for the context propagation. Here is an ugly gist:

https://gist.github.com/aaaristo/85415d4581ac6d719a5a

There the run function has a context variable that can be accessed by any generator function deep in the call stack (by calling the run.withContext generator), since essentially any generator function in the call stack is implicitly talking to the root function, to have its yield(s) resolved. I found this presentation much useful: https://youtu.be/DqMFX91ToLw?t=23m29s.

[DaSpawn]

I have recently ran into an issue with a module that would kill my express process before it could reply with an error message. I tried promises, try/catch, etc and nothing could catch the error where I could reply to the connection with an actual error message (I had to resort to secondary web sockets to carry back the error, sometimes). I have been trying to find a solution to this for years as my request handlers have no clue what socket they came in on, and the socket can not be easily passed to child processes

I found domains the other day, appear to be the greatest things since sliced bread, I am able to catch any error that happens within request processing and always reply with something, error or not, before I exit process (even all the error event catchers I had to litter add all over the place could not catch them all). When a browser returns a blank page the site is 100% broken to the end user, no matter what the problem really is.

and then I read domains are being depreciated, I was hoping to integrate into everything, that way all errors are isolated and easily identifiable (already shortened a module by half just eliminating all the try catch madness all over the place)

What is the alternative to domains that will allow me to catch an uncaughtException BEFORE it hits the main process? My exception handlers at the top should never be reached, they have no idea about the connections, and the fatal error is usually something pointless (like connection abort, bad file parsing, etc). My uncaughtException handler is going to have to become a huge beast just to handle various error handling from the different areas of the application

I found it incredibly easier to just throw whenever there was a problem anywhere, I knew the function would exit right there, but there is no way to catch a throw before hitting main process handler. Domains allowed me to catch any error, even if I threw them myself, and always return a response of some kind

Am I missing something here? what is the problem with domains? Why are they being depreciated without any real simple alternatives?

[ofrobots]

@DaSpawn There are a bunch of reasons why domains were deprecated. While there doesn't exist a definitive post-mortem on what all is wrong with domains, here is a list of things that didn't work quite well with domains: 1) domains didn't nest automatically, 2) subtle differences in sync vs. async error handling, 3) lack of proper resource finalization semantics, etc.

Having said that, domains are probably not going to be removed from core anytime soon. You can continue using them, just be aware of subtle semantic weirdnesses.

Node.js core recently landed AsyncWrap which contains enough low-level hooks that an alternative to domains to could be implemented on top. What should the user level API for this look like is an open problem. It is not even clear if async error-handling should be in core vs. the more minimal hooks that solve a more fundamental problem.

Some of us (at least me) think that Dart's zones provide good API and semantics at least for async context propagation and possibly for async error handling. Here's proposal on zones for JS.

This is far from decided at this point, but I expect some active work on this in the coming months.

[DaSpawn]

(Dart) zones look like a good idea, but zones should be as simple as domains if they are to replace them (ie using standard callbacks and not Promises)

[Marak]

Thought it over some more. Domains are still a bad idea.

You've got three choices instead of using domains:

1. Write correct code that always continues with error callbacks
2. Add a global exception handler to catch the uncaught error, fix it, remove global handler
3. If you are really stuck having to run unreliable code, then push it to a worker process via child process spawn and monitor stdout and stderr and standard spawn events ( error / close / end ).

FYI, I maintain a service which allows users to run arbitrary JavaScript code anonymously ( http://hook.io ), so I am somewhat familiar with this topic of discussion.

[Jeff-Lewis]

@Marak I agree with your strategies listed as best practices for node development. Though Domains offer more than just isolated error handling. There are many use cases for maintaining proper context across async boundaries for which many have tried to use Domains to solve.

Both Domains and CLS have their issues and drawbacks maintaining context and the node community needs a solid solution for this.

See recent proposals and discussions from the Ecma International group.

• Zone Proposal
• Async context propagation in various language ecosystems

Other related work:

• AsyncWrap

[Marak]

Yes. I read this document and also this library.

Smells like the smart people have gone off into a corner again. Kinda like when peerDependencies hit npm.

All of this is over-complication.

In a functional language, you should be use to passing scopes from top to bottom, and you should be good at it. If you don't like the mental model of this type of programming, you should switch to something more imperative like Python or Ruby.

Once you introduce the idea of domains or CLS into your JavaScript stack, you've essentially removed a level of simplicity in favor for complexity. I'd argue that with proper functional encapsulation and application design, you should never need either of these solutions and your application will be much easier to reason about.

[scottnonnenberg]

@Marak I have a really hard time with this line of thinking. It seems like you've got some sort of Javascript/async Stockholm Syndrome. "You mess up with async, you deserve it! We don't deserve better!"

I can't accept your item number 2 or 3 above, since it doesn't allow us to associate the error with the incoming request that caused it. This is important both for proper debugging and for surfacing some sort of error back to the original client request which caused the error. And number one is obviously impossible.

Additionally, I find it interesting that you consider domains to be a complication, when for client code they simplify everything greatly. hapi users don't need to their why their server stays up after an error. Most low-level libraries implement domains properly at this point, so the average developer doesn't have to think about it at all. And their applications aren't any harder to reason about.

[Marak]

since it doesn't allow us to associate the error with the incoming request that caused it

I've literally never needed to know this information. In the rare case where I need to run unreliable code and actually want to show the stack trace to the client, I simply use the third item I described above ( which is spawning a child process, letting it die, and piping stderr to client )

hapi users don't need to their why their server stays up after an error

Servers should not stay up after an uncaught error. They should die and restart immediately. An uncaught error by definition means that something unexpected happening in your application code. The moment something unexpected happens in your application you have entered an indeterminate state. Trying to reason about something in an indeterminate state is by definition difficult.

[scottnonnenberg]

@Marak 'unreliable code' is the same thing as 'all code.' Therefore, I do run multiple instances of my servers so one can go down an the others will continue to run. And I agree with you that servers should go down after an error.

But I also think that anything in-process in that server at the moment should be completed before that happens. We're not talking about toy apps here, we're talking about large-scale apps with many connected clients and in-process tasks spanning multiple data stores and third-party APIs. Yes, really bad things will happen (like power outages) but we shouldn't allow destructive things to happen any more often than they absolutely must.

[Marak]

@scottnonnenberg -

Why exactly are your applications crashing in the first place? Is there a classification of error which is frequent? Are you dealing with multiple contributors and insufficient test coverage?

Simply speaking, the only time I've ever encountered actual uncaught errors in high-traffic node.js applications has been around things like calling res.writeHeader with invalid data. Was very easy to identify and fix without much debugging.

I don't need tasks to complete in my applications. If any task is important enough that completion must be guaranteed, then what I really need is acknowledgment receipts with a distributed consensus ( since the task could complete, but power-out on sending acknowledgment ).

If you are interested in which syndromes I may or may not have, maybe try taking a read at this article:

https://hook.io/blog/the-monolith-versus-the-microservice-a-tale-of-two-applications

My general attitude is not You mess up with async, you deserve it!, but more like, We are all going to mess it up. Better to isolate the problem in a separate process.

[ofrobots]

Folks, lets keep this issue for discussion on deprecation of domains. This thread should not be used to debate over merits of different application architectures.

[Fishrock123]

Most people haven given up on replacing/using domains at this point. Generally it's an unreasonable amount of work. Anything like this ties so deeply into everything it is almost impossible to maintain well. Also coming up with the correct API and error handling heuristics is extremely difficult.

Perhaps there will be an alternative in some future, but don't count on it. Please do not use domains unless you absolutely need them.

So in the time between my last comment, @trevnorris has been working a good deal on something known as AsyncWrap.

Domains are a huge headache for us to maintain. Subtle edge cases abound which are difficult to fix, and exceedingly hard to reason about.

While simpler domain use cases may be nice, working magic, those edge cases are a nasty minefield and only make things worse for everyone.

In light of this, AsyncWrap has been a long time in the coming. Base parts have already been implemented for a considerable amount of time but nailing down a good JS API for AsyncWrap is important, which is why it's taken quite some time.

While AsyncWrap is not a drop-in replacement for domains, we're very confident it will be able to handle more cases, better, and also be far less of a burden for us to maintain.

"AsyncWrap dictates how things will work. whereas domain has to consume how everything else has been built." — @trevnorris

Note that the timeframe for which domains may possibly be removed is not within sight yet and they will continue to work. We'll update everyone once more progress has been made. :)

For now however, the official recommendation on domains will continue to be the same: Please do not use domains unless you absolutely need them.

(Also for anyone who asks about Zones: AsyncWrap pre-dates them, and is likely able to handle more use-cases.)

[Fishrock123]

(Also please don't post +1 / -1 etc comments or things that have already been discussed above, I'd rather not have to remove comments or lock this. Thanks!)

[DaSpawn]

This appears to come down to personal preference/perceived place in node rather than specific need to remove it for a security or functionality reason.

@Marak you make excellent points, but change the perspective; I need a safety net from the end user input, not an error handler:

1. Domains should not be used specifically for error handling overall, that would appear to be a lazy/bad way to use them, and should still fall under process handler. My code should never do anything I do not expect it to, so I still have error handling for specific cases as needed/expected. What I can not guarantee however is user input. If a user uploads a file that is somehow unexpectedly incompatible, now or in the future, I do not want that to crash the process, there is no reason to, what we need to do is gracefully exit without affecting any other current user requests/actions, which could also be in the middle of processing a task and have no problems, and it does not matter what the task is, it is all simply bubble wrapped in a domain as needed
2. global exception handler should be simple and never change, it should be the all else has failed and ensure exit and cleanup handler that never gets called. The possibilities of problems arising from changing the handler randomly are limitless, plus, you should never mess with objects above you unless you are supposed to (top down)
3. unreliable code is not the purpose/need for domains. I already make use of child processes to handle requests, but the code is stable and error free, the user input is not. I need to ensure one users actions do not kill a web server process just because they inadvertently uploaded the wrong file and abort other requests in the process at the same time. Child processes are extremely slow to fork, as expected, and would make a site crawl if I had to fork every request just so I have simple request isolation to prevent unexpected errors affecting anyone but the user that caused it. I can maintain the same task isolation I need with domains without child processes. Child processes also add much complication to a project just by themselves, and unless you have linux background/knowledge they can be easily implemented wrong in scale

domains are the perfect fit for many situations where you want to ensure task containment but not for overall error handling, but how people use them is their choice. Domains is very simple small amount of code, and part of the core of node, ie they require no other modules but core modules to function, and their simplicity ensures that complications in the domain implementation is not the creator of the error, so adding/needing promises or other large modules at the top of code structure could be more problematic. Domains follow the core principle of node, callbacks

This is not about ensuring node stays like other functional languages, this is about ensuring node gives the greatest flexibility in functions for it's users needs. Domains ads a perfect simple dimension to an application that gives much greater confidence in application stability and uptime that would require adding other complicated modules and methods to achieve, and as @ofrobots said this discussion is not about this, just domains themselves

@Fishrock123 AsyncWrap sounds interesting, but that was not the use case here and ads much more complications/code, all I needed was a safety net and domains provides it perfectly. What is the edge cases you are talking about that makes it difficult to maintain? AsyncWrap sounds like it could be a better replacement in those cases though.

Just having domains listed as depreciated means that very simple built in functionality could disappear at any time, leaving usage 100% unreliable for future planning, even if domains stays forever

[algesten]

@DaSpawn the counter argument to your point has already been made below. in short: no one is challenging the usefulness of domains. however the technical debt/consequences throughout core are too much to keep them.

[Fishrock123]

Domains ads a perfect simple dimension to an application

Right, it's a magical poison chalice.

Domains is very simple small amount of code

This is most definitely not true.

What is the edge cases you are talking about

There's some info here on AsyncWrap vs Domain vs Zones: angular/zone.js#231 (comment)

that makes it difficult to maintain?

See my above post: "AsyncWrap dictates how things will work. whereas domain has to consume how everything else has been built." Domains tie into absolutely everything in the worst ways.

Here is a domains bug no-one knows how to fix because it makes no sense or reason at all: #1271

Just having domains listed as depreciated means that very simple built in functionality could disappear at any time, leaving usage 100% unreliable for future planning, even if domains stays forever

I just said above we won't remove it until a good while after AsyncWrap is in. Please be sure to re-read my above post as I cover a lot of things I don't feel like having this thread go much longer than it already is.

AsyncWrap sounds interesting, but that was not the use case here and ads much more complications/code, all I needed was a safety net and domains provides it perfectly.

You should still be able to implement the same thing in AsyncWrap, and it will probably be more reliable.

[Fishrock123]

(Also to be clear: The decision has already been made long ago that we are moving away from domains.)

[DaSpawn]

Thank you for taking the time to clarify everything, lots of good information; hopefully a solid replacement recommendation (like AsyncWrap) can replace/compliment the long term depreciated page/flag

[ORESoftware]

I am not sure I understand, any replacement for domains would basically have the same behavior as domains so why replace it?

[algesten]

@ORESoftware

the technical debt/consequences throughout core are too much to keep them.

[ORESoftware]

@algesten I think you mean that domains require way too much global state and are therefore way too expensive to maintain, which makes more sense than what you just said? I would buy the argument that I just made, but we need something like domains, the functionality is critical. I haven't checked out the alternatives to domains, but the only good alternative is one that somehow manages to avoid global state somehow, otherwise it's the same poison.

[benjamingr]

@ORESoftware your promises example would not suppress the error. You need to add a rejected promise handler - https://nodejs.org/api/process.html#process_event_unhandledrejection

[ORESoftware]

@benjamingr thanks that's actually some pretty useful info - I think it's on the wrong thread though

[Fishrock123]

I am not sure I understand, any replacement for domains would basically have the same behavior as domains so why replace it?

Not necessarily. Please see my above comments for more of a write up. AsyncWrap is capable of achieving the same goals, at almost no cost to us. The API and behavior of AsyncWrap itself will not be identical, but you would be able to make a domains-esque module with it, that should cover all the same use-cases.

[ORESoftware]

@Fishrock123 nah, from what I have read, I don't think asyncwrap can do what domains can do. domains provided the possibility of better stack traces and universal error handling for the entire node.js ecosystem. AsyncWrap looks like it might be only useful for applications, where applications can protect themselves from their own code but it cannot protect applications from unexpected errors thrown in other libraries nor can AsyncWrap allow for development of test frameworks where errors are expected to be thrown from asynchronous code. The reason why I am fairly well familiar with the issue at hand is because I am trying to develop a test framework and I am realizing it is impossible to create high quality software in Node.js without 100% domain support. See this issue in Ava, an up and coming testing library that surely will I believe be thrown for a loop by the issue of domains no longer being supported

avajs/ava#567

I repeat, you cannot create a solid asynchronous testing library for node without something very much like domains and asyncwrap and promises do not solve the problem, not even close.

[DaSpawn]

@ORESoftware this is the same wall/conclusion I arrived at also as I had implemented domains in a multi-fauceted dev project and it works perfectly as expected/needed to isolate everything and still catch the errors that are expected/needed to debug quickly. I came to node docs to get more domain info, then seen the depreciated and arrived just where you are also

Node has stated that this decision was decided long ago to depreciate domains (not going to just remove domains without something that can cover the use-cases though), and there is nothing we can do about it really. Honestly this makes absolutely no sense to me as there is significant need and use for domains we both have arrived at in creating enterprise grade software, but they do not see these use cases and insist the functionality should be in another module since too many people use domains exactly as they should not, expect them to work, then complain when their projects break due to issues already known with domains

[Fishrock123]

I'm pretty sure this has been stated before, but: we are not going to just remove domains without something that can cover the use-cases.

[ORESoftware]

@Fishrock123 well, first of all, Node core did "remove" domains because they aren't supported by ES6 native promises, so that's already a problem for me. There are always use cases that are beyond the comprehension of a single person, but hopefully not all of Node TC. here is what you might not understand - the only thing that can do what domains do is domains. Or something exactly like them, different in name only. They are a clever solution to a difficult problem and currently I believe the only solution to that problem. Any other solution will not cover all of the use cases that domains cover. The use case that I have, as I said, is creating a test framework that can run tests asynchronously (in "parallel") in the same Node process. What this means is that at the global scope I really don't know for certain which test is running at any moment, because of asynchronous callbacks that could be invoked at any time. So if errors bubble up to 'uncaughtException' (which was at one point deprecated by the way, but I guess they backtracked on that) or 'unhandledRejection', it's already too late, I cannot pin the error to a specific test. Domains, however, solve my problem perfectly. I am sure supporting domains is difficult, but I think it's way worth it.

[rvagg]

If you have a usecase for Domains where they are currently broken, native Promises for instance, then you are welcome to work up a pull request to fix it how you consider appropriate and we can have that discussion then.

There is currently nearly zero interest amongst the core team for maintaining promises so complaining here will get you nowhere but this is how open source works. Contribute something acceptable and you'll be welcomed with open arms, in fact it'd probably be nice to have someone working in core who cares about domains (rather than actively hating the difficulties they cause like many of the team). Just keep in mind that anything beyond strictly "fixing" domains, such as adding new functionality to domains, is not likely to get anywhere either due to the deprecation status.

It might also be worthwhile investigating the potential that AsyncWrap has for building solutions to the same kinds of problems that domains were attempting to address. You may find that use cases are already solved in fact. The API hasn't been finalised or formally approved but it exists undocumented in v4 and v5 if you want to start trying it out and contributing to its evolution.

[ORESoftware]

@rvagg thanks, and thanks to @Fishrock123 for the discussion - I want to patch ES6 promises so that they work with domains, I presume the location for native promises is here

https://github.com/v8/v8/blob/40d3095c2e16106f122db744a29a9e1a804442ce/src/js/promise.js

looks pretty nasty to me, not sure I am capable of creating a quality patch, TBH, but I will give it a go tomorrow

I am not against AsyncWrap if it can solve the problems I have now that are only solvable with domains, but right now I am simply doubtful on that.

[jondubois]

Domains are useful for frameworks - They're a good way to separate 'user logic' from internal 'framework logic' for the purpose of failure handling and recovery.

Note that 'failure handling' is different from 'error handling' - A failure is a side effect of an Error - The user of a framework should be allowed to handle errors however they like, but sometimes you want the framework to decide how to handle the subsequent failure which resulted from an error (perhaps even if the error was caused by the user logic).

For example, the framework itself might invoke a user-defined function (for example, middleware) which might branch off (in unpredictable ways) and invoke various functions synchronously and/or asynchronously - Under such circumstances, if the user's code throws an Error, you may not want to just kill the whole server (especially when it comes to WebSockets because then you would end up killing tens of thousands of perfectly good connections along with it) - You just want to shut down the single socket which was related to the problem.

Basically, as a framework developer, domains offer you the ability to control the failure handling (and recovery) aspects around code which was written by a third party.

A big practical problem for example is if there is a logic error in user code - For example, if the user code tries to invoke a function which doesn't exist - You DO NOT want the server to crash (and take down all other sockets) - You just want to kill a single socket.

If the error happens within the app's core logic (not related to a specific socket or request, but to the system as a whole), then in that case, you may want to bring down the whole server.

By removing domains, we may be forcing users of frameworks to devise their own 'failure recovery' procedures which would otherwise have been handled seamlessly by the framework itself (in the most optimum way).

EDIT Upon further consideration, it appears that Promises offer similar features as domains when it comes to failure containment. Anyone tried this approach? For example, you can just add a .catch(handler) to a promise which is wrapped around the user logic.

In any case, I think this approach should be investigated thoroughly (and documented) before domains are removed completely.

[ORESoftware]

100% agree with that assessment. As a test framework developer where user
code calls the framework api, domains are an imperative. In the case of
test framework, errors/exceptions are supposed to happen and be handled
without explicit use of try/catch.

On May 4, 2016 6:13 AM, "Jonathan Gros-Dubois" [email protected]
wrote:

Domains are useful for frameworks - They're a good way to separate 'user
logic' from internal 'framework logic' for the purposes of failure handling
and recovery.

Note that 'failure handling' is different from 'error handling' - A
failure is a side effect to an Error - The user of a framework should be
allowed to handle errors however they like, but sometimes you want the
framework to decide how to handle the subsequent failure which resulted
from that error.

For example, the framework itself might invoke a user-defined function
(for example, middleware) which might branch off (in unpredictable ways)
and invoke various functions synchronously and/or asynchronously - In such
a situation, if the user's code throws an Error, you may not want to just
kill the whole server (especially when it comes to WebSockets because then
you would end up killing tens of thousands of perfectly good connections
along with it) - You just want to shut down the single socket which was
related to the specific action which caused the problem.

Basically, as a framework developer, domains offer you the ability to
control the failure handling (and recovery) aspects related to code written
by a third party.

A big practical problem for example is if there is a logic error in user
code - For example, if they try to invoke a function which doesn't exist -
You DO NOT want the server to crash (and take down all other sockets) - You
just want to kill a single socket. (That would classify as a security
vulnerability).

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#66 (comment)

[jondubois]

The other really nice feature which domains offer is the ability to listen to the 'error' event on an EventEmitter without explicitly binding a listener to its 'error' event. E.g. using domain.add(emitter).

This relates to a common use-case for frameworks - You don't want to allow the user to accidentally unbind 'error' handlers which were setup by the framework for its own internal use (e.g. if the user calls eventEmitter.removeAllListeners()) - Because that would mess everything up internally (the user should not be able to unbind error handlers which they did not attach themselves).

[ORESoftware]

does anyone know if there is indeed an effort to patch native ES6 promises to work with domains? Will that land in a Node version soon?

[trevnorris]

@ORESoftware Either we would need to overwrite native Promise, or use a V8 provided API to propagate the information. The later is slow going and won't happen soon, and doubt the CTC would be willing to patch Promise directly.

[ORESoftware]

@trevnorris ok thanks for that info.

[jondubois]

@trevnorris

Instead of messing with native Promises, couldn't we just rewrite the domain module itself to leverage Promises instead? It looks like doing this would allow it to automatically work with nested Promises and also with async operations like setInterval, nextTick etc...

So for example, the domain.run(fn) method could just be:

Domain.prototype.run = function (callback) {
  var self = this;

  (new Promise(function (resolve, reject) {
    var args = Array.prototype.slice.call(arguments, 1);
    callback.apply(self, args);
  })).catch((err) => {
    this.emit('error', err);
  });
};

I tried this in a number of async scenarios and it seems to behave the same as with the Node.js domain module's run(fn) method - It's actually better because unlike the domain module, this doesn't pollute Error objects with error-domain-related properties... The promises do everything magically and capture any async errors.

I guess it doesn't capture emitted 'error' events though...

[ORESoftware]

@SosZ please search and read the whole thread before, I patched promises to use domains myself, in a much simpler way

Your code is a little bit of a horror show. What about this instead?

const then = Promise.prototype.then;
Promise.prototype.then = function (fn1, fn2) {

  if (process.domain) {
    fn1 = fn1 && process.domain.bind(fn1);
    fn2 = fn2 && process.domain.bind(fn2);
  }

  return then.call(this, fn1, fn2);
};

Note that you do not need to patch the Promise constructor, because the executors run synchronously, but you can patch Promise.prototype.catch to work in the same way as the above patch on Promise.prototype.then. That being said, pretty certain Node version 8 has allowed domains to work with Promises, so the above patch may only be useful for Node.js versions < 8.

[ORESoftware]

I just want to add my 2 cents to the long run discussion. I have a job. I want to keep my job. I really prefer to not have my server(s) crash, even if we use forever or supervisor to restart upon crashes.

Using the following allows us to avoid many crashes:

let domain = require('domain');

app.use(function(req, res, next){

  let d = domain.create(); // create a new domain for this request

  res.once('finish', function(){
    d.exit();
    d.removeAllListeners();
  });

  d.once('error', function(e){
    // log the issue right here
    if(!res.headersSent){
        res.json({
          error: e.stack || e
       });
    }
  });

  d.run(next);   // we invoke the next middleware
});

we use domains in production. Hopefully no memory leaks though 😂

[nickmccurdy]

Domains are still a hack, letting the process crash is often the better option especially with forever/supervisor.

[ORESoftware]

@evanlucas censorship always appreciated, but I guess you're right, best to keep politics off of Github lol...@nickmmcurdy - you'd have to back up that statement - domains will fail in their intention if an error is raised for the wrong path of execution, and if it's easy to misuse them than that's also bad. But I don't think they are a hack perse, just perhaps imperfectly implemented.

[ORESoftware]

Imma read the CoC right now to see if my microaggression actually violated it.

[ORESoftware]

Yep, looks like I violated point 1

• Using welcoming and inclusive language

whoops, my bad fam