Better error reporting from gen_servers and other erlang applications about chef-server

[sdelano]

@jmink - We're triaging tickets now. Do you have any more context from this conversation that you can provide here?

[jmink]

This came out of a brief conversation with Phil from Facebook. Basically he doesn't know how to read erlang stack traces and would prefer some error message that's a bit more focused on what the user can do to fix things.

[MathildeBarr]

Hey @sdelano @jmink -- @mauriciosilva and I are on-site at Facebook, this is still an issue. This could be a CE project that @mauriciosilva can help out on.

[jaymzh]

@seth once said of this (roughly) "We totally need to do that - it would save everyone from customers to CSE countless hours. It's easy to do in a way we can re-use across all our erlang daemons"
@jmink echoed the clear wins here.

Filing tickets with a stacktrace because a config file doesn't have the right password in it, waiting for a CSE person to decode and explain possible things and then fixing the config file is a ginormous waste of time and energy. "Authentication to Postgresql failed" means no ticket, no waste of CSE times, no waste of customer's time, and a far, far better customer experience. You chose to use erland for the clear performance and testability wins - and that's awesome. But all languages have pros and cons, and the cost of using erland is there are no sane error messages, so you get to do that yourself.

Versions of this bug keep getting closed. It's incredibly frustrating, please stop. This is a clear win for everyone involved. Multiple engineers agree. Please figure out when to schedule it on a sprint (or your nearest internal equivalent).

[adamedx]

@seth, @sdelano, is there already thinking on how this can be done, e.g. a list of common cases where we can do something better than a stack trace?

[adamedx]

I'll take a first stab after getting some feedback from Phil / CSE:

• Our component data flow can guide the most common cases, i.e. failure to connect to services / resources such as databases, bifrost, bookshelf, solr, rabbit, etc.
• Such connection failures can be classified around authentication, authorization, general availability (resource does not exist), or worst case some "other" failure type.
• The message should include a human meaningful name of the resource, e.g. a hostname, postgres connection string, etc.
• Where possible, such errors should suggest corrective actions or diagnoses for the operator, e.g. ensuring a certificate is valid and verifiable, ensuring the database is reachable through some command line tool, etc.

I have more specifics in some notes. The goal here would be to let operators know information that will let them fix things quickly, and avoid the need for outside help.