Spec the return value of audits about lighthouse

[brendankenny]

I've been looking over some of the equivalents of our architecture within catapult and I'm increasingly not sure here. At this point I don't know if we can plan a declarative tag structure that's sufficient to express all we'll need for our audits, as #61 (comment) anticipates. There's some evidence of this happening within catapult, with tacked on meta information where it looks (at first impression) like there were a few cases of "oh, we actually need to pipe one more thing through to evaluate this" happening :)

At this point I'd say I want us to avoid ending up writing an audits of audits layer here (and an audits of audits of audits layer...) until (and if) we really really need them, which means it's going to be hard to express categories with tags beyond ones that are scored with a simple combination of boolean states or subscores.

Sooo...barring a magical perf taxonomy handed to us on some stone tablets, maybe it's best to just avoid any architecture astronautism and continue with the more straightforward path we're on now. I could be very wrong on the above and tags will work just fine :) I think this will become increasingly clear one way or the other, and we just need to settle on something simple for now so we can continue with some auditing momentum:

Promise.resolve({
  rawValue: {}, // could be a primitive value or object, as long as it can easily be stringified and displayed, e.g. 'your score is bad because you wrote ${rawValue}'
  score: 0 // boolean or number
  tags: ['what have you'],
  description: 'whatnot',
  warning: 'some *specific* error value worth display, e.g. the location and nature of the syntax error you made' // the reporter can handle *general* feedback on how to fix, e.g. links to the docs,
  fault: 'some reason the audit has failed you, Anakin' // Optional argument when the audit doesn't cover whatever it is you're doing, e.g. we can't parse your particular corner case out of a trace yet. Whatever is in `rawValue` and `score` would be N/A in that case
});

After walking through some of catapult this afternoon, I feel like I have less insight here than I had before, so I'm interested in others' thoughts :)

[paullewis]

Cool, we need to make sure that we know how to bubble up to the top-level category / tag. For example, if you fail a manifest audit, what does that say for its tags? What if the category has 10 audits, and you fail 1?

[brendankenny]

Where we currently are:

Promise.resolve({
  value: 0, // "score", in our current use. boolean or number
  rawValue: {}, // could be anything, as long as it can easily be stringified and displayed, e.g. 'your score is bad because you wrote ${rawValue}'
  debugString: 'some *specific* error string for helping the user figure out why they failed here' // e.g. 'your manifest 404ed'. The reporter can handle *general* feedback on how to fix, e.g. links to the docs
  name: 'audit-name',
  tags: ['what have you'],
  description: 'whatnot',
  fault: 'some reason the audit has failed you, Anakin' // Optional argument when the audit doesn't cover whatever it is you're doing, e.g. we can't parse your particular corner case out of a trace yet. Whatever is in `rawValue` and `score` would be N/A in that case
});

We don't handle fault yet.

type definition for Closure here: https://github.com/GoogleChrome/lighthouse/blob/b921dd975394198cedd013e32e7483c2d2542ee4/closure/typedefs/AuditResult.js