Estimator's error bars about rfcs

[blakejohnson]

Reporting a confidence interval is about the closest thing we can get to allowing a user to ask for an "error bar" without making assumptions about the underlying statistics. So, I agree with Jake here that the interface should be explicit about returning an interval. The 1σ interval also makes sense as a default (a 68% confidence interval).

[1ucian0]

cc @ajavadia @blakejohnson @jakelishman

[jakelishman]

Personally my feeling is that if the "error bars" are going to be in the abstract interface, then returning some arbitrary confidence interval is the strongest trade-off between statistical approximations, not constraining implementations, and allowing users to do some very basic statistics on the output in the abstract. The absolute most basic Estimator use (estimate a single bit probability) isn't normally distributed, so having the abstract interface be estimations of the first and second statistical moments (without any sort of covariance) seems like an awkward initial choice, because the underlying distribution will be known to have significant skew for means away from 0.5.

Detailed statistics can be returned separately as an implementation detail, and given how specialised these would be per implementation, it's not clear to me that anything would be bought by trying to abstract that component (just as the RFC doesn't).

I don't feel strongly about this. My vote would go to returning the 16-84 confidence interval ($1\sigma$) such that the standard error is also trivially accessible for distributions that are normally distributed. I don't feel strongly about whether the confidence interval is represented absolutely or relative to the returned mean. I think matplotlib uses relative error bars, but it's one of the things I always have to look up every time I draw a graph with it.