Graphical posterior predictive checks about shinystan

[bob-carpenter]

I don't see how we could do this one from R. And I don't
see how we could do it easily directly in Stan C++.

In any case, I don't think it should be shinyStan's job
to do this.

• Bob

On Mar 1, 2015, at 9:52 AM, Jonah Gabry [email protected] wrote:

For users with posterior predictive simulations from generated quantity block we should allow them to input their original data from R and shinyStan should automatically generate a bunch of plots for graphical posterior predictive checking

—
Reply to this email directly or view it on GitHub.

[jgabry]

I actually already have some code from a while ago (before shinyStan when I was just starting to play with shiny) that does this to some extent. It would only apply to a subset of Stan models, but I made some tweaks to that old code and have a simple version of it working in the feature/pp_check branch. Here's an example of how it works at the moment:

1. I have the data in R that I pass to stan (e.g. an N-vector y)
2. In my stan model code I have a generated quantities that includes something like

  vector[N] y_rep ;
  for (n in 1:N) y_rep <- foo_rng(...) ; 

3. After Stan finishes I launch shinyStan and there's a new tab where I can select a parameter name (y_rep in this case) and select the name of an object from my R session (y in this case)
4. shinyStan will then generate a bunch of plots like the ones in BDA3 for posterior predictive checks

I think it would be really cool if this were part of shinyStan. Some users wouldn't or couldn't take advantage of it, but some could and maybe it would encourage people to do post. pred. checks if we make it easier for them (and if they see it as an available option they're not currently taking advantage of).

I can try to put add a working example of this to the online demo on shinyapps.io so you and Andrew and anyone else can see it in action and see if you think it's a good idea to include.

[jgabry]

oops, should be y_rep[n] <- ...

[bob-carpenter]

Great --- that would be useful.

I thought you were trying to do the work of the
generated quantities block from within shinyStan.

• Bob

On Mar 1, 2015, at 11:39 AM, Jonah Gabry [email protected] wrote:

I actually already have some code from a while ago (before shinyStan when I was just starting to play with shiny) that does this to some extent. It would only apply to a subset of Stan models, but I made some tweaks to that old code and have a simple version of it working in the feature/pp_check branch. Here's an example of how it works at the moment:

1. I have the data in R that I pass to stan (e.g. an N-vector y)
2. In my stan model code I have a generated quantities that includes something like

vector[N] y_rep ;
for (n in 1:N) y_rep <- foo_rng(...) ;

3. After Stan finishes I launch shinyStan and there's a new tab where I can select a parameter name (y_rep in this case) and select the name of an object from my R session (y in this case)
4. shinyStan will then generate a bunch of plots like the ones in BDA3 for posterior predictive checks

I think it would be really cool if this were part of shinyStan. Some users wouldn't or couldn't take advantage of it, but some could and maybe it would encourage people to do post. pred. checks if we make it easier for them (and if they see it as an available option they're not currently taking advantage of).

I can try to put add a working example of this to the online demo on shinyapps.io so you and Andrew and anyone else can see it in action and see if you think it's a good idea to include.

—
Reply to this email directly or view it on GitHub.

[andrewgelman]

If it can’t be done automatically it could make sense to have a template for it, a worked example in the doc.

On Feb 28, 2015, at 6:56 PM, Bob Carpenter [email protected] wrote:

I don't see how we could do this one from R. And I don't
see how we could do it easily directly in Stan C++.

In any case, I don't think it should be shinyStan's job
to do this.

• Bob

On Mar 1, 2015, at 9:52 AM, Jonah Gabry [email protected] wrote:

For users with posterior predictive simulations from generated quantity block we should allow them to input their original data from R and shinyStan should automatically generate a bunch of plots for graphical posterior predictive checking

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub #12 (comment).

[andrewgelman]

Ultimately, once we have the graphical models wrapper for Stan in R (or Julia, Python, etc), we can replace “y_rep” by something more general like REP(y) which figures it all out from the model.

On Mar 1, 2015, at 11:39 AM, Jonah Gabry [email protected] wrote:

I actually already have some code from a while ago (before shinyStan when I was just starting to play with shiny) that does this to some extent. It would only apply to a subset of Stan models, but I made some tweaks to that old code and have a simple version of it working in the feature/pp_check branch. Here's an example of how it works at the moment:

1. I have the data in R that I pass to stan (e.g. an N-vector y)
2. In my stan model code I have a generated quantities that includes something like

vector[N] y_rep ;
for (n in 1:N) y_rep <- foo_rng(...) ;

3. After Stan finishes I launch shinyStan and there's a new tab where I can select a parameter name (y_rep in this case) and select the name of an object from my R session (y in this case)
4. shinyStan will then generate a bunch of plots like the ones in BDA3 for posterior predictive checks

I think it would be really cool if this were part of shinyStan. Some users wouldn't or couldn't take advantage of it, but some could and maybe it would encourage people to do post. pred. checks if we make it easier for them (and if they see it as an available option they're not currently taking advantage of).

I can try to put add a working example of this to the online demo on shinyapps.io so you and Andrew and anyone else can see it in action and see if you think it's a good idea to include.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub #12 (comment).

[jgabry]

@bob-carpenter @andrewgelman

I just put up a (very rough) draft of the graphical pp checking in shinyStan online here so you can take a peak and let me know what you like or don't like, etc. Just click on the PPcheck tab (next to Explore) at the top.

I preloaded some toy data so you should be able go through the plots right away. Also, I haven't yet thought much about the layout, aesthetics, descriptions, etc, so this is definitely not what it needs to look like. And the About and Tutorial sections are little things I wrote up in 10 minutes to be placeholders until we have better versions.

[bob-carpenter]

That all looks great.

The only thing I don't understand are the scatterplots.
Is average y_rep the average of y_rep[n] over multiple
replications plotted against observed y? I prefer
seeing the line to shading the points. You're going to
get in trouble with alpha and overplotting I'd think if
you try to use shading. Similarly, I'd like to see a line
at 0 in the residual plot.

The "show a different replication" button is fun. Is
there any reason not to show a bunch at once?

• Bob

On Mar 2, 2015, at 9:05 AM, Jonah Gabry [email protected] wrote:

@bob-carpenter @andrewgelman

I just put up a (very rough) draft of the graphical pp checking in shinyStan online here so you can take a peak and let me know what you like or don't like, etc. Just click on the PPcheck tab (next to Explore) at the top.

I preloaded some toy data so you should be able go through the plots right away. Also, I haven't yet thought much about the layout, aesthetics, descriptions, etc, so this is definitely not what it needs to look like. And the About and Tutorial sections are little things I wrote up in 10 minutes to be placeholders until we have better versions.

—
Reply to this email directly or view it on GitHub.

[jgabry]

On Sun, Mar 1, 2015 at 5:34 PM, Bob Carpenter [email protected]
wrote:

That all looks great.

Thanks!

The only thing I don't understand are the scatterplots.
Is average y_rep the average of y_rep[n] over multiple
replications plotted against observed y?

Yup, exactly.

I prefer seeing the line to shading the points. You're going to
get in trouble with alpha and overplotting I'd think if
you try to use shading. Similarly, I'd like to see a line
at 0 in the residual plot.

Indeed, lines would be good.

The "show a different replication" button is fun. Is
there any reason not to show a bunch at once?

No reason, except that you get to click the button more! I can definitely
have it show multiple histograms of residuals and then the button can bring
up a whole new set of them instead of one at a time.

[bob-carpenter]

On Mar 2, 2015, at 11:49 AM, Jonah Gabry [email protected] wrote:

On Sun, Mar 1, 2015 at 5:34 PM, Bob Carpenter [email protected]
wrote:

The "show a different replication" button is fun. Is
there any reason not to show a bunch at once?

No reason, except that you get to click the button more!

Now that's the spirit of UI design! (That wasn't intended to
be sarcastic, by the way.)

I can definitely
have it show multiple histograms of residuals and then the button can bring
up a whole new set of them instead of one at a time.

On reflection, I think one set makes more sense. I just like being
able to see multiple sets of them like in Andrew's plots.

• Bob

[jgabry]

Small implementation of this will be included in v1.1.0 to be released soon. Will open another issue about expanding and improving the capabilities for later versions.