We had envisioned these being either called via plot.sensemade(type=), or directly by the user.
The latter is not generally working -- special quantities are computed in plot.sensemade, such as benchmarks$benchmarks_2plot1, which are not present if user calls directly. For now I'm removing the "@export" on these, but we should decide if we want them to be user facing, and clean up as needed if we do.

save for end: remove internal reminder / comment notes

worstcaseplot, if called directly, fails when no default limit is given:

worstcaseplot(sense.out)
Error in seq.default(0, lim, by = 0.001) :
'to' cannot be NA, NaN or infinite
In addition: Warning message:
In max(r2d, na.rm = TRUE) : no non-missing arguments to max; returning -Inf

Thanks for drafting the vignette, MT! However I'm having trouble getting it to open using vignette(). I've checked and the yaml looks right, the directory name looks right, the Description appears to have updated. I've used devtools::build() to force it to build the vignette, and it does, but I still can't open it using vignette(). Any ideas?

• include reference to the paper in appropriate sections
• include citation directions to the paper when loading package

I wonder if it is possible to activate clustered standard errors for the sensemakr function, in settings where the treatment is assigned at a group (cluster) level?

Should be an option, right?

• include total r2 bound
• include partial r2 bound without conditioning on d
• include option to group covariates for bounding

Finishing touches: add refs / cites

Include tests to make sure axis of contour plots are correct: r2dz.x in x-axis and r2yz.dx in y-axis.

After plotting done, repeated messages appear in the console:
Error in fetch(key) :
lazy-load database '/home/chad/R/x86_64-pc-linux-gnu-library/3.3/sensemakr/help/sensemakr.rdb' is corrupt

These continue even after dev.off()

• include numerical tests for every sensitivity stats function
• include numerical tests for every sensitivity bias function
• include numerical tests for every sensitivity bound function
• include numerical tests for every plot (plots return data invisibly)

In the vignette
https://github.com/chadhazlett/sensemakr/blob/master/vignettes/sensemakr.md

edit the worstcaseinterpret() matches graphic of worstcaseplot() one parameter is x axis and other parameter is curve type

@chadhazlett

The two different non-symmetric sentence structures is intended right? Along with 'two dimensions', intended to provide users with two grammatical variants?

Result of interpret

## 
## ### Benchmarking ###
## 
## ---Using the covariate most strongly associated with the treatment assignment as a benchmark---
## 
## An unobserved confounder explaining as much of the treatment as 'female' (0.009)  would be able 
to cause at most a bias of 0.062 with an adjusted treatment effect of 0.035 in the extreme case where 
the confouder explains all the residual variance of the outcome (R2y = 1).
## 
## ---Using the covariate most strongly associated with the outcome as a benchmark---
## 
## An unobserved confounder as associated with the outcome as 'female' (R2y = 0.109)  would have to 
be at least 18.7 times as strongly associated with the treatment (reaching R2d = 0.17) in order to reduce 
the treatment effect by 100%

Hi

I how one could conduct sensitivity analysis using sensemakr based on machine learning algorithm for causal inference, in particular the GRF or SuperLearner?

I see that the sensemakr function takes the estimate and SE, but it also requires the degrees of freedom (dF). In ML setting (for non-parametric models), it is unclear what dF is.

Your input is most appriciated

now in cleaning / orgnizing mode

i'm thinking a lot of the low level functions should NOT be exported, while some of the higher level functions should be.

Things NOT to export

getstats
benchmarkr
groupR2
get_bias
get_se
get_t
adjust_estimate
t_to_r2

Things to Export

S3method(plot,sensemakr)
S3method(print,sensemakr)
S3method(print,summary.sensemakr)
S3method(sensemakr,lm)
S3method(summary,sensemakr)
export(class_df_from_term)
export(contourplot)
export(interpret)
export(sensemakr)
export(worstcaseinterpret)
export(worstcaseplot)

are users really going to use these directly? I don't think so. Evenmore so, users will not use these functions OUTSIDE of the sensemakr package thru

sensemakr:::benchmarkr(some_non_sensemakr_application)

On the other hand, as argued in #14
some of the plotting functions might be directly accessed by users

https://cran.r-project.org/web/packages/roxygen2/vignettes/namespace.html

save for the end, add safety bumpers, assertion stopifnot checks for function args

Let's remove the column "covariate" from the output -- it's redundant with the row names, and as a character it prevents easy manipulation of the rest of the matrix.

not a priority, something to think about

Dear developers,

First, thank you for the really nice package and vignette!! It helped me a lot and ease the transfer to other communities.

I have a question regarding plot. Would it be possible to do the plot from the output of the main function and then using ggplot? From the output it is clear to me how to have the x and y axis and the different points, but I am not 100% about how the contour plot should be done?

Why do I want to use ggplot? Just because it seems more flexible for colors, font size, labels, and so on. :)

Thank you in advance for your help!!
Best,
Benedicte

We already have relative comparisons with the covariate mostly associated with the treatment and the covariate mostly associated with the outcome. Include relative comparison as a multiple of the "stronger" covariate, in the sense that the unobserved confounder with those associations would cause the largest bias. Example:

• "An unobserved confounder would need to be at least X times as associated with the treament assignment and with the outcome as 'covariate' to reduce the estimated effect in q%"

• write the documentation for darfur's example data

dear Carlos, great STATA implementation. Any guidance on how to include robust standard errors - vce(robust) and cluster(var) into the sensitivities. Obviously without this ts are off ...
Thank you!

question posed in vignette

https://github.com/chadhazlett/sensemakr/blob/master/vignettes/sensemakr.md#benchmarking-summary

QUESTION: there is a mismatch between these two. Is this what we want?

MT: here?
## ---Using the covariate most strongly associated with the treatment assignment as a benchmark---
## ---Using the covariate most strongly associated with the outcome as a benchmark---

not clear what the mismatch is

Idea: a function such as fullreport(obj, filename) that produces a pdf with all the plot types and the natural language interpretation. We could even add there various warning and caveats such as "whether this is strong evidence for the credibility of your result depnds on whether you feel variables such as [varname] are likely to be more powerful confounders than any unobservable, which depends on ...".

Need argument, perhaps "showvars" to restrict which covariates get shown as benchmarks on plots. Should show up in the top level plot wrapper so that people can see the argument name.
Should check is showvars%in%X and throw an error if not.

QUESTION: is it important that showvars accepts a list? Standard usage would seem to be c(). I guess its because we are overloading it by also making "masked" and "all" options? XXX

We have class(sensemade) at the moment. While cute, for a new user there is an expectation that the class will be the package name, and I think we should adhere to that.

After some time using sensemakr, the following appears in the console:

Error in fetch(key) :
lazy-load database '/home/chad/R/x86_64-pc-linux-gnu-library/3.3/sensemakr/help/sensemakr.rdb' is corrupt

Also, possibly related, if the vignette is loaded, then attempting to access the help file thereafter does not seem to work.

Allow change of R2 scales in plot, instead of axis with the partial R2 of confounder, axis with the total R2 of the confounder.

Hi,

I'm having some trouble saving the plots made by sensemakr.
For example, if I run

plot(sensitivity)

it gives a plot to my output, but neither

pdf("plots.pdf", width=6, height=3)
plot(sensitivity)
dev.off()

nor ggsave() works to save the plot.

I also tried storing the plot as https://stackoverflow.com/questions/29583849/save-a-plot-in-an-object but this doesn't seem to work with your package.

What's the right way to do this?

Thanks,
Will

My model requires me to adjust for cluster standard errors. I use the coeftest function in R to do this. Unfortunately, the object created by the coeftest function is not compatible with the model argument in sensemakr. Thankfully, the estimate, se, and dof can be inputted manually into the sensemakr function.

However, I do not see a way to input the benchmark covariates manually? How can I input the benchmark covariates manually into the sensemakr function?

from proposed questions in rmd

bound" is too strong. XXX

i agree, going to change to lower-limit

Create as shiny app with all functionalities. Ideally, one runs

sensemakrShine(model)

And the UI appears to run all analysis with click and point.

lets save it to the end where we enforce our code style

eg using <- instead of =

Based on an example somebody just sent me, the when the effect estimate has a negative sign, it confuses the worst case plots. Not certain if this remains so in the demo code for the paper, but something to check.

Hello,

I am trying to run a sensitivity analysis using the sensemakr() function, based on the syntax provided in the Darfur vignette. I keep getting the following error: "Error in check_covariates(rownames(coefs), covariates) : Variables not found in model: BG" even though the variable in question is definitely included in the model and exists in the dataframe. Might you have an idea why this is happening?

Thank you for your time, and for this great package!

QUESTION: Are there cases where we'd want to use a covariate as a benchmark but not have it in the model?

Maybe a variable that we worry is post-tx? XXX

We knew this would be an issue, and if we can't solve it right now, okay. But the overlap in the label positions is problematic indeed.