declaredesign / ddwizard

DeclareDesign Wizard -- Shiny app for creating, editing and inspecting research designs.

DeclareDesign Wizard

A Shiny app for creating, editing and diagnosing research designs.

Authors:

• Clara Bicalho [email protected]
• Sisi Huang [email protected]
• Markus Konrad [email protected]

Dependencies

Installing dependencies using renv

DDWizard employs renv and we recommend using it, when you want to run DDWizard on your computer. If you haven't installed renv yet, please do so by running install.packages('renv'). After that, open the DDWizard RStudio project and run:

renv::restore()

This will install all required packages in an isolated package environment for DDWizard, i.e. it will not mess with your existing R package versions. Every time you open the DDWizard RStudio project, this package environment will be used.

List of dependencies

All dependencies are listed in the file renv.lock.

Overview of files

This shiny app displays several tabs, each of which is a separate shiny module (http://shiny.rstudio.com/articles/modules.html) implemented in the respective "tab_ ... .R" file. Hence app.R only implements the "skeleton" of the application.

• app.R: Shiny application skeleton
• tab_design.R: "Design" tab for loading and manipulating existing designs
• tab_inspect.R: "Diagnose" tab for visual design inspection
• common.R: Common utility functions
• inspect_helpers.R: Utility functions for design inspection
• uihelpers.R: UI related utility functions
• conf.R: Configuration options

Documentation

The _docs folder contains the following documentation files that provide an overview:

• ddnotes.Rmd: introduces designers from the DesignLibrary package from a DDWizard development perspective; read this first
• app_structure.Rmd: gives and overview about the source code structure for DDWizard and some details that are quite specific for this app such as use of namespaces, use of Shiny UI extension packages as well as some notes on reactivity and running diagnoses

Tests

Unit tests

Unit tests are implemented in the tests folder with RUnit.

A shortcut to run all tests was added to the Makefile so you can run in the console:

make run_tests

Functional tests with shinytest

We tried to setup shinytest for DDWizard to record app states and then replay them to check automatically if (and where) the app doesn't provide the expected output if we changed the source or updated one of its dependency packages. Unfortunately, shinytest doesn't work well with our app and will fail with an already reported error for all but the very simplest test, which is available in tests/startup.R.

Once the shinytest package works with our package, we should generate more test cases like this:

library(shinytest)
recordTest(seed = 1234)    # setting a seed is important since simulations need to be the same

And test them via:

library(shinytest)
testApp()                  # optionally provide `testnames = ...`

ddwizard's Issues

Include "tips" for design parameters in design tab

Include help text when hovering over parameters in the design tab. This will require changes in DesignLibrary too.

Changing designer after using inspector produces error in plot panel

Probably no proper reset in inspector.

Error: Aesthetics must be either length 1 or the same as the data (1): ymin, ymax, x, y

[inspect] Display and allow to download table of diagnosands

In center pane

Only enable fix buttons if designer contains `fixed` parameter

There's a good chance we will have to revert some (or all) the designers in the library to functions without 'fixed' argument, so fix UI functionality should be flexible.

[inspect] Display and allow to download simulated data

In center pane.

Question: In "Inspect" or "Design" tab?

[inspect] Add option to force re-running simulations

Atm if you have the same combination of designer, parameter space and simulation conf., the simulated data (and the results) will always be taken from cache.

Add this option somewhere near the other simulation options (see #2)

Inspector fixed N issue

Changing 'N' argument to 'fixed' in design tab removes N input box from inspector tab but does not overwrite the default N input string sequence (i.e., diagnosands data frame still shows diagnoses for varying N values).

Properly reset inspector when designers are changed

At the moment, if a user changes a designer, the prev. designers' design's inspection results are still there and, even more confusing, are used for an ad-hoc diagnosis.

Import designers from library

Currently the designer input is done manually. Could we instead read functions in the library, and take input from designers that have definitions attributes? It would require string manipulation from designer name for labels, but the neat thing is that the app could be updated with more designers as we update the library.

"Import" button should only be enabled once the list of designers is loaded

Otherwise if you're quick you can generate an error.

[inspect] Ability to change num. of simulations and bootstraps

Two inputs in the "inspect" tab, probably in the center pane above "Update plot"?

Make action button in inspector more explicit

Consider changing from "Update plot" to "Rerun diagnosis" or "Rerun simulations." Currently unclear because plot gets updated automatically when only plot parameters are changed.

tailored diagnosands

The process tracing design doesn't seem to produce a diagnosis in the inspect tab

Related perhaps to the issue that the code output does not include the diagnosands. Though think it will need to. See the set_diagnosands lines at the end of DesignLibrary::process_tracing_designer

Note using the library diagnose_design(process_tracing_designer(), sims = 3) produces a design with diagnosands as an attribute and this yields a proper diagnosis

[inspect] Plot updates automatically although it shouldn't

I implemented the plot output so that it only updates whenever you press
the "Update plot" button. For reasons I don't understand, Shiny
nevertheless re-renders the plot whenever you change anything in the plot
configuration on the right, but without re-running the simulations. So this
can lead to strange outputs. Only when you press "Update plot" again, the
correct plot is rendered.

Integrate "Design Sandbox" (only UI at first)

Make a UI similar to sandbox in https://github.com/DeclareDesign/wizard_shiny/tree/cb/sandbox (but a little more flexible) to allow to create designs by chaining several steps.

Only UI at first, real functionality will be added later (separate issue).

Make compatible with recent DD/DL version

There were several changes to DD that make it incompatible (i.e. produce errors) with the wizard.

top tabs for wizard

• would be nice if these were frozen so that you an always change tab at the top even if you are low down inside a tab

thoughts on top level names:

• think there are really two meta tabs: "Design declaration" "Design diagnosis"
  within design declaration there are two subtabs -- build design from library, build design from steps, and maybe later a third "load design" (eg from a registry)
  one of these subtabs should be "live" and that is the one that the diagnosis tab works with (that the inspector inspects)

Allow for the character input in inspect_tab

refer #26 not only in the design tab, also need to work in the inspect_tab.

Inspector when "fix all"

Currently inspector tab does not show parameter options when users choose "fix all" in design tab.
Following @macartan's suggestions, can we either (a) show diagnosis table of single fixed design (would require allowing for action button to generate a diagnosis table (and omit the plot) or (b) print a message that all parameters are fixed.

I personally prefer doing both (a) and (b). The inspector should allow for a diagnosis of a single design, while telling users that is the case.

Diagnosand table `Error: undefined columns selected`

Some designers (e.g., randomized response) return Error: undefined columns selected in the diagnosands table panel. I think because it calls "term" column that some diagnosands tables don't have. Maybe use union() when indexing column names?

[inspect] Do not show fixed parameters in plot conf

• if N is set to fixed, do not allow to select it as argument used for plotting

Stop process

Would be nice if we could cancel the diagnosis process somehow, instead of having to reload the app. When we x on the progress bar, the modal closes but the calculation is still going on in the background it seems. Any suggestions?

Modularize code

The code for this app should be modularized as explained in http://shiny.rstudio.com/articles/modules.html. Otherwise the source won't be maintainable any more when the code base grows. I regard this a prerequisite for #20. This will produce mature code changes, so I will prioritize this.

Construct inspect UI quicker

After the fix for #51 is merged, the "Design" tab will react much quicker.
Can we also make the "Inspect" tab quicker to show up?

The thing that makes the "Inspect" tab load slowly starts in tab_inspect.R:476:

        # create the design instance and get its estimates
        d <- design_tab_proxy$design_instance()
        d_estimates <- draw_estimates(d)

Is there a way to speed this up, i.e. maybe put the required information on the estimators and coefficients into the designer's attributes? @clabima

[inspect] Ability to configure plot for showing SEs / conf. intervals

Option:

• std. errors (via geom_pointrange)
• conf. intervals (via geom_ribbon)
• none

Option to fix all arguments

Could we add a button that selects all arguments at once for the fixed parameter?

[inspect] Make it possible to change diagnosand parameter

Ability to set alpha for DeclareDesign:::default_diagnosands.

This is only relevant to diagnosands that use this parameter (power), so display the input next to the diagnosand selection only when such a diagnosand is selected.

Recalculate diagnosis results without re-running simulations!

Fix trailing `fixed` tick boxes

There are extra fixed tick boxes trailing at the end of the sidebar for some of the designer options (see factorial for example).

Error when loading randomized response designer

Happens in "Inspect" tab.

This is probably a DesignLibrary issue @clabima. Should I post such things to the DesignLibrary repository in the future?

Warning: Error in <Anonymous>: The following arguments in `fixed` do not match a designer argument:design_name
  126: stop
  125: <Anonymous> [.../R/randomized_response_designer.R#40]
  113: <reactive:design_instance> [tab_design.R#156]
   97: design_tab_proxy$design_instance
   96: renderUI [tab_inspect.R#419]
   95: func
   82: origRenderFunc
   81: output$tab_inspect-plot_conf
    1: shiny::runApp

Include rnorm() and runif() options to sandbox fabricatr functionality

Allow for character and vector input

Need to incorporate somehow arguments of the form vector of vectors or strings. For example outcome_sds in factorial_design and conditions in multi-arm design.

Update "fix all" option

@macartan comments:

• the fix all button seems to do two things--- toggles all on or all off. So eg say I fix all, and then unfix some, and then hit all fixed again it unfixes them all rather than fixing them all. Maybe we need "fix all" and "unifx all"

In terms of position, I think it would be nice if the function generating the design parameters card could be broken down into two to allow for the "fix all" button to sit atop the parameters options, just below parameter name, or just above. @zerinahuang @internaut do you have better suggestions?

[inspect] Progress indicator

During "Update plot".

See http://shiny.rstudio.com/articles/progress.html

Remove NA diagnosand column from inspector options

Remove from the diagnosand data.frame used for the plot configurations UI the option to display a diagnosand when data is missing.

[inspect] Download plot

Underneath the inspection plot, add an option to download it.

[design tab] Immediately hide "Set design parameters" panel

Currently, it is shown quickly at the beginning although no design is loaded yet, because conditionalPanel is evaluated slowly.

Make the wizard compatible to all designers in DesignLibrary

• allow to choose designer from DesignLibrary (dropdown on top left?)
• check for each designer if it works
  • they all must have proper attributes (argument definitions)

design names

it looks like the current default name ofr a design is xx_designer but it should be xx_design I think;
designers make the designs

also these seem not to be changing when the name changes

[inspect] Add instruction and/or example input for getting started in the "inspect" tab

Also, disable the "Update plot" button at the beginning?

Importing designs (for discussion)

For the future:

To connect with contributed designs (not deigners) in the design library or with designs registered eg at EGAP, suggest:

1. request contributors to provide a list of arguments that can be called by redesign; contributors responsible for declaring what these are and add as a design attribute

2. the declare_design tab of wizard has an upload design which can take a url as an argument and bring in a design from a url (either from text or a design orbject or both)

3. The uploaded design's code and sample data etc can be read as currently in the declaration; in the inspector (diagnosis) tab the left arguments correspond to the redesign arguments; everything else works the same

[inspect] Always use a fixed argument's value for inspection

... and don't use a varying argument value that might have been specified before.

Ability to upload, edit and inspect a design

Example use case: Design was created with the wizard and downloaded as RDS file. Now this file can be uploaded (by someone else) into the wizard. The wizard should then be able to work with the design as if it was generated from a DesignLibrary's designer.

This should only be added after #11 is working.

SECURITY ISSUES!

• theoretically allows to execute any R code

Enable caching of calculated diagnoses

currently only caching of simulated data is implemented; see run_diagnoses() in common.R

[inspect] (Not) setting default values when switching back and forth design and inspect tab

Atm, if you set N = 10 in "design" tab, switch to "inspect" then this is the default value there which is fine. Now you set N = 10, 20, ..., 100 in "inspect" which is also fine. If you now go back to "design" and change N to say 15 and then switch back to "inspect", this value will override your previous sequence "N = 10, 20, ..., 100" there and will be set to N = 15.

process tracing in wizrd / vector arguments

I cannot see how to change this argument:

process_proportions <- c(X_causes_Y = 0.25, Y_regardless = 0.25, X_causes_not_Y = 0.25, not_Y_regardless = 0.25)

[inspect] Plot: Indicate variable name in legend for color scale

Plot: If you set a secondary parameter (color), the legend should indicate the name of the variable.

design_name argument not reacting in code panel for Design Library option

Plot can't be shown under the two_by_two designer

value in the vector input is too long

e.g. In the binary_iv designer, the arg type_probs = c(0.3333333, 0.3333333, 0.333333, 0), which is too long to be shown in the design tab. in my opinion, to round the value in the vector, or do you have any suggestions? @clabima @internaut

download diagnosis

diagnosis table downloads ok I think but it does not have a file extension -- should this be .csv?