rstudio / shinytest

Automated testing for shiny apps

shinytest

NOTE: shinytest is deprecated and may not work with Shiny after version 1.8.1, which was released on 2024-04-02. This is because it is based on a headless browser, PhantomJS, which was last released on 2016-01-24 and is no longer being developed. Going forward, please use shinytest2, which makes use of headless Chromium-based browsers. See the shinytest to shinytest2 Migration Guide for more information.

shinytest provides a simulation of a Shiny app that you can control in order to automate testing. shinytest uses a snapshot-based testing strategy: the first time it runs a set of tests for an application, it performs some scripted interactions with the app and takes one or more snapshots of the application’s state. Subsequent runs perform the same scripted interactions then compare the results; you'll get an error if they're different.

Installation

To install the current release version:

install.packages("shinytest")

Usage

See the getting started guide to learn how to use shinytest.

shinytest's Issues

Error whenever using a DT datatable

Using the following app:

library(shiny)

ui <- fluidPage(
  numericInput("num", "number", 5),
  DT::dataTableOutput("table")
)

server <- function(input, output, session) {
  output$table <- DT::renderDataTable({
    head(iris, input$num)
  })
}

shinyApp(ui = ui, server = server)

When I run recordTest(), change the input value to 6, and exit, I get this test script:

app <- ShinyDriver$new("..")
app$snapshotInit("mytests")

app$setInputs(table_rows_current = c(1, 2, 3, 4, 5))
app$setInputs(table_rows_all = c(1, 2, 3, 4, 5))
app$setInputs(num = 6)
app$setInputs(table_rows_current = character(0))
app$setInputs(table_rows_all = character(0))
app$setInputs(table_rows_current = c(1, 2, 3, 4, 5, 6))
app$setInputs(table_rows_all = c(1, 2, 3, 4, 5, 6))

app$snapshotCompare()

And the following error in the console:

Saved test code to C:\Users\Dean\Documents\R\test/tests/mytests.R
Saving baseline...
Error in session_makeRequest(self, private, endpoint, data, params, headers) : 
  Unable to find input binding for element

Create separate classes for widget types

I.e. textInput, selectInput, etc. It will be much simpler to deal with them.

Some nice API for tabs

E.g.

• a function to change tabs
• change tabs for set_value (?)
• change tabs for expect_update (?)

Numeric input doesn't recover from receiving bad input

Using this shiny app:

library(shiny)
ui <- fluidPage(
  numericInput('num', "num", "5")
)

server <- function(input, output, session) {
  observe(cat(input$num))
}

shinyApp(ui = ui, server = server)

If I set the value of num to a bad value (not a number), and then set it to a valid number, the input still returns an NA value.

shinyapp$new(path = ".")$find_widget("num")$set_value("bogus")$set_value("8")$get_value()

If you take a screenshot, you'll see that the actual text inside the input is "bogus8", which means that the second "set_value" didn't actually replace the previous value

It's very possible that other inputs have similar issues, this is the first input I tried breaking

Think about set_value() methods

Ideally they should just emulate the user.

slider can have dates

Install phantom on Appveyor & Travis

But first I want to fix webdriver on Travis: rstudio/webdriver#22

Shall we inject JS code from shiny itself?

E.g. add a tags$head() call somewhere in the UI. I guess for some fancier apps this might not be possible. Or, if the UI is created when a package is installed, then there is no way to inject code there. But for these cases we could provide a workaround.

Support DT tables

And I guess most HTML widgets need special treatment.

Think about how to use shinytest for apps w and w/o packages

Expose the wait_for API from webdriver

For doing advanced stuff, like shinyjs.

Idea: list all inputs and outputs that are available in the app

I can't decide if this is a good idea or if it's useless. I just started running shinytest on an app and I was trying to play with it in the console, but I forgot what IDs I need to use. Instead of having to look back at the source code, I wanted to type something like app$list_inputs()

Thoughts?

Provide reproducible example in documentation

+1 for adding man pages

In the Examples section of ?shinyapp, instead of saying shinyapp$new("050-kmeans-example") it'd be nice to give a path to the code of that app

Idea: ability to browse into the app at its current state

This is something that wouldn't have been possible a month ago, but with shiny's new state features, I think this could is achievable.

When I have an error in my app, I want to dig deeper and see what's happening, and so far I've noticed I often use the take_screenshot() function as a debugging tool. But an even better option would be to be able to open up the app in its current state in a browser so I can see what is really going on.

This could be a bit weird to implement, especially because the state functionality of shiny currently requires you to declare ahead of time that you want to enable state, and it might interfere with the state saving functionality of a shiny app that already supports it. It's definitely prone to a lot of weirdnesses. But could be a nice idea to think about.

Think about tables

Remotes for processx, debugme, etc. should be in DESCRIPTION

to allow for automatic installation by devtools

shiny::runApp() does not have an argument "test.mode"

When running the following:

library(shinytest)

app <- shinyapp$new("shiny-examples/050-kmeans-example")

I get the following error:

Error in app_start_shiny(self, private, path) : 
  Cannot find shiny port number. Error: Error in shiny::runApp("shiny-examples/050-kmeans-example", test.mode = TRUE) :unused argument (test.mode = TRUE)Execution halted

I belive this is caused by line 4 of the function shinytest:::app_start_shiny() :

runApp('%s', test.mode=TRUE)"), libpath, path)

which refers to a test mode. In the current version of the shiny function runApp() there is no argument test.mode. Unsure whether removing the corresponding argument from shinytest:::app_start_shiny()

runApp('%s', test.mode=TRUE)"), libpath, path) -> runApp('%s'), libpath, path)

since it might remove some testing nature that shinytest depends on. Is it possible the equivalent functionality has been replaced with options(shiny.testmode = TRUE)?

Cheers!

Running shinyapp$new() gives "Error: Argument 'txt' must be a JSON string, URL or file."

When on the testmode-inject-js branch of shiny, running the following code

library(shinytest)
app <- shinyapp$new("shiny-examples/050-kmeans-example")

yields the warning

No encoding supplied: defaulting to UTF-8.

which I can probably just ignore as presumably a httr function just isn't being given a default somewhere. I also get the error:

Error: Argument 'txt' must be a JSON string, URL or file.

and I'm not sure where this originates from.

Cheers, Dan

Think about "sharing" a phantomjs process for multiple apps

So that we don't need to start a new one in each test_that block.

But I am not sure if it is worth it....

Testing slow on windows

See rstudio/webdriver#27

Better logging

Do not hide output from background processes. More precisely, have some API to access them, show them on error, etc.

Some of this might go to webdriver.

Make starting the app faster

I guess we could cache spare R processes somehow....

Eliminate curl error message when processx finalizers kick in

Not sure why it is happening

destructor on GC

shinytest doesn't work unless attached to search path

Here's what I get if I try to record a test without library(shinytest) first:

> shinytest::recordTest()
 Error in eval(expr, envir, enclos) : object 'ShinyDriver' not found 

Use more precise relative paths in messages

I get this on initial testApp() run:

  No existing snapshots at mytests-expected/. This is a first run of tests.
  To save current results as expected results, run:
    snapshotUpdate("mytests", ".")

Shouldn't it be "No existing snapshots at tests/mytests-expected/."?

shinyapp$new should unset R_TESTS

You practically cannot run another R process during R CMD check without doing this....

More complex updates

The idea is the following: you can also specify the expected output of a widget. Sg. like

app$expect_update_to(xcol = "blah", clusters = 4, output = list(plot1 = corrent_plot))

This will basically perform the update sequence, and then wait until the output has the desired value, or a timeout kicks in.

The tricky thing is to guess when shiny is done. My initial idea:

1. listen to update events (shiny::value) for the output widget (plot1 above) and then
2. wait until Shiny becomes idle. When it does, compare the output to the desired one.
3. If they match, success.
4. If they don't wait a bit more (until the timeout?) and compare again.

API for plot interaction

Make it possible to click on plots, select areas, etc. Basically the stuff in this app: https://gallery.shinyapps.io/095-plot-interaction-advanced/

Use r-hub as CI

Probably easier than fixing Travis. #35 #19

Make it work with older version of testthat

For ValidR, that has an older version.

Is element visible

See https://w3c.github.io/webdriver/webdriver-spec.html#list-of-endpoints
https://w3c.github.io/webdriver/webdriver-spec.html#element-displayedness

Add README/vignette to make it extremely painless to start testing the package

app$set_value method

Like get_value, a shortcut.

Why does devtools::install() not work?

But devtools::build() and R CMD INSTALL does????

Use Shiny's JS code to get widget values

From #14 (comment)

Feature request: initialize a shinyapp using a "shiny.appobj" object

When I test quick small shiny apps, I don't like to have to save a new file and then delete it, I prefer to just run some code in the console. It'd be convenient to be able to do

library(shiny)
library(shinytest)

ui <- fluidPage(
  numericInput('num', "num", "5")
)

server <- function(input, output, session) {
  observe(cat(input$num))
}

app <- shinyapp$new(app = shinyApp(ui = ui, server = server))

Should check for duplicate IDs and give some sort of error

It would be nice to check for duplicate IDs, though I'm not sure when the right time would be to do the check, or how/when to signal an error.

Finding duplicates is simple. This is some code from: http://stackoverflow.com/a/509965/412655. I've added some code for timing as well.

t0 = new Date();
$('[id]').each(function(){
  var ids = $('[id="'+this.id+'"]');
  if(ids.length>1 && ids[0]==this)
    console.warn('Multiple IDs #'+this.id);
});
t1 = new Date();
console.log(t1-t0);

Running on the widget gallery app takes between 1 and 3 ms on my computer (in Chrome), so it's pretty fast: https://gallery.shinyapps.io/081-widgets-gallery/

It's possible that this should just be in Shiny, though.

Any way to get the tests working on travis/CRAN?

CRAN is maybe not super important, but since the entire focus of this package is testing, it'd be great if I could add tests for my shiny apps that could run on travis instead of relying on myself to manually run them.

I suppose in order for that to work, the build would need to install phantomjs and set it in the PATH. If that can be done (and it should be doable? But I've never dealt with building custom software on R package builds so I don't know how) then the tests should be able to run smoothly I'd imagine.

But maybe it's way harder than that.

App that works can't be tested: Unable to find input binding for element...

My app "pfitmap" in https://github.com/erikrikarddaniel/pfitmap-shiny basically works the way I expect, but shinytest (1.0.2, shiny 1.0.0 or 1.0.0.9000, Rstudio 1.0.136) fails to save snapshots. (Note that the app needs an environment variable PFITMAP_DATA set to ../../data/test/classified_proteins.10000.tsv.) Testing the template app "Old Faithful Geyser Data" works for me.

After "Saving baseline..." I get:

Error in session_makeRequest(self, private, endpoint, data, params, headers) :
Unable to find input binding for element

In about half of the cases the element in the function is undefined (I added code to the javascript to print out the "el" variable to find out), in the other cases it's an HTMLDivElement (not printable with JSON.stringify; circular). I don't do any interaction with the app, just start it, press "Take snapshot" and then exit.

In the first case ("el" being undefined), the test script is very short:

app <- ShinyDriver$new("..")
app$snapshotInit("t0")

app$setInputs(mainmatrix_rows_current = c(1, 2, 3, 4))
app$setInputs(mainmatrix_rows_all = c(1, 2, 3, 4))
app$snapshot()

app$snapshotCompare()

In the second ("el" being an HTMLDivElement), more things are set (seemingly a more reasonable case, since I set defaults in my code):

app <- ShinyDriver$new("..")
app$snapshotInit("t1")

app$setInputs(pclasses = character(0))
app$setInputs(tfamilies = character(0))
app$setInputs(torders = character(0))
app$setInputs(pfamilies = character(0))
app$setInputs(tphyla = character(0))
app$setInputs(tgenera = character(0))
app$setInputs(mainmatrix_rows_selected = character(0))
app$setInputs(mainmatrix_rows_current = character(0))
app$setInputs(mainmatrix_rows_all = character(0))
app$setInputs(mainmatrix_state = character(0))
app$setInputs(mainmatrix_search = "")
app$setInputs(mainmatrix_cell_clicked = character(0))
app$setInputs(tclasses = character(0))
app$setInputs(tspecies = character(0))
app$setInputs(mainmatrix_rows_current = c(1, 2, 3, 4))
app$setInputs(mainmatrix_rows_all = c(1, 2, 3, 4))
app$snapshot()

app$snapshotCompare()

I also get warnings for using the same name in inputs and outputs. Changing input names gets rid of the warnings, but doesn't fix the error in writing the snapshot.

I use the DT package for my table, referred to in issue #31.

Idea: throw a warning if the shiny app has multiple elements sharing an ID

As mentioned in #14

This could be out of scope for a testing package, but could be useful too.

A few more function ideas

I played with shinytest for a bit and it looks very promising, works well so far with the basic stuff I tried testing. Just a few more function ideas that I think could be useful:

• app$update_value(id, value)
• app$click(id/selector)
• app$isElementVisible(element)

I also wasn't 100% sure what send_keys does, does it send keyboard keypresses? If that's the case, it could be clarified. Would love to play around with this package some more later

Make sure html widgets are handled in a reasonable way

Make sure Shiny modules work

Very slow (or maybe this is expected?) on Windows

A simple app like this one takes 5 seconds on Win7 to start up with recordTest():

library(shiny)

ui <- fluidPage(
  numericInput("num", "number", 5),
  plotOutput("plot")
)

server <- function(input, output, session) {
  output$plot <- renderPlot({
    plot(seq(input$num))
  })
}

shinyApp(ui = ui, server = server)

After changing the value to 6, taking a snapshot, and exiting, this is the script:

app <- ShinyDriver$new("..")
app$snapshotInit("mytests")

app$setInputs(num = 6)
app$snapshot()

app$snapshotCompare()

And it takes 10 seconds to complete the saving/running of the test. Maybe it takes this long on Mac as well and it's expected, but I feel like if it takes so long every time it will be a huge deterrent to using this tool

shinyapp class should be renamed

The name shinyapp is pretty close to shinyApp, which could lead to confusion. Maybe it could be shinytestApp?

Make sure dynamic UI is handled properly

Maybe we don't need any extra API for it, but nevertheless there should be an easy way to test apps with dynamic UI.

set_value() should emulate the browser more closely

E.g. for checkboxes, we want click()s or toggle()s, not a JS update.

Running in a build/CI process

I'm guessing shinytest will be heavily used in build scripts and CI. Those workflows usually rely on the test tool to exit with a nonzero status to signal test failure. Perhaps testApp should have an option, or make another function, that does the q(save="no", status=1) call for you.

And in any case, testApp needs to return a programmatic representation of passing/failure (or better yet, of the difference information that resulted from the tests).

tabset and tab messup

Details coming soon, this is just a reminder.