inzightvit / lite

Shiny Application for iNZight.

iNZight Lite

This is the online version of iNZight (http://lite.docker.stat.auckland.ac.nz)

To run locally:

With Docker

1. Ensure that Docker is installed and running
2. Either pull the image from dockerhub, or within the root lite directory, run make build TAG=latest to build the latest image
3. In the root lite directory, run make run TAG=latest to run on localhost at port 3838

Directly

iNZight Lite is a shiny app, so you can run it locally if you have R installed. If you use RStudio, see instructions further down.

1. Download and unzip or clone this repository

   Note: if you use Git to clone the repository, you can easily keep Lite updated via git pull

2. Open R and set the working directory to the Lite folder

setwd("/path/to/Lite")

3. Install the devtools R package (if you don't already have it):

install.packages("devtools")

4. Install package dependencies (NOTE: we host iNZight packages on our own server, see below)

devtools::install_deps(repos = c("http://r.docker.stat.auckland.ac.nz/R",
                                 "https://cloud.r-project.org/"))

5. Run the iNZight Lite app!

shiny::runApp()

RStudio users:

1. If you have git installed: in RStudio > File > New Project > Version Control > Git

   • For the URL, enter: https://github.com/iNZightVIT/Lite.git
   • And choose a location to save the app
   • Then Create Project

   If you don't have git:

   • Download and unzip the app
   • In the folder, double click Lite.rproj to open the project in RStudio

2. To install the dependencies, enter the following into the Console:

install.packages("devtools")  ## skip if you know you already have devtools
devtools::install_deps(repos = c("http://r.docker.stat.auckland.ac.nz/R",
                                 "https://cloud.r-project.org/"))

3. To run iNZight Lite, open the ui.R file and click the Run App button at the top.

Send bug reports/errors to [email protected].

Directories:

• data: This directory holds all the data. The data must be placed in sub-directories, which will come up as data categories within iNZight-Lite.

• www: This directory is used for graphics and/or web-related files such as .css and .gif/.jpeg/.png javascript files. Images are stored in a sub-directory called "images". Javascript code is stored in a subdirectory "js".

• panels: This directory contains the gui, server, help, and other functions for each module.

Main files:

• README: Documentation for iNZight Lite. Update as required.

• global.R: Defines functions and variables that are used globally. For now, all non-GUI functions are stored here.

• server.R: Script for shiny server.

• ui.R: Defines the main GUI.

• functions.R Defines al functions used in iNZight-lite, some of them are also in iNZightTools.

• data: This directory holds all the data. The data must be placed in sub-directories, which will come up as data categories within iNZight Lite.

• www: This directory is used for graphics and/or web-related files such as .css and .gif/.jpeg/.png. Images are stored in a sub-directory called "images".

• panels: This directory contains the gui, server, help, and other functions for each module. The directory/file names will be of the form:

  • [Directory] N_(sub-)panel-name
  • [File] 1_panelname-panel-gui.R
  • [File] 2_panelname-panel-server.R
  • [File] 3_panelname-panel-help.R
  • [File] 4_panelname-panel-null.R
  • [File] 5_panelname-panel-other.R

iNZight-lite URL feature

Some GET parameters can be added to the URL to access iNZight-lite. In the moment the following are implemented.

• url: An URL to a dataset sitting somewhere in the internet.
• example: A name of an data set inside the data folder.
• land: The place where iNZight-lite starts. If not specified it is "About". Three possibilities (visualize, timeSeries, regression).
• x: The x variable in the visualize module
• y: The y variable in the visualize module
• time: The variable in time series where the dates are stored. -seriesVars A comma seperated list of column names. This names are set in time series to be plotted
• Y: The dependend variable in regression.
• predict: Comma seperated list with predictor variables.
• confound: Comma seperated list for confounding variables.

lite's Issues

merge VITonline into Lite

test for a single p capability of binary variables

Anna:

add a feature to iNZight lite that you could test a single proportion. At present, for categorical data, Chi-square tests are the only option, and for one variable, a test for equal proportions. I know the logic would have to involve (1) choosing a level of the categorical variable to focus on and (2) allowing for the null hypothesis to be set, but it seems weird I can do all the other tests but not this one. Would help STATS 100 and STATS 10x :-)

Also, for categorical data/chi square tests, some sort of thing that checks expected counts would be nice...... e.g. single proportion, np, and a warning that using a norm-based test might not be such a great idea when it's < 5 etc.

UI of Model fitting

“Factor Level Comparison” to be the last plot type not the first. “Graphical diagnostics” is the most sensible default with just Residuals vs Fitted appearing by default (“All plots” risks slow loading esp with larger data sets)

Missing packages

The GGalley package requires reshape, which isn't in the repo.

Loading google sheet

Data set to test here: https://lite.docker.stat.auckland.ac.nz/?url=https://docs.google.com/spreadsheets/d/e/2PACX-1vRmivvQYtYiWhpsjHX2o5gq3ctAufja6Ohe9Oz7lSpLm3DMU9zh2Dup2Zl7qSatRQ4PLNoQUel03CqR/pub?gid=257119653&single=true&output=csv&land=visualize

The binary variable causes the problem in the inference panel.

One thing in iNZight not in Lite – multiple response data

Rework Modelling

new 'survey specification'

add the ability of import a survey design from a single file (i.e., data + survey in one) (based on iNZightTools (1.10))

Depends

In order to run 'Lite' application one needs to have required packages
Maybe they can be listed in ireadme.md, e.g.

library(devtools)
install_github("iNZightVIT/iNZightRegression")
install_github("iNZightVIT/iNZightMR")
install_github("iNZightVIT/iNZightPlots")
install_github("iNZightVIT/iNZightTS")

install.packages("gpairs")

Maps problem

When Selecting a variable I got this error message ..

Error: 'st_normalize' is not an exported object from 'namespace:sf'

Display 'last updated' instead of 'version'

Since Lite runs a rolling-release cycle, it makes sense to display the last updated date rather than a version #. It might be worth considering starting a 'NEWS' file to log all changes for both us and users.

add "open in new window" for interactive plots (plotly)

This would create temp files on the server.

htmlwidgets::saveWidget(as_widget(plotly::ggplotly()), "index.html") is really slow (may need to take 10 secs)...

Experimental design

Compute group mean, LSD, TSR

Pulling Lushi’s Network data module into Lite

Tidy up

Some thoughts, in no particular order. I'll get to these with the next summer student in the next few weeks..

• Probably want uniform font + font size + panel width + colour scheme across ALL modules;
• Buttons seems to overlap in "Model Fitting" module;
• Tables - could probably display a few more rows and tidy up column names;
• Reduce footer icon size in "About" page;
• Center logo in the header; maybe even merge CSS with that of iNZight website..?;
• Reduce spacing between paragraphs in "About" page;
• Consider using rcharts for graphics? It's on a browser, so might as well take advantage of it..
• A "Help" page would probably help, since the footer is only displayed for the "About" page.

bug in Visualize with 2 categorical variables

put a variable in each of slots 1 and 2 – works fine (side-by-side bar chart)

Then use the arrow to interchange 2 and 3 – works fine (separate bar charts)

Then use the arrow to interchange 2 and 3 – it should go back to the way it was originally but instead crashes.

Time Series Frequency

In the original iNZight Time Series module, you can specify any integer frequency you wish. For example, if I have weekly data (which I don't, but we can pretend), I can produce a plot as follows:

When I try the same approach in Lite, however, the session crashes when I enter 5 as the frequency. The Javascript output is:

No arguments supplied.
Warning: Unhandled error in observer: arguments imply differing number of rows: 0, 154
observe({
    input$provide_actionButton
    isolate({
        if (!is.null(input$provide_actionButton) && input$provide_actionButton > 
            0) {
            time = NULL
            if (!is.null(input$provide_frequency) && input$provide_frequency %in% 
                "Day") {
                time = seq(as.Date(input$provide_startdate), 
                  by = "day", length = nrow(get.data.set()))
                time = unlist(lapply(strsplit(as.character(time), 
                  "-"), function(x) {
                  paste0(x[1], "D", strftime(as.Date(paste(x, 
                    collapse = "-")), format = "%j"))
                }))
            }
            else if (!is.null(input$provide_frequency) && input$provide_frequency %in% 
                "Month") {
                time = seq(as.Date(input$provide_startdate), 
                  by = "month", length = nrow(get.data.set()))
                t [... truncated]

so it just looks like the function is assuming the user selects frequency from the drop down rather than allowing them to enter one specifically.

Fixing stringAsfactors for R4

Segfault when using survey feature

Using feature/survey branch, following the example from Survey design doc, hit segfault at plot of api99. Does this replicate?

 *** caught segfault ***
address 0x68, cause 'memory not mapped'

Traceback:
 1: grid.newpage()
 2: (function (x, y = NULL, g1 = NULL, g1.level = NULL, g2 = NULL,     g2.level = NULL, varnames = list(), colby = NULL, sizeby = NULL,     symbolby = NULL, extra.vars, locate = NULL, locate.id = NULL,     locate.col = NULL, locate.extreme = NULL, locate.same.level = NULL,     highlight = NULL, data = NULL, design = NULL, freq = NULL,     missing.info = TRUE, xlab = varnames$x, ylab = varnames$y,     new = TRUE, inzpars = inzpar(), layout.only = FALSE, plot = TRUE,     xaxis = TRUE, yaxis = TRUE, xlim = NULL, ylim = NULL, zoombars = NULL,     hide.legend = FALSE, df, env = parent.frame(), ...) {    m <- match.call(expand.dots = FALSE)    if (missing(df)) {        if (!is.null(design)) {            if (any(grepl("as.svrepdesign", design$call[[1]],  
...
174: ..stacktraceoff..(captureStackTraces({    while (!.globals$stopped) {        ..stacktracefloor..(serviceApp())    }}))
175: shiny::runApp()

figure out the code writing from url approach

file.writable is not working for OSX

I do not know if OSX is worth testing
but for OSX in the function file.writable() the condition
"unix"%in%.Platform$OS.type
is TRUE yet OSX stat does not have option -c
and in result one gets
''stat: illegal option -- c''

passing name object

Info from Tom:

if (is.numeric(settings$x)) ... will need to become if (is.numeric(data[[settings$x]])) ....

Alternatively, if in places (for example the Add To Plot panel) you use them a lot, something like this:
https://github.com/iNZightVIT/iNZight/blob/feature/summary-controls/R/iNZPlotModWin.R#L2675-L2676

every occurence of settings$x, settings$y$, ...$g1, ..$g2, $colby, $sizeby, etc

access a variable from the data frame, you need to convert the name to a character,

xvar <- data[[as.character(settings$x)]]

up/down arrows don't work when selecting plot types

when changing plot types, the selectInput would update so that up/down arrows do not work. The plot type also does not change when the data are changed. Need to be fixed.

does not read tab separated text files properly

Bug in footer

When the screen size shrinks (i.e., on a tablet), the logos get cut off.
My guess is that it's using some fixed-position and fixed-height CSS that isn't being altered by media queries. Possibly can blame bootstrap? :)

Improvement the survey design stuff

may need to rewrite the whole panel

Code writing

make the code writing stuff work for

• Example dataset
• import dataset
• gg plot
• Manipulate variables
  • Convert To Categorical
  • Categorical Variables
  • Numeric Variables
  • Date and Times
  • Rename variables
  • Create variables
  • Missing to category
  • Delete variables
• Dataset
  • Filter Dataset
  • Sort data by variables
  • Aggregate data
  • Stack variables
  • Reshape Data
  • Unite columns
  • Separate columns
  • Merge/join datasets
  • Restore data
  • Survey design

Import data by copy and paste

May add a textInput in Import data panel...

expand table

the relevant code in iNZight:
https://github.com/iNZightVIT/iNZight/blob/3ac68c6138b52e69fa53e604b53a87dfb4dd2b3e/R/iNZChangeDataWin.R#L1008

Catch errors on data import (and maybe check extensions)

Since it's a combination of macOS being stupid, and students blindly following instructions, it might be worth

1. wrapping import in tryCatch (smart_read should return a warning and NULL if the file extensions is not supported), see https://github.com/iNZightVIT/iNZightTools/blob/dev/R/import_data.R#L89

2. checking for commonly bad extensions (e.g., .numbers) and throwing a warning.

Lite/panels/B1_ImportDataset/2_import.data.set.panel-server.R

Lines 22 to 36 in 533b22d

	fpath = input$files[1, "datapath"]
	fext = tools::file_ext(fpath)
	if(fext %in% c("RData", "rda")){
	temp = as.data.frame(stringsAsFactors = TRUE,
	iNZightTools::load_rda(input$files[1, "datapath"])[[1]])
	} else {
	if (fext == "txt") {
	temp = as.data.frame(stringsAsFactors = TRUE,
	iNZightTools::smart_read(input$files[1, "datapath"],
	delimiter = "\t"))
	} else {
	temp = as.data.frame(stringsAsFactors = TRUE,
	iNZightTools::smart_read(input$files[1, "datapath"]))
	}
	}

new features

• "Deff" comes from iNZightPlots, so all you need to do is update it once it's ready (I'll be releasing the new version on Monday, along with updates to iNZightTools and iNZightMaps).

• I've added a "Remove intercept" checkbox to the model fitting interface: iNZightVIT/iNZightModules@387a54f - this basically just add " - 1" to the model (i.e., lm(y ~ x - 1))

• unsure if it affects you, but check that you are passing NULL to scale and rscale (in svydesign()) when the replicate survey type is neither "bootstrap" nor "other". iNZightVIT/iNZight@2f5e5bd
  The file you can use to test: https://github.com/iNZightVIT/iNZight/blob/dev/tests/testthat/apiclus2-jk1.csv - this uses JK1.

Time series with values zero or less log zero or negative problem

grey out the radio input when got zero's

Class checking

No idea if this is just because of Christoph's Python background, or if it's a quirk of Shiny, but I noticed this type of thing a lot: class(x) %in% "numeric"
e.g., https://github.com/iNZightVIT/Lite/blob/dev/panels/C1_Visualize/2_visualize-panel-server.R#L176

%in% is 15x slower than is.numeric(x) and the equivalents for factor/character etc, and about 2x slower than class(x) == "numeric".

Anyway not a bug, just a curiosity - it would seem that every optimisation we can think of should be implemented in Lite to reduce strain on the server, but I am aware that Shiny has some nifty quirks that make obvious code unhelpful.

Clean up old files and update readme

There's some pretty ancient stuff in here! Some (e.g., DOCKER.md) are from the original deployment on a single hacked-together box. There are also some dot files which should be replaced with a .gitignore (.Rapp.history, .Rhistory, .DS_Store ...).

the decomposition plot for TS

• iNZightTS - the decomposition plot has been changed, so it might break the TS module

The main change is:

decompositionplot(ts)

becomes

plot(decompose(ts))

Here's the diff for the desktop implementation:
iNZightVIT/iNZightModules@098a8e5#diff-efd9af239a53df23ccf0166a0ce47511L922-L929

Busy Indicator

Do we need a busy indicator? When clicking on the plotly stuff, it will take a few secs to open... So we may need to add a busy indicator.

But also need to ensure this works well for all other panels.

cannot import the .shp file (maps module)

class intervals for a numeric variable

save plot as pdf

the first page is blank

GUI of Inference Panel

• When bootstrap is clicked currently none of the other Inference buttons are relevant

• (Lite – numeric vs binary ) When “none” is clicked shouldn’t get a test result

                            When “none” or  “anova” is clicked shouldn’t see Null and Alternative hypotheses etc
  

• (Lite – binary) When “none” is clicked shouldn’t get a test result, shouldn’t see Null and Alternative hypotheses etc

                            When “Test proportion” is clicked shouldn’t see Simulate p-values
  
                         Prefer  “Test proportion” above Chi-squ Test as more basic
  

• (Lite – categorical with more than 2 categories & cat vs cat)

• (Lite – numeric vs numeric)

cursor position in textInput

this would be useful when the user types their own model.
cannot be done directly in shiny, so may need some javascript.

URL Import

Can't seem to read in datasets via URLs...

Extend Imports Interface to Accept Further Specifications for Complex Filetypes

Match iNZight capacity for further specifications in import such as a dialog for choosing specific sheets from an xlsx file; or specific dataframe from an rda file

Is `gpairs` needed???

I thought Christoph used GGally instead of gpairs but maybe I'm wrong?

Problem with tying to read in a new file when we have a url as query string

Check functions working

This file contains some functions which were duplicated in iNZightTools - however, they are being deprecated (fully deleted, actually) which will take effect in the next release of iNZightTools.

So, could you check that the functions marked "to be deleted once iNZightTools is working" are using the versions specified in functions.R, and not the functions in iNZightTools.

Several of these functions have been rewritten using tidyverse (e.g., combine.levels() is now collapseLevels()), so at some stage, we will need to change these over (but there's no rush yet) if/when Lite starts storing code history (as these tidyverse versions include the code).

One way to check things are working would be to---if you can run Lite locally---install the development version of iNZightTools:

devtools::install_github("iNZightVIT/iNZightTools@bugfix/deprecate-functions")

and then check that the following work:

• combine factor levels
• form class intervals
• sample data
• search name(?) - honestly I don't know what this is used for 😕

Added features

The new feature has been added to colour-by a variable, and allow user to specify the colours.

Still being improved, but essentially specify the argument col.pt = c('list', 'of', 'colours') (one for each level of the factor).

Will in future be extended to let user select a colour scheme.

survey design in Model fitting

example for testing https://homepages.ecs.vuw.ac.nz/~rarnold/STAT392/iNZightManual/_book/regression-estimation.html#example-13

the data set information (and survey design) may not be getting properly cleared out

(1) opened a clean session, loaded apiclus2, specified its survey design, and could do inference on api00
(2) I then loaded apistrat, specified its survey design, and could also do inference on api00
(3) I then reloaded apiclus2, and when I tried specifying its design I got the error:
There is a problem with the specification of the survey design:

Error in popsize < sampsize : non-conformable arrays

If I delete apiclus2, and then go back to the sample design - the variables I had chosen are still there, rather than having empty entries. So it does look like the survey design is somehow hanging around in the background.

Survey spec panel - show pop size

A recent addition to iNZight was to display the estimated population size when creating a survey (manually, the old way). I need to figure out exactly the correct way to do it if only fpc is specified, but in most cases surveys use weights, which can simply be summed to get the (approx) population size:

wtvar <- user_select_box()
estpopsize <- sum(data[[wtvar]])

I've had a few people asking if this feature is on Lite, so it would be good to integrate (see below)!