gadenbuie / mctestanalysis

📚 Apps and reports for multiple-choice test analysis in R with Shiny

MC Test Analysis

This R package provides a nice set of interfaces via Shiny for the analysis and review of multiple choice tests. View full package documentation at https://gadenbuie.github.io/mctestanalysis/.

Try Online

You can try out the Shiny web interface online at https://gadenbuie.shinyapps.io/mctestanalysis/, which demonstrates the test results explorer interface but cannot produce the PDF or HTML reports. For the complete features, you can install the package and use the interface locally (without have to upload your data) by following the instructions below.

Table of contents

• Required Software
  • R
  • RStudio
  • LaTeX
• Installation
• Usage
• Data Format

Required Software

You need to download and install the latest version of the following software. If you already have recent versions of R, RStudio and LaTeX (optional) installed, you can skip ahead to the section on installing the MCTestAnalysis library.

R

Using the MCTestAnalysis package does not require any R programming. If you are interested in learning to program in R, RStudio have collated some excellent resources for Learning R Programming on their webpage.

To install:

• http://cran.r-project.org

• Click the "Download R for ..." link for your OS (Linux, Mac, Windows)

• Click the link for the base distribution. This package was built and tested on R 3.3.2, although both this package and R may have been updated.

RStudio

RStudio is an IDE (an application that supports the development and use of a language) for R. The MCTestAnalysis package is best used from within RStudio, but the user only needs to be able to find the Console window to run the few commands described below.

For a quick introduction and reference for the RStudio IDE, RStudio publishes an up-to-date RStudio cheatsheet. The RStudio Essentials Webinar Series is also a good place to start learning about the RStudio IDE.

To install:

• http://www.rstudio.com/products/rstudio/download

• Choose the appropriate installer for your OS under the Installers for Supported Platforms heading.

LaTeX

LaTeX is required in order to create PDF reports. The package will fall back on HTML reports if pdflatex is not installed. The HTML version of the report works well on screens but does not print nicely.

If you'd like to output the report as well-behaved, nicely-printing PDF, a LaTeX distribution is required.

To install:

• Windows (MiKTeX)

  • Install the "Basic MiKTeX Installer" from https://miktex.org/download.
  • Optional: After installation, run the update wizard by running "MiKTeX Update" from the Windows start menu.

• Mac OS X (MacTeX)

  • Install the full MacTeX installation from http://www.tug.org/mactex/. (Warning: the download is big!)

• Ubuntu (texlive)

  • Install texlive via the package manager or on the command line with sudo apt-get install texlive-core.
  • Also install fonts and extra things with sudo apt-get install texlive-fonts-recommended texlive-latex-recommended.
  • Or alternatively install the full texlive package with sudo apt-get install texlive-latex-extra.

Installation

Mac/Linux

On Mac OS X or Unix-based systems, install this package by running the following commands inside RStudio or the R command line:

install.packages("devtools")
devtools::install_github("gadenbuie/mctestanalysis")

Windows

On Windows, the current version of devtools doesn't correctly install all package dependencies, so they need to be installed manually before installing the MCTestAnalysis package.

Copy and paste the following command into the RStudio console or the R command line to install the MCTestAnalysis package dependencies.

install.packages(c("devtools", "dplyr", "ggplot2", "shiny", "miniUI",
                   "ltm", "psych", "psychometric", "reshape2", "tibble",
                   "DT", "rmarkdown", "backports", "pander", "gridExtra",
                   "survival"))

Then run the following to complete the installation of the MCTestAnalysis package.

devtools::install_github("gadenbuie/mctestanalysis")

Usage

Open RStudio or R on the command line or GUI and run the following commands:

library(MCTestAnalysis)

# Launch test exploration interface
explore()

# Create a test analysis report
report()

When running explore() a browser window will open with the MC Test Analysis exploration application. The report() function will launch a window within RStudio to guide the user through the creation of a PDF report.

Data format

This package requires both an Answer Key and a table of student responses. An example of each table is provided, in the preferred format, in the inst/extdata folder of this repo, or from links on the "Import" tab of the explore() or report() interfaces.

A detailed overview of the required data format is available at http://www.eng.usf.edu/~kaw/MCTestAnalysis/MCTestAnalysis_input.pdf. Additional example test results and answer key CSV files are also available.

mctestanalysis's Issues

plotTetrachoric group by concept doesn't work if concepts and questions not already grouped

If the concepts groups are sequential, then the concept group annotation in plotTetrachoric works well -- ie. Q1 = A, Q2 = A, Q3 = B, Q4 = B, etc.

If the concept groups are random throughout the test, then the concept grouping overlays on plotTetrachoric get confused.

Fix report download

seems PDF report download doesn't find the generated PDF

Shiny app does not work

https://gadenbuie.shinyapps.io/mctestanalysis/ does not work. It gives a blank page. Tried different browsers.

https://miktex.org/download service unavailable

https://miktex.org/download Is there an alternative for pdflatex

Desktop version

Same problem of no pdf report

utput_format(out_fmt) :
pdflatex not found, defaulting to HTML output.
Install MiKTeX (Windows), MacTeX (Mac), or texlive (Linux) for PDF support.
Warning in matrix(unlist(drop.item), ncol = 10, byrow = TRUE) :
data length [16] is not a sub-multiple or multiple of the number of columns [10]
Warning in matrix(unlist(drop.item), ncol = 10, byrow = TRUE) :
data length [16] is not a sub-multiple or multiple of the number of columns [10]
Warning: Removed 2 rows containing missing values (geom_bar).
Quitting from lines 132-133 (mct-report.Rmd)

Tabel 3 shows numbers without labels in PDF format

When choosing an export option for output, Table 3 (classic test theory summary) appropriately shows seven numbers, but the labels are missing. They do show labels in web browser though.
[Page from PDF file to report to Github MCTestAnalysis Developer.pdf](https://github.com/gadenbuie/mctestanalysis/files/7456411/Page.from.PDF.file.to.report.to.Github.MCTestAnalysis.Developer.pdf

)

loadAllData (and test/answer) should be able to import data.frames

Currently these functions accept file names to csv files, but they should be able to import data.frames just as easily.

Add support for letter test options

The options for loading data target users who are using numbers to indicate options, rather than letters.

Using letters seems like a reasonable use case and should be supported:

• Sort options in plots, etc. alphabetically
• Allow user to pick a letter or enter a character for the "blank" or "missing" indicator

Better error catching, messaging and fallbacks for malformed or insufficient data

We need a data checker that can catch when badly formatted or insufficient data are provided.

Current failure point: when using the example test and answer key data, the explore() function fails without providing any kind of legible error message.

pdflatex

https://miktex.org/download Is there an alternative for pdflatex

Add ability to download report from explore() app

Just an idea to document here, but it should be possible to generate and download the report document from within the explore() app.

Individual ICC plots missing

When ICC curves are reported individually rather than groups, printed PDF is missing plots for item numbers >18.

Quitting from lines 397-442 (mct-report.Rmd)

The report generation is quitting due to figure margins being too large. In Explore mode, I'm not having issue seeing any of the figures, and the figures are populating in folders in my directory. This has been an issue for item banks that have exactly 59 or 60 items, but never for banks that are smaller or even much larger (e.g., 200 items). It is happening only for one subdomain of my exam - when I run the entire exam, there is no issue with the report, but when I submit only the 60 items from one domain, the report does not generate. There are 6 exams with identical numbers of items, and the same issue appears with the same subdomain with each exam (always the one with 59 or 60 items).

Error is "Quitting from lines 397-442 (mct-report.Rmd)"

Generating PDF file does not work.

This is for the online version at https://apps.garrickadenbuie.com/mctestanalysis/. One can see the data by clicking around. But clicking on PDF gives error: "PDF file says File not found An error occurred during a connection to apps.garrickadenbuie.com. Check the file name for capitalization or other typing errors. Check to see if the file was moved, renamed or deleted."

Concept group alpha estimate unclear

When MCTest estimates alpha for a concept group as part of an overall exam, the value is much lower than when I submit just the concept group to an analysis. Is the overall test alpha somehow informing the value for the subgroup?

Also, could there be an option for handling missing data? As is, missing values are imputed as 0 (incorrect) which requires filtering before submitting the data for analysis. Without listwise deletion prior to analysis, large number of missing values (for, say, practice tests) result in heavily-biased parameter estimates

Create example test data that "works" everywhere in Repo

The current example test and answer data provided in extdata with the package is referenced in several places and is used to provide an example of the test and answer key formats.

But the data was not designed to be used within the MCTestAnalysis functions. It's a fairly straight-foward use case and expectation that the data would work as an example in places like explore(), so better example data should be produced to be used throughout.

Shiny app page not found

https://gadenbuie.shinyapps.io/mctestanalysis/ Page not found error

Report upgrades and tweaks

• Add summarizeCTT(mctd, 'item') to the report
• Remove titles from Discrimination Index plots already in y-axis
• Item text for overall vs question plots
• Add table and reference for Versatile criteria
• Add ICC curves for all questions as well as concepts (as a choice in report())
• Add text for Tetrachoric plot
• Spelling error in Scree plot section: *analyst
• Move distractor analysis to subtopic of Classic Test Theory
• Change "Factor Analysis" to "Introductory Factor Analysis"