rubenarslan / formr.org

Chain simple surveys into longer runs to build complex studies. Use R to generate pretty feedback and complex designs.

formr survey framework

how to cite

See formr.org/public/about if you are publishing research conducted using formr.

chain simple forms & surveys into long runs, use the power of R to generate pretty feedback and complex designs

This is a framework that allows you to create simple and complex studies using items spreadsheets for the surveys and "runs" for chaining together various modules.

The creator and most users of this software work in personality and developmental psychology, but the framework can also be used for sociological panels, simple experiments or as part of an integrated toolkit.

There are three main components: surveys, runs and the R package.

Surveys

ask questions, get data

Surveys are simple or complicated forms and surveys used to gather information in a single session.

There is a wide variety of items to choose from: text and number inputs, Likert scales, sliders, geolocation, date pickers, dropdowns and many more. They are geared towards power users, so instead of dragging and dropping elements till your fingers bleed, you upload item spreadsheets that can easily be re-used, combined and shared.

Runs

control your study like a boombox

Runs enable you to link surveys and chain them together. Using a number of boombox-themed control elements to control the participant's way through your study, you can design studies of limitless complexity. You can

• manage access to and eligibility for a study:
• use different pathways for different users: – send email invites and reminders:
• implement delays/pauses:
• add external modules:
• loop surveys and thus enable diaries and experience-sampling studies:
• give custom feedback, through OpenCPU's R API.
• randomise participants into groups for e.g. A-B-testing or experiments

The following designs and many more are possible:

• simple one-shot surveys
• complex one-shot surveys (using skipping logic, personalised text, complex feedback
• surveys with eligibility limitations
• diary studies including completely flexible automated email reminders
• longitudinal studies (ie. wait 2 months after last participation or re-contact after they return from their exchange year). The items of later waves need not exist in final form at wave 1.
• longitudinal social networks and other studies that require rating a variable number of things or persons

R package

accompanying R package

Wherever you use R in formr you can also use the functions in its R package. If you want to use the package in a different environment, you'll need to install it using these two lines of code.

install.packages("devtools")
devtools::install_github("rubenarslan/formr")

The package currently has the following feature sets

• Connecting to formr, importing your data, correctly typing all variables, automatically aggregating scales.
• Easily making feedback plots e.g. qplot_on_normal(0.8, "Extraversion") The package also has a function to simulate possible data, so you can make feedback plots ahead of collecting data.
• Some shorthand functions for frequently needed operations on the site: first(cars); last(cars); current(cars); "formr." %contains% "mr."

OpenCPU + R + Knitr + Markdown

OpenCPU is a way to safely use complex R expressions on the web. We use it for all kinds of stuff.

In surveys, pauses, emails and pages you can display text to the user. This text is easily formatted using Markdown a simple syntax that formats text nicely if you simply write like you would write a plain text email. Markdown can be freely mixed with HTML, so you can e.g. insert icons from the Font Awesome library using <i class="fa fa-smile-o"></i>.

If you use knitr syntax, where Markdown can be used, the text will not just be parsed as Markdown (which is mostly static), but also be parsed (anew each time) by knitr. Knitr allows for mixing R syntax chunks and Markdown.
R is a popular open-source statistical programming language, that you can use via OpenCPU, a RESTful interface to the language that deals with the problem that R was not meant to be used as part of web apps and is insecure. R data frames with the same names as the surveys they derive from will be available in this knitr call, they contain all data that the current user has filled out so far.
Combined with norm data etc. you can tailor feedback to the user's input, e.g. show where the user lies on the bell curve etc.

Installation

If you want to test formr or contribute to its development, follow our instructions.

The setup instructions are not suitable for production, i.e. running a secure formr installation on the public web.

Credit

See formr.org/about for funding and contact info.

See composer.json for the PHP components we use and bower.json for the Javascript and CSS components we use.

formr.org's Issues

allow custom CSS for runs

Combined with image upload this would enable users to give studies a different look.

item - rank / sortable / q-sort

i.e. http://jqueryui.com/sortable/

requirements?

• sort choices vertically via drag-n-drop. (do we need a graceful fallback?)
• put multiple options on one level (to reflect the fact that there's many middling options)
• decide on UI

@Deleetdk

Allow users to reset access codes?

Use Case: My access code has been compromised, reset?
This would cause problems with external data, maybe external data should get a different token than the one we send via mail, but don't want to make merging to much of a bother. Maybe we need to store old codes..

switch all markdown to github-flavoured markdown

Because PHPMarkdown still has some of those annoying MD features (intra-word-emphasis, two spaces at the end of a line for linebreaks), it would be nice to switch, I should either find a PHP-equivalent of knitr's github-flavoured MD, or learn to live with the delays caused through openCPU parsing everything. Will have to look into whether this can be sped up.

should check email log so that it never sends more than three emails to the same user in a short amount of time.

when a study is deleted, it leaves behind a run_unit and unit fragment that cause fatal errors in run

Somehow the cascading deletes here are incorrectly set up, deleting a study should of course remove it from any runs it is associated with as well.

better error handling when openCPU is down

currently

Notice: Undefined index: X-ocpu-session in /var/www/formr/Model/OpenCPU.php on line 220

item - "blank"

Once we've got the dynamic evaluation of R in survey items down, we could make an item, all aspects of which can be dynamically determined:

• number of choices
• text and values of choices
• item text
• styling etc.

The result column should be something flexible like TEXT.

Use Ace for knitr (rmd r markdown) editing

Unfortunately Rstudio has extended Ace to support interspersed markdown and R and has not yet given back these modifications to Ace.

Steps so far:

1. cloned into ace
2. npm install dependencies
3. go to tool folder npm install there too
4. move the knitr-Markdown.tmLanguage file from this repo there.
5. Rename it to knitr.tmLanguage in case the dashes are problematic.
6. Open file and change name from "knitr (Markdown)" to simply knitr, in case Ace is picky there.
7. Check generated files: knitr.js and knitr_highlight_rules.js. These might have to be manually adapted according to ace. Haven't tried this yet.
8. cd ../ and node Makefile.dryice.js full --target ../ace-builds to build ace in the parent directory (including kitchen sink).
9. Manually add line:
   Knitr : ["knitr"], before Lua in kitchen-sink/demo.js
10. Go to http://localhost:8888/ace-builds/kitchen-sink.html
11. Select Knitr as mode
12. Errors. Yay.

Problems:
The code is full of todo and fixme:

1. fix grouping (apparently left behind by the Ace converter).
2. Some lookaheads, which JS doesn't have, seem to be used.

Considering that the Rstudio people in their issue said that implementing this in Ace was lots of work including messy dependency stuff, I leave this to future Ruben (or hopefully Rstudio if they release it).

better API

At the moment the only semblance of an API we have allows for closing external modules and for adding users to a run (but see #3).

We need a more principled approach, in the meanwhile users can use e.g. R packages like httr to login and download data, but that requires state.

• automatically generate single use keys, for back and forth with openCPU e.g. in admin overview
• possibly also tokens for authentication for people who want to log in via httr (not a high priority, need to think on what the best way to actually keep these things safe in R is, that doesn't require too much expertise from users)
• should be storing an API key hash, not the API key itself.
• should allow not querying data in full (could be linked to automatic addition of data as in rest of formr, but not via json)

Switch user-supplied run logic to R instead of SQL.

At the moment, admins can specify complex SQL expressions and many servers don't even allow for multiple MySQL users (to enforce the least privilege principle).
R would be a more powerful and possibly more familiar way to handle this for many, the openCPU API is also reasonably safe. However, this might turn out to be (a lot) slower.

rethink formr modules - let problems from existing studies guide us?

Existing modules:

Pause

This simple component allows you to delay the continuation of the run until a certain date, time of day or to wait relative to a date that a user specified (such as her graduation date or the last time he cut his nails). See the OpenCPU + R + Knitr + Markdown section to find out how to personalise the text shown while waiting.

Branch

Branches are components that allow to execute readonly SQL commands on all surveys you created. Depending on whether the results evaluates to true/1 or false/0, you can go to different positions in the run - these can be later in the run or earlier in the run (thus creating loops).

TimeBranch

These components are the bastard children of a Pause + Branch: If the user accesses the run within the specified time frame (like a Pause), the run jumps to one position in the run (like a Branch). If she doesn't, the run progresses to a different position in the run (e.g. to a reminder email). This component is useful, if you need to set up a period during which a survey is accessible or if you want to send reminders automatically after some time elapsed.
See the OpenCPU + R + Knitr + Markdown section to find out how to customise the text shown while waiting.

Email

Using an SMTP gateway that you can set up in the admin area, you can send emails to your users. Using the tag {{login_link}}, you can send users a personalised link to the run. See the OpenCPU + R + Knitr + Markdown section to find out how to personalise email text.

External link

These are simple external links - you can use them to send users to other, specialised data collection modules, such as a social network generator. If you insert the placeholder %s, it will be replaced by the users run_session code, allowing you to link data later. You can choose to "end" this component before the user is redirected to the link or by enabling your external module to call our API to close it, when it's done.

Page

Simple text pages. See the OpenCPU + R + Knitr + Markdown section to find out how to generate personalised feedback.

Survey

Surveys are series of questions that are created using simple spreadsheets/item tables (ie. Excel, OpenOffice, etc.: *.xls, *.ods, *.xlsx, *.csv etc.).
Survey results are stored in MySQL tables of the same name, which can be used for various conditions later on.

You can add the same survey several times or even loop using branches.
You can also use the same survey across different runs. For example this would allow you to ask respondents for their demographic information only once. You'd do this by using a Branch with the condition COUNT(demographics.session_id) > 0 and skipping over the demographics survey, if true.

Survey names may only contain the characters a-zA-Z0-9_ and need to start with a letter.

Known problems

1. All users had trouble with grasping the concept that the run always keeps on running until it meets a deadend. Maybe a module simply named "deadend" (now: Page without ability to continue) would make this more explicit.
2. At the moment it is not clear enough that Pause allows for text & feedback just as fully-fledged as Page (Email should also support the same features, at the moment, the base64 encoding needed for images is not done properly, as they're sourced via data: url schema #25).
3. Multiple reminders for a study are something that almost everybody wanted so far - people had an easy time employing TimeBranch to this effect, but it looks cumbersome in the run, maybe we can simplify this?

Should ask users for more known difficulties.

Move study settings to run settings

should allow a "service message" to be displayed in runs, to make run completely inactive.

As a quick way to takedown a buggy survey or while moving stuff around in a live run.

More and smarter item checking

One gotcha that I recently eliminated was uploading a choice-type item without choices.

Could also check other stuff, here's a preliminary list:

• are all choices unique?
• were options set that don't make sense for this item?
• were CSS classes that do not exist added? need to add a master list. this should only yield a warning, since users can add their own classes.
• automatically trim item names and item types
• sometimes adding spaces to type_options breaks things, should be trimmed automatically

switch custom-rolled btn-radio and btn-checkbox to new native bootstrap in v3

switching to bootstrap v3 will probably mean some breaking changes

Shuffling with blocks

I've added the DB fields for adding a shuffling logic to the survey item table, and an R-function that would do the shuffling, I'm just not sure if I'm happy with this approach (requires sorting the items after loading them from the DB).

Data takeout from run

Maybe it would be easily possible to generate a merged uber-file that includes all data in a run.

reset password

The relevant DB fields are present already reset_token_hash VARCHAR(255) & reset_token_expiry.

item - random number

Easy to do, the only problem would be that a naïve implementation (simple hidden item, saves random number on submission) would mean one minimum submission before users can use the random number.

port social network component to formr

The current version was done using CakePHP and is already somewhat able to be inter-mixed with formr, but it's not ideal.

I don't know whether it would be feasible to use the existing backend codebase, but it would at least be nice to port the JS.

The main problem here is the question "how to generalise the approach to implementing social network items", at the moment they are all hard-coded.
In formr it's currently not possible to repeat one item for multiple people on the same page (except if we're not working with flexible limits. Maybe this is sufficient.).

It is possible to do a person-add form, followed by a SkipBackward IF user said "I want to add another person".

Time zone support

This is a huge pain to do manually in the run.
Have to think of a good way to keep and use up-to-date timezone information on users.

Current plan:

• default to formr timezone in both formr, mySQL and openCPU
• if a researchers adds a timezone field to a survey, as soon as this is filled out, automatically rely on the most recent reported timezone
  • in formr pauses
  • in openCPU
• store the relevant time data with time zone information (usually we we store timezone as part of the datetime mysql field, users will have to be aware of this though)

error message regarding email account should have link

item - file upload / record image / video / sound / speech

allow runs to be deleted and renamed

dynamic showifs

Currently optional questions are only displayed after a pageload. If the item that they are predicated on is still unanswered, they block progress.

collapse Study and Survey class into one, allow placeholder/dummy surveys

This is a leftover duplication. This change should also bring along the ability to add dummy-surveys to runs (atm they are just displayed, not saved, this requires all sorts of extra code).

Email should support figure attachments.

Currently we are using the data URI scheme because of laziness and because it works well for most applications. However, email using data URIs for images (e.g. feedback figures) is currently only supported by few mail clients (eg Mail on iOS). We need to properly attach images (ie MIME encode). Maybe this can be done in R.

If we are to support generating "pretty feedback PDFs" using pander & pandoc, generated PDFs should get attached properly too.

image upload

probably we could "simply" upload images and store the original names together with a large random filename in the DB, when the images are to be displayed, the filenames get replaced. The replacing of image filenames seems bothersome though. Alternative paradigm?

should switch FLOAT to DECIMAL

but need to add some logic to determine the number of decimals

Should be storing an API key hash, not the API key itself.

Allow values of form fields to be preset

I want to do this using R, probably so that all expressions are executed as

tail( expr , 1) # last one if users forget to specify and there are multiple options

This would allow numbers as 1, strings as 'blabla' and expressions as survey1$item1 (bit confusing for some maybe, might catch this and "assume non-existing variable was meant to be a string") and if people want to do complex stuff like using the first choice in a diary they can still do head(diary$item3, 1).

email address verification

the relevant DB fields are present already: email_verification_hash VARCHAR(255) & email_verified TINYINT(1) .

allow runs to be exported and imported (JSON)

This way you could for example easily copy-paste a complex diary study and share study designs.

I imagine exported JSON for a "Filter" would e.g. look like this:

{
    "1":
    { 
          "type": "SkipForward",
          "condition": "survey$item == 'wrong'",
          "if_true": "2" ,
          "automatically_jump": true,
          "automatically_go_on": true
    },
    "2":
    { 
          "type": "Stop",
          "title": "You cannot continue",
          "body": "You may not participate."
    }
}

People would both be able to paste JSON for import into a textbox, but we would also hook JSON snippets to buttons, so that users would be able to click a "filter" icon and get the above Skip/ Stop combo. The same would be possible with way more complicated combos.

allow image upload to admins

for uploading logos, illustration etc. logos should become specifiable in the survey settings or maybe for runs.

Use pandoc instead of knitr (at least for email feedback generation).

openCPU supports pandoc and Latex too, so this should be easy.

Should provide a feature comparison table with other tools

But only once we look best in comparison ;-)

Maybe this should go in the formr github Wiki...

highlight.js should be managed using bower

currently just minified JS taken from knitr.
https://github.com/isagalaev/highlight.js

add more options for tinkering with the layout via preset CSS classes (and document them)

currently there is almost no documentation on the available CSS classes.
these were requested

label_align_left / _ right _center (already exist, but with hyphens)
answer_align_left / _ right _center (would allow e.g. rating_button to line up better)

maybe options to fix the width of the rating button labels.

Item table import breaks hard when type column is not placed leftmost

Should be easy to check for and fix. Could also make this more flexible, but would imply some rewriting and another massive loop, so not for now.

formr R package

We plan to release a dedicated formr R package

• some necessary shortcuts for the run-environment
• easy feedback graphs
• some data munging stuff
  • data takeout for all surveys in a run,
  • building scales from items automatically

allow changing email address in edit_user

should JS-change progress bar

Would have to store the number of remaining items in a data attribute of the form, so that it can only grow to a certain limit.

External module should know R

At the moment we just have a placeholder variable. Simplicity is nice, but it would be cool, if API calls etc. could precede the redirection. Should probably always result in a URL.

Unfortunately this implies forcing users to add quotation marks, a source of errors. Could check whether the text starts with http: and auto-quote if so, everything else might be too invasive.

improve mobile usage

• offline data storage, so that forms (especially experience sampling) can be uploaded later
• offline reminders - dont know how yet
• improve layout and responsiveness for mobile devices
• skipping logic should work offline

remove substitutions for the benefit of dynamically evaluated r+markdown in survey item and choices text

• The "substitutions" feature in studies can be removed, once we
  1. add a text_parsed and choice_parsed field to the schema
  2. check these fields on display - if they're null this means there is a dynamic component which needs to be evaluated every time using the existing knitForUserDisplay function.
  3. It would be very powerful if we had an item that determines not only choice text, but everything about choices dynamically (low prio #23 ).

Excel has this habit where cells beginning with a ' single quote are understood to be text.

Unfortunately, some users might begin a string with a single-quote to mark up a character string for R.
This leads to parsing errors in R. Not sure what the best solution is, because the string is already erroneously displayed in Excel. Prefixing it with another single quote is the only solution that yields a correct display in Excel and in formr apparently.

Maybe this is a reason to use double quotes in examples throughout.

Another way to minimise trouble with this issue, would be: if an object is not found, catch that error and interpret it as string (how PHP deals with missing constants). Might lead to other horrible misunderstanding down the road.

Or check for fields ending with a single apostrophe and not starting with one?

progress can become smaller when questions enabling a lot of optional questions turn on

Unsure about how to handle this. Treat optional question as 1/2 as long as they haven't been ruled out?

mass delete testing remnants from runs and study sessions

Not possible yet. It's also possible that some user_session deletes don't go through because deletion cascades are restricted.

store cronjob log in DB, so that runs can be separated

currently just a .log file and a bit unsystematic error/exception handling