fredcallaway / heroku-experiment Goto Github PK

Can we build an experiment using JsPsych that functions using Psiturk and Heroku with a similar structure?

Context
PsiTurk Version: 3.3.1
Hosting Platform: Heroku
Experiment Repository: fredcallaway/heroku-experiment
Problem Description

The experiment runs perfectly fine, and participants can complete and submit the experiment. However, the data is not being saved in the local participant.db file. Consequently, when attempting to approve workers, I receive the following error:

Response(Operation: approve_assignment_by_assignment_id | Status: error | Exception: AssignmentIdNotFoundInLocalDBError: )

Steps to Reproduce

1. Deploy the experiment to Heroku following the steps provided in the repository fredcallaway/heroku-experiment. changes the config file with the ad url
2. Run the experiment to completion.
3. Submit the experiment.
4. Check the local participant.db file for saved data.
5. Attempt to approve a worker using the worker approve command.

Observed Behavior

1. Experiment runs and completes without issues.
2. Data submission appears to be successful on the frontend.
3. No data is saved in the participant.db file.
4. Attempting to approve workers results in AssignmentIdNotFoundInLocalDBError.

Expected Behavior

1. Data should be saved in the participant.db file upon experiment completion and submission.
2. Worker approval should proceed without errors.

Any help or suggestions on resolving this issue would be greatly appreciated. Thank you!

Our configuration currently has threads=1. Given my experiences running the server on Heroku I think we should probably set this to auto. Below is a plot of Dyno Load (see definitions for metrics in plot here. load is defined as "The load value indicates a runnable task (a process or thread) that is either currently running on a CPU or is waiting for a CPU to run on, but otherwise has all the resources it needs to run. The load value does not include tasks that are waiting on IO.")

On the left, you can see traffic as a result of a small pilot (4 participants) on a Hobby core. On the right, you can see traffic as a result of a larger pilot (160 participants) on 5:Standard-2x, and later 9:Standard-1x cores. Since the 1M load max never exceeds 50% (and in many cases doesn't exceed 33%), I think it's worth increasing the number of threads somewhat substantially to make better use of compute resources. Since threads=auto sets workers to 2 * # CPUS + 1 (code here), this makes it a natural choice for the single and dual core case (which would result in 3 and 5 threads respectively).

Essential feature request: Emulate TurkPrime's (now "Cloud Research") HyperBatch feature to avoid accruing an extra 20% MTurk fee for having more than 9 subjects / HIT. I've used TurkPrime quite a bit. The feature is tried-and-true, but there's a fee for it (smaller than the 20% amazon fee but still worth avoiding).

See:
https://gist.github.com/deargle/f454a156d3a72adf62869e2ef09101d8
https://github.com/ddbourgin/mturk_utils

Psiturk docs state that the psiturk.js static file is actually generated, not read from the filesystem:

Q: Where is the /static/js/psiturk.js file? It doesn't appear in any of the experiments I have downloaded!
A: psiturk.js doesn’t actually “exists” as a file in the static folder of any project. Instead, the psiturk server/command line tool automatically generates this file. The best way to view it is by “view source” in your browser while debugging your experiment. While somewhat unintuitive, this ensures that changes to psiturk.js are linked to new versions of the overall psiturk command line tool (since they are tightly interdependent).

https://github.com/NYUCCL/psiTurk/blob/c847801e9c02a9aa16a70407e91de5d0d2ac21f3/doc/faq.rst#where-is-the-staticjspsiturkjs-file--it-doesnt-appear-in-any-of-the-experiments-i-have-downloaded

May be worth considering how to write to endhit and status from Prolific, since those variables are part of the logic for counterbalancing. Code here.

This would address a common pain point where people don't see the data when they just visit /exp

(From Jonathan)

I’m getting a bizarre error when I go to fetch the data from my experiment (running bin/fetch_data.py version; this has been working completely fine until now).

psycopg2.OperationalError: connection to server at "ec2-44-215-1-253.compute-1.amazonaws.com" (44.215.1.253), port 5432 failed: FATAL:  password authentication failed for user "pecjzmgvmcffme"
connection to server at "ec2-44-215-1-253.compute-1.amazonaws.com" (44.215.1.253), port 5432 failed: FATAL:  no pg_hba.conf entry for host "84.33.153.8", user "pecjzmgvmcffme", database "d1gh6l0jffv33e", no encryption

The complicated startup solution that I wrote last year should actually be superseded by psiTurk's builtin support for Heroku. I think the only change we'd need to add to the README is heroku config:set ON_CLOUD=1 (though b/c we use an older psiTurk version, it may require ON_HEROKU), though I'd want to read the docs and read the script more closely to ensure that's the case. We wouldn't want to directly run those things since they create new files (like Procfile and requirements).

should update!