mhtess / submiterator-batch

Scripts for posting batches of HITs to MTurk via submiterator

Running submiterator in batch mode

Scripts for posting batches of HITs to MTurk via Submiterator

Setup

• Check that submiterator is available as a system-wide command so that you can run, e.g. submiterator posthit example (follow the instructions from the Submiterator README).

• Make a folder for your experiment containing a submiterator .config file

Usage

To post batches of experiments:

./post_batches.sh nameOfExpt nBatches nTurkersPerBatch

• nameOfExpt: a string, name of the folder you created above
• nBatches: the number of batches to run
• nTurkersPerBatch: the number of turkers you want for each batch (note, if you the number of turkers per batch is 9 or fewer, you avoid the +20% mturk fee)

To get batches of experiments:

./get_batches.sh nameOfExpt

To merge the data (or invoices, subject information, etc.)

File merge_invoices.R should be run inside R. (need to edit the details of the file)

submiterator-batch's Issues

Add more instruction for integrating with unique turker

This was an email I wrote. Some information could be put in the readme for helping run batched submiterator expts

Note: this uses submiterator

Here are the steps for your convince:

1. Unique Turker

   • Go to http://uniqueturker.myleott.com and create a unique ID for your expt: I usually do something like “mht-hab-priors-20151221a”
   • Add this to the init function in your expt.js

   repeatWorker = false;
   (function(){
   var ut_id = "mht-hab-priors-20151221a";
   if (UTWorkerLimitReached(ut_id)) {
   $('.slide').empty();
   repeatWorker = true;
   alert("You have already completed the maximum number of HITs allowed by this requester. Please click 'Return HIT' to avoid any impact on your approval rating.");
   }
   })();

   • Source the attached uniqueturker.js file in your html

     <script src="../../../_shared/js/uniqueturker.js"></script>

   • Add a warning for your turkers on the first slide, for example:

Please note: There will be multiple posted versions of this HIT (name: Language Study - Actions 494) throughout the day. Please attempt only one HIT this series. You will not be allowed to complete multiple HITs in this series.

1. Set up your config file (same config file attached)

   • match the study name you put in your warning: “Language Study - Actions 494”
   • number of assignments = 9

2. Run post_groups_of_9.sh (attached):

   ./post_groups_of_9.sh experimentName nRounds

where experimentName is the prefix to the .config file
and nRounds is the number of batches of 9 your want to post

1. Re-run post_groups_of_9.sh to add more subjects

What I do is: ./post_groups_of_9.sh experimentName 1 to collect 9 subjects
Wait until those are done. Make sure there are no bugs.

Run ./post_groups_of_9.sh experimentName 4

For big expts, it is probably to your advantage (time wise) to post a small number of batches (3-5) every so often, so that your experiment appears towards the top of the queue. This will potentially speed up data collection for big expts.

If you have an N that is not a multiple of 9, be sure to change the .config file (to whatever you need to top it up) when you run your last round

1. Run get_groups_of_9.sh to pull all the data
2. Use the attached merge_invoice.R script to compile the invoices, trial information, subject information, and whatever other expt specific files into one file for ease later on.

Let me know if you run into any snags. You can test it on the sandbox as well. Just post 2 batches, do 1 of them, then try to do the other. You should get a warning. If you don’t, make sure you did all of the steps in number 1.

If you have permissions issues, run
chmod u+x post_groups_of_9.sh
chmod u+x get_groups_of_9.sh