webis-de / mturk-manager Goto Github PK

I might be wrong but to me the costs calculation below the current balance in the finances view does not seem right. But I am also not quite sure what those numbers mean in connection to the table.

Max costs it shows 99.80$ which are also the max costs for my most current batch. The costs so far shows 52.10$ and I don't really know where that number comes from...

Overall, I don't know what to make of the numbers, but it could be because I don't understand what they are supposed to mean if they're not connected to a batch instance.

When I test my HIT in the sandbox, it does not show a submit button => I can not submit the HIT. The same Worker template works when I submit it in the official web interface.

For reference: webis19, project "Editorials Summarization"

Add a dashboard including summary information, for example

• submitted assignments
• statistics
• quicklinks

When I click "UPDATE" in the "Edit Worker Template" panel, all changes are reset.

Add a feature that allows for blocking a worker to get new tasks internally while not blocking them on MTurk. This can be accomplished similar to the uniqturker tool.

Add features that allows for internal rejection while approving externally.

Save and restore the pagination info (page size, page, ...) for the (data)-tables.

At least in the Assignments view, when I select the "Rows per page" to be "All", it still shows only 25 (only thing that changes is the text next to the selection drop-down changes as if it would show all.

Related: when I select all (top checkbox), it of course also selects the 25 shown only.

Prevent worker template deletion if this template has any unannotated assignments

It would be nice to see why a CSV file failed validation, for example like this https://blog.mturk.com/csv-file-upload-validation-enhancement-676e57a37c07

After submitting a number of large-ish batches (10 batches at 2mb csv size) and clicking 'sync with mturk', the sync takes forever and eventually redirects to the 'Connection Error'-page.
However, the sync itself seems to succeed, since after updating the credentials and re-establishing the connection, the project is up-to-date.

... to select all assignments on a specific page. Currently there is just a box to select all assignments disregarding of the page.

Let the user customize the preset of different page sizes.
Currently defaults to 5, 10, 25, 30, all items per page.

When I create a batch ("submit"), nothing happens. I can see in the network tab of the web developer tools that I get a 400 response

{"settings_batch":{"qualification_locale":["This field is required."]}}

I noticed that the request contains a field qualificationLocale

There is a "Locale" field, but I do not know what to put there?

Test a worker and requester templates in the MTurk Manager

• custom qualifications
• rewards

If you approve or reject an assignment the status_internal will still show "SUBMITTED", i.e. it will not update.

However, if you internally approve or reject an assignment the status_internal will update to "APPROVED" or "REJECTED".

Since we're all human and we all make mistakes it would be nice to be able to revoke a decision (approval / rejection). :-)

Implement some drag'n drop functionality maybe with https://github.com/SortableJS/Vue.Draggable
https://medium.com/vuetify/drag-n-drop-in-vuetify-part-i-166f2c7a4a82

e.g.: A, RI, R, AI instead of Accept, Reject internally, Reject, Accept internally. In order to safe vertical screen space.

For all templates, the manager does not open the edit window when clicked on the edit symbol (there is something flashing, though, so it might just close it again immediately?). However, I noticed that the area on which a click opens the window is rectangular, and that when I click within the rectangle but outside of the highlight-circle, the edit window opens as usual (which is weird, but lowers the priority for this issue).

Here the two cases:

Click does not open edit window

Click opens edit window

• Add a feature to block and unblock workers.
• Show, if a worker is currently blocked or unblocked.

When I select assignments to annotate, nothing shows up in the view:

The dev tools show that it looks for a "http://127.0.0.1:8000/static/mturk_manager/js/dist/index_annotate.js", which gets a 404. I found that a file of the same name at "http://127.0.0.1:8000/static/mturk_manager/js/annotate/index_annotate.js" (with "annotate" instead of "dist")... did things move?

When annotating the assignments the "Approve internally button" only works with the default reject message, i.e. when I click directly on the button.

If I try to choose a different reason by clicking the down arrow, nothing will happen when a new reason is selected.

Migrate code to Typescript once Vue 3.0 is out

• Review time
  • over time
  • histogram
• Approve/Rejects distributions
  • pie, optional internal rejects
• worker distribution
  • histogram

It seems that the default reject messages update globally for every project on the server.
Updating the default reject message for individual projects might make more sense to avoid clutter or messages that are specific to other projects. :-)

I think this problem was mentioned by Johannes regarding the assignment view before.

If I want to show Rows per page: "All" in the worker view it only shows 25 listings.

If I create a batch that runs 24 hours and assignments are still open at the end, these are still displayed as "pending".
It would be helpful if you could:

• can see how long the batch will remain running
• unedited assignments are listed separately
• a way to create a new batch containing only the unedited assignments.

It would be helpful to be able to access worker statistics from the "assignment template", like:

• accepted/rejected HITs per account, per mturk-manager instance, per project, per batch etc.
• counter of the softblock limit
• a link, which directs you to all assignments a specific worker has done so far

It would be nice if you could display the number of workers per batch. Currently you can only see how many and which workers have worked on a project, there is no overview/filtering of the workers per batch.

If I annotate a lot of assignments that require me to scroll down, the page will jump back up after selecting a reason for rejection. As a result the user has to scroll back to the last annotated assignment again.

Also, in my opinion I think that the "submit" section in the assignment view should be scrolling with the user / be in a fixed place on the screen. If you approve a lot of assignments it would be nice not to scroll all the way back up. :-)

I would like to have some informations (Worker ID, Approve rate, number of approved Hits) about the workers inside the result.csv file.

After working with the manager for quite a bit now I noticed that it would be much nicer if the entries in the batch view (also maybe in the assignment view) would be sorted by creation date, i.e. from newest to oldest.

By default it sorts by name in the batches view but every time I use that view I sort after creation to see the progress of my newest assignments.
Sorting by name will quickly get confusing if there is no pattern to the naming. Especially if you test a lot in the sandbox.

Of course this could be a user specific problem, but I can imagine that other people could face the same problem. :-)