asreview / asreview-makita

Workflow generator for simulation studies using the command line interface of ASReview LAB

ASReview: Active learning for Systematic Reviews

Systematically screening large amounts of textual data is time-consuming and often tiresome. The rapidly evolving field of Artificial Intelligence (AI) has allowed the development of AI-aided pipelines that assist in finding relevant texts for search tasks. A well-established approach to increasing efficiency is screening prioritization via Active Learning.

The Active learning for Systematic Reviews (ASReview) project, published in Nature Machine Intelligence implements different machine learning algorithms that interactively query the researcher. ASReview LAB is designed to accelerate the step of screening textual data with a minimum of records to be read by a human with no or very few false negatives. ASReview LAB will save time, increase the quality of output and strengthen the transparency of work when screening large amounts of textual data to retrieve relevant information. Active Learning will support decision-making in any discipline or industry.

ASReview software implements three different modes:

• Oracle Screen textual data in interaction with the active learning model. The reviewer is the 'oracle', making the labeling decisions.
• Exploration Explore or demonstrate ASReview LAB with a completely labeled dataset. This mode is suitable for teaching purposes.
• Simulation Evaluate the performance of active learning models on fully labeled data. Simulations can be run in ASReview LAB or via the command line interface with more advanced options.

Installation

The ASReview software requires Python 3.8 or later. Detailed step-by-step instructions to install Python and ASReview are available for Windows and macOS users.

pip install asreview

Upgrade ASReview with the following command:

pip install --upgrade asreview

To install ASReview LAB with Docker, see Install with Docker.

How it works

Getting started

Getting Started with ASReview LAB.

Citation

If you wish to cite the underlying methodology of the ASReview software, please use the following publication in Nature Machine Intelligence:

van de Schoot, R., de Bruin, J., Schram, R. et al. An open source machine learning framework for efficient and transparent systematic reviews. Nat Mach Intell 3, 125–133 (2021). https://doi.org/10.1038/s42256-020-00287-7

For citing the software, please refer to the specific release of the ASReview software on Zenodo https://doi.org/10.5281/zenodo.3345592. The menu on the right can be used to find the citation format of prevalence.

For more scientific publications on the ASReview software, go to asreview.ai/papers.

Contact

For an overview of the team working on ASReview, see ASReview Research Team. ASReview LAB is maintained by Jonathan de Bruin and Yongchao Terry Ma.

The best resources to find an answer to your question or ways to get in contact with the team are:

• Documentation - asreview.readthedocs.io
• Newsletter - asreview.ai/newsletter/subscribe
• Quick tour - ASReview LAB quick tour
• Issues or feature requests - ASReview issue tracker
• FAQ - ASReview Discussions
• Donation - asreview.ai/donate
• Contact - [email protected]

License

The ASReview software has an Apache 2.0 LICENSE. The ASReview team accepts no responsibility or liability for the use of the ASReview tool or any direct or indirect damages arising out of the application of the tool.

asreview-makita's Issues

Makita overwrites existing README.md

If files like README.md are present in the folder, Makita overwrites them without a warning.

Generate project folder automatically

Currently the user has to manually create the project folder, the data folder and add the data to the data folder. I would like Makita to generate this folder structure for me, and copy the data as well.

Rename Multiple models

While I like the template, I think the name could do with a renewal. Below are some ideas. If you have an idea of your own, please add them to the thread.

• compare_models: This name is clear and concise. It also aligns with the purpose of the template, which is to compare the performance of different models on a given dataset.
• multi_model_sim: This name highlights the fact that the template is designed to simulate multiple models.
• model_comparison: This name is a bit longer, but it is very descriptive and communicates the purpose of the template well.
• model_eval: Short for "model evaluation", this name is succinct and clearly communicates that the template is used to evaluate different models. It's also short and easy to type.
• model_tester: This name implies that the template is used to test different models, which aligns with its purpose.

Templates have broken comment line

#13 made changed "#" to "::" in the template files.
For instance:

asreview-makita/asreviewcontrib/makita/templates/template_arfi.txt.template

Line 49 in 513984e

:: Generate plot and tables for dataset

Duplicate priors can randomly appear

Because of this line:

asreview-makita/asreviewcontrib/makita/template_arfi.py

Line 29 in 9b8a307

prior_irrelevant = list(np.random.choice(relevant_irrecord_ids, n_priors))

Requirements.txt generation

The package should automatically generate a Requirements.txt file for all used packages such as wordcloud, datatools and insights. I am planning on adding this to a next version.

Makita does not run Jobs.bat

Describe the bug
Succesfully I installed Makita (see README-file)
README.md.

When I run the script jobs.bat, I recieve the following message: "Error while finding module specification for 'jobs.bat' (ModuleNotFoundError: no module named 'jobs').
(see: Statements CLI after running jobs_bat.docx)
Statements CLI after running jobs_bat.docx.
Does this mean that Makita is not correctly is installed?

At the end it states 'asreview is recognized as an internal or external command, operable program or batch file.' On my laptop asreview runs only when I use the statement 'python -m asreview'. Not giving the command 'python -m' results in the same error (asreview lab).

If 'python -m' is necessary for running the jobs.bat file, how can that script be adjusted?

To Reproduce
Steps to reproduce the problem:

Screenshots
If applicable, add screenshots to help explain your problem.

Version information

• OS: Windows
• ASReview version [e.g. 0.6.2]

Additional context
Add any other context about the problem here.

Next Thursday I'll be at the Drift in the morning.

Regards, Tale!

Missing legend with multiple models template

Hi,

In the multiple models script there is a missing legend in the plot_recall, to compare the different modes. We managed to create a legend for the plot with customized plot script on this site: https://github.com/asreview/asreview-insights#example-multiple-curves-in-one-plot. Could anyone include this in the Makita script?

Kind regards