mohamedouladi / wipp-1

Web Image Processing Pipeline (WIPP) - deployment scripts

WIPP - Web Image Processing Pipelines

What is WIPP?

Web Image Processing Pipelines (WIPP) is an open-source web-based algorithmic plugin platform for trusted image-based measurements from terabyte-sized images developped at the National Institute of Standards and Technology (NIST), in collaboration with the National Institutes of Health (NIH) - National Center for Advancing Translational Science (NCATS).

The goal of this project is to lower the bar for users and algorithm developers to execute and share image analyses over terabyte-sized image collections in high-throughput and high content microscopy imaging applications.

• Upload collections of microscopy images,
• Register algorithmic plugins,
• Configure image processing workflows,
• Stitch grids of small fields of view,
• Visualize and analyze large fields of view interactively using the pyramid view,
• Extract image features and customize scatterplots,
• Train and use Tensorflow models for image segmentation,
• Prototype algorithms with polyglot Jupyter notebooks.

WIPP is designed as an expandable platform leveraging containerization and orchestration technologies such as Docker and Kubernetes and is meant to be deployed on a Kubernetes cluster (see Installation instructions).

The core of the WIPP framework consists of:

• the WIPP REST API, based on the Java Spring framework,
• the WIPP web UI, an Angular 7 application,
• a MongoDB database,
• Argo Workflows, a container native workflow engine for Kubernetes,
• WIPP Plugins, algorithmic plugins packaged as Docker images.

Additional tools can be added to the WIPP core for a more complete system:

• JupyterHub, to develop polyglot Jupyter notebooks that can be used in WIPP workflows,
• Plots, for scalable and interactive scatterplots visualization,
• Tensorboard, for Tensorflow jobs monitoring,
• Elastic Stack for application, logs and system monitoring.

Installation

For a quick start, we provide instructions for the deployment of a complete installation of WIPP on a single-node Kubernetes cluster.

Installing WIPP on a single-node cluster

Instructions are provided for popular operating systems using simple solutions for setting up a single-node Kubernetes cluster:

• macOS (with Docker Desktop)
• macOS (with Multipass+microk8s)
• Linux (Multipass+microk8s)
• Windows 10 (Multipass+microk8s)

The following components are included in the single-node deployment:

• WIPP REST API and UI,
• MongoDB,
• Argo Workflows,
• JupyterLab,
• Plots,
• Tensorboard.

GPU support - optional

We provide WIPP plugins for CNN-based segmentation using Tensorflow. These plugins require access to a GPU in the Kubernetes cluster.
The requirements for running the using the WIPP UNet CNN Semantic-Segmentation Training plugin and WIPP UNet CNN Semantic-Segmentation Inference plugin are:

• Cuda 10.1 installed,
• nvidia-docker-runtime and set as default docker runtime
• GPU support enabled in the cluster (for example, by installing the NVIDIA Device Plugin for Kubernetes or enabling the gpu addon in microk8s)

Alternatively, some GPU cards may be compatible with the preview release of the NVIDIA GPU Operator for Kubernetes, which aims to simplify NVIDIA GPU resources in Kubernetes clusters.

Important security information

The WIPP framework is currently in beta version and instructions are provided for deploying a testing instance of the system.
The Web Image Processing Pipelines system (WIPP) version 3.0.0-beta does not include any web security management. WIPP 3.0.0-beta allows unrestricted uploading of files via the web browser interface and the uploaded files interact with the file system as well as with an instance of MongoDB database. WIPP 3.0.0-beta is intended for deployment on private networks behind a firewall. Future releases will include account and upload access management.

User Guide

User guide detailing how to perform routine operations in WIPP, such as uploading images, registering plugins and configuring workflows is available in the user guide folder.

Test datasets

To test a fresh installation of WIPP, we provide three sample datasets with instructions on how to run test workflows using these datasets in the data folder.

WIPP plugins

Links to manifest files to install the currently available WIPP plugins can be found in the plugins folder.
Documentation about how to create custom WIPP plugins coming soon!

Troubleshooting

If you encounter issues during the installation or usage of WIPP, you can get help using one of the following options:

• open an issue on the repository,
• send an email to the WIPP team.

References

• NIST article and video:
  • A Map App to Track Stem Cells, NIST news, February 13, 2018
• Reference book:
  • Web Microanalysis of Big Image Data, published in January 2018 by Springer Verlag.
• Publications:
  • Peter Bajcsy, Antoine Vandecreme, Julien Amelot, Joe Chalfoun, Michael Majurski, and Mary Brady, “Enabling Stem Cell Characterization from Large Coverage Microscopy,” IEEE Computer, 2016: vol. 49, no. 7, pp. 70-79, URL
  • Antoine Vandecreme, Tim Blattner, Michael Majurski, Peter Bajcsy, Keana Scott, and John Henry J. Scott, “From Image Tiles to Web-Based Interactive Measurements in One Stop,” Microscopy and Microanalyses, Portland, OR, August 2-6, 2015, URL

How to cite our work

If you are using WIPP in your research, please use the following references to cite us.

Peter Bajcsy, Joe Chalfoun, and Mylene Simon (2018), "Web Microanalysis of Big Image Data", Springer-Verlag International, URL

Disclaimer

NIST Disclaimer