alexismlopez / common-workflow-service

NASA-AMMOS Common Workflow Service (CWS)

This repository is the open-source release of the NASA-AMMOS Common Workflow Service (CWS). It is the culmination of many years of development internally at JPL, and we are now bringing it to the public in hopes that the open-source community might benefit from its release.

CWS is built on top of the BPMN Workflow Engine. CWS extends Camunda's functionality by layering an intuitive user interface, auditable logging, extensibility with code snippits and adaptation layers, and plenty of other useful tools such a powerful external task engine, custom process initiators, and much more.

While this repository is mostly complete, the documentation will be a work-in-progress for some time as we parse through our internal docs and add them here. The cws-test package is also in need of an update, but we've included it here as it contains useful examples of integration testing for CWS.

While documentation is still in the works, please feel free to open an issue with your inquiry.

See the wiki for more information.

Installation

Prerequisites

• Maven: Used to dynamically download libraries and other required project dependencies.
  • For Home-brew users:
    • Install Maven using: brew install maven
    • Verify installation using: mvn -v
• Docker: Used to run external Elasticsearch, and create and configure mariaDB database container
  • Recommended minimum system requirements from Docker Resources window:
    • CPUs: 5
    • Memory: 14.00 GB
    • Swap: 1 GB
    • Disk image size: 64 GB
• MariaDB or MySQL database set up on either your local machine or a remote host. You will also need to create the following:
  • A database for CWS to use. cws_dev is a good default name.
  • A database user with full access to the above database.
• ITerm2: Currently these build scripts include commands to open new terminal windows using ITerm2, so they are best run from that terminal.
• Logstash 8.8.0+: Download Logstash for your platform. Uncompress it (only if it is a .tar.gz) and then ZIP back it up with the filename 'logstash-8.8.0.zip' and place in install/logging/. This is a temporary workaround while we clean up our installation process. You can find the zip download here.
• Elasticsearch 8.8.0+: CWS requires an externally-configured elasticsearch cluster to be set up. You can use an SSL Secure Elasticsearch with or without authentication, or an Insecure HTTP Elasticsearch.
  • The "Elasticsearch Setup" instruction below provides a contained Dockerized way of running Elasticsearch. This serves as an alternative to installing Elasticsearch.
• Tomcat keystore and truststore files (needed for CWS web console to work properly):
  • You will need to add your own Tomcat keystore file to this path: install/.keystore
  • You will need to add your own truststore file to this path: install/tomcat_lib/cws_truststore.jks
  • See: https://tomcat.apache.org/tomcat-9.0-doc/ssl-howto.html
  • Java 11 JDK: CWS only runs on JDK 11 now, but planning for JDK 17 soon.
    • For Homebrew users:
      • Install OpenJDK 11 using: brew install openjdk@11
      • Check the exact version installed using /usr/libexec/java_home -V
      • Add to your Shell startup (e.g. .zprofile): export JAVA_HOME=$(/usr/libexec/java_home -v X.X.X)
        • Replace the X.X.X version above with the OpenJDK 11 output from the /usr/libexec/java_home -V command.

Development Environment Configuration

MariaDB Setup

Generate mariaDB Docker Container and Create Database Instance for CWS:

docker run -d -p 3306:3306 -e MYSQL_DATABASE=__DB_NAME__ -e MYSQL_ROOT_PASSWORD=__ROOT_PW__ -e TZ=America/Los_Angeles --name mdb103 mariadb:10.3

Replace __DB_NAME__ with your desired database name.
Replace __ROOT_PW__ with your desired password.

__DB_NAME__ and __ROOT_PW__ must match parameters set in script file: <personal-dev>.sh

Directly access mariaDB with:

mysql -h 127.0.0.1 -u root -p

Enter the password above when prompted.

CWS must have been built, in this case using a build script, in order to directly access mariaDB with the MySQL monitor, as the build script contains required information to access the database. See the Building CWS section for an example build script.

Make sure cws_dev database in created mariaDB instance before moving forward to build CWS

Elasticsearch Setup

Open new Shell terminal designated for running ElasticSearch.

• cd into install/docker/es-only directory and run Docker Compose:

docker-compose up

Updating Presets and Default Settings

Preset configuration variables like default_smtp_hostname and default_cws_ldap_url can be found in files:

• /install/installerPresets.properties
• /install/example-cws-configuration.properties
• utils.sh

---

Building CWS

In a different terminal window cd into root of common-workflow-service folder and follow Build CWS instructions.

For development we tend to create our own separate build script <personal-dev.sh> (firstinitial-lastname.sh), i.e.:jsmith.sh, that calls dev.sh. Here's an template for your personal build script that will work for development on a local machine:

• Correctly set the Elasticsearch configuration within your personal script by assigning the proper protocol, HTTP or HTTPS, to ES_PROTOCOL with Elasticsearch hostname assigned to ES_HOST.
  • Example:
    • ES_PROTOCOL="HTTP"
    • ES_HOST="locahost"

#File: jsmith.sh

#!/bin/bash

HOSTNAME=localhost

# Used in cws-test
echo "$HOSTNAME" > cws-test/src/test/resources/hostname.txt

SECURITY="camunda"

# Stop CWS is it is currently running
./stop_dev.sh

# DB config
DB_TYPE=mariadb
DB_HOST=127.0.0.1
DB_NAME=cws_dev # needs to match the db you set up beforehand
DB_USER=root # needs to match the user you set up beforehand
DB_PASS=     # could also be specified with environment vars
DB_PORT=3306 # mariadb default

USER=   # Username
CLOUD=  # Enable cloudwatch monitoring

EMAIL_LIST="{email}"

ADMIN_FIRST="{first}"
ADMIN_LAST="{last}"
ADMIN_EMAIL="{email}"

# ES config
ES_PROTOCOL="HTTP"  # options: 'HTTP' or 'HTTPS'
ES_HOST="localhost"
ES_PORT=9200
ES_USE_AUTH=n
ES_USERNAME="na"
ES_PASSWORD="na"

# Num of workers to start. 1 is the minimum.
NUM_WORKERS=1

# Default value is 16. 1 is the mininum.
WORKER_MAX_NUM_RUNNING_PROCS=16

# Default value is 1. Specifies the number of days (int) until the
# abandoned workers in the cws_workers database table are cleaned out.
WORKER_ABANDONED_DAYS=1

# Run the dev script
./dev.sh `pwd` ${USER} ${DB_TYPE} ${DB_HOST} ${DB_PORT} ${DB_NAME} ${DB_USER} ${DB_PASS} ${ES_PROTOCOL} ${ES_HOST} ${ES_PORT} ${ES_USE_AUTH} ${ES_USERNAME} ${ES_PASSWORD} ${CLOUD} ${SECURITY} ${HOSTNAME} ${EMAIL_LIST} ${ADMIN_FIRST} ${ADMIN_LAST} ${ADMIN_EMAIL} ${NUM_WORKERS} ${WORKER_MAX_NUM_RUNNING_PROCS} ${WORKER_ABANDONED_DAYS}

Run Personal Dev Script

To build and run CWS, use your <personal-dev.sh> i.e.:jsmith.sh script - its usage is as follows:

./jsmith.sh

The above script will build CWS, verify your configuration, then will start the CWS console and workers. The script will provide a link to access the console dashboard once everything has started up!

Stopping CWS

You can stop CWS by running ./stop_dev.sh. The script will bring down the console and all local workers.

Running Unit and Integration Test: cws-test

Start test.sh script by running:

./test.sh

This will produce jacoco reports with code coverage measurements.

Contributing

Please see our contribution guidelines.

License

The source files in this repository are made available under the Apache License Version 2.0.