nextflow-io / nf-nomad

[WIP] Hashicorp Nomad executor plugin for Nextflow

nf-nomad plugin

This plugin implements a Nextflow executor for Hashicorp Nomad.

Maintainers

Please note that this is a community contributed plugin and is a collaboration between

1. Abhinav Sharma (@abhi18av) as part of his PhD work at the Stellenbosch University and Jorge Aguilera (@jagedn) as a contributor from Evaluacion y desarrollo de negocios, Spain.
2. Tomas (@tomiles) and his team from Center For Medical Genetics Ghent, Belgium.

The contribution roles during the development of initial plugin and testing along with the long term commitments have been discussed in development and infrastructure group.

Feel free to reach out to us on the #platform-nomad channel on Slack for discussions and feedbacks.

Plugin Assets

• settings.gradle

  Gradle project settings.

• plugins/nf-nomad

  The plugin implementation base directory.

• plugins/nf-nomad/build.gradle

  Plugin Gradle build file. Project dependencies should be added here.

• plugins/nf-nomad/src/resources/META-INF/MANIFEST.MF

  Manifest file defining the plugin attributes e.g. name, version, etc. The attribute Plugin-Class declares the plugin main class. This class should extend the base class nextflow.plugin.BasePlugin e.g. nextflow.Nomad.NomadPlugin.

• plugins/nf-nomad/src/resources/META-INF/extensions.idx

  This file declares one or more extension classes provided by the plugin. Each line should contain the fully qualified name of a Java class that implements the org.pf4j.ExtensionPoint interface (or a sub-interface).

• plugins/nf-nomad/src/main

  The plugin implementation sources.

• plugins/nf-nomad/src/test

  The plugin unit tests.

ExtensionPoints

ExtensionPoint is the basic interface which uses nextflow-core to integrate plugins into it. It's only a basic interface and serves as starting point for more specialized extensions.

Unit testing

Run the following command in the project root directory (ie. where the file settings.gradle is located):

./gradlew check

Testing and debugging

To run and test the plugin in a development environment, configure a local Nextflow build with the following steps:

1. Clone the Nextflow repository in your computer into a sibling directory:

   git clone --depth 1 https://github.com/nextflow-io/nextflow _resources/nextflow

2. Generate the nextflow class path

   cd _resources/nextflow && ./gradlew exportClasspath

3. Compile the plugin alongside the Nextflow code:

   cd ../../ && ./gradlew compileGroovy

4. Run Nextflow with the plugin, using ./launch.sh as a drop-in replacement for the nextflow command, and adding the option -plugins nf-nomad to load the plugin:

   ./launch.sh run main.nf -plugins nf-nomad

Package, upload and publish

The project should be hosted in a GitHub repository whose name should match the name of the plugin, that is the name of the directory in the plugins folder (e.g. nf-nomad).

Follow these steps to package, upload and publish the plugin:

1. Create a file named gradle.properties in the project root containing the following attributes (this file should not be committed to Git):

   • github_organization: the GitHub organisation where the plugin repository is hosted.
   • github_username: The GitHub username granting access to the plugin repository.
   • github_access_token: The GitHub access token required to upload and commit changes to the plugin repository.
   • github_commit_email: The email address associated with your GitHub account.

2. Use the following command to package and create a release for your plugin on GitHub:

   ./gradlew :plugins:nf-nomad:upload

3. Create a pull request against nextflow-io/plugins to make the plugin accessible to Nextflow.

nf-nomad's Issues

Upgrade to last version of Nextflow

evaluate if we can upgrade to NF 24.x

Fix gh pipeline

Current pipeline is using old versions of Java (8 & 11) and it's failing

Testing infrastructure with Github Actions using custom-runners on nf-core infra

The goal of this task is to

• Use GitHub actions
• Setup a test nomad cluster
• Use CSI-hostpath volumes

Create a new translation class to address NF -> Nomad conversion

@jagedn initially suggested a new TaskResource class addressing this translation layer.

I do think that we can explore the idea regarding a specific "translation" class which covers all aspects of NomadJobDefinition from a NextflowTask, however I suggest that we still continue the current iterations and revisit this once the plugin has stabilised in its design - what do you think?

Originally posted by @abhi18av in #27 (comment)

Improve the polling logic for Nomad Job latest allocation

The current job polling logic (in v0.0.2) focuses on the job status at the moment, whereas within Nomad, one job can have multiple allocations.

When observed on Tower https://tower.nf/orgs/ABHINAVSHARMA-ORG/workspaces/nf-nomad-dev/watch/59CibE1sgsySZI , the count for runnings jobs is not decremented when they transition to Succeeded stage.

Implementation of Job definition logic

• Implement the NomadService to generate the expected JSON

• Explore the use of template to send the command.* files to Nomad client

• Implement the file transfer logic in .command.run file

• Accommodate the various specifications from TaskRun within NomadJobOperations

• #21

Implement Nomad Job/Task level settings in configuration

• clientToken #24
• clientCertificate https://developer.hashicorp.com/nomad/docs/commands#mtls-environment-variables
• jobAffinity
• jobConstraint
• taskEnv
• jobVolume #22
• jobMeta
• jobReschedule
• jobRestart
• taskLogs
• taskResources #27

rnaseq-nf Nomad definitions

(Manual) implementation of the rnaseq-nf pipeline as a vanilla Nomad Job

• Process-1
• Process-2
• Process-3
• Process-4

Add clientToken

secure Api calls using clientToken

• create a new optional config key
• if present use it in the calls

see https://developer.hashicorp.com/nomad/api-docs#authentication for reference

Missing plugin information in zip

After refactor the gradle tasks related to generate a release version current implementation doesnt produced following information in the META-INF file:

Plugin-Id: nf-nomad
Plugin-Version: 0.0.1
Plugin-Class: nextflow.nomad.NomadPlugin
Plugin-Provider: nextflow
Plugin-Requires: >=23.10.0

add integration tests

create some scripts to run a local nomad where run samples

Add initial documentation for usage, development and introduction

Use gh-pages to publish an initial version of the documentation to explain the aim, how to use the plugin and so on

Allow storage config using nomad `volume` blocks

While docker volumes is a quick way to prototype adding storage to a job/task it is not considered best practice in nomad. Since it is very driver specific approach and bypasses any native storage (Volume) handling in Nomad. Also most production clusters will lock down mounting volumes this way for security reasons. So we should find a user friendly way to do this using the nomad volume block.

Originally posted by @tomiles in #12 (comment)

Roadmap v0.1.0

Assumptions for the plugin

1. docker executor is available on all nodes
2. A shared file system is present across the nomad nodes, with a persistent volume
3. A prod setup of a minimal of two nodes -> server node and client node

BONUS: Internet connection and wave enabled for fusionfs (?)

Pipelines to be tested on nomad cluster setup

• nextflow-io/hello
• rnaseq-nf
• nf-core/fetchngs
• nf-core/rnaseq
• MAGMA

Plugin dev group (Jorge, Abhinav) responsible for

• Developing the plugin structure and local testing setups via a nomad -dev node - #4
• Making sure the Nextflow abstractions translate well to the nomad job abstractions #5
• Testing the standard pipelines for usability and a smooth Nextflow experience locally

Infrastructure group (Tom, Matthias) responsible for

• Setting up a common nomad testing infra (liasoning with nf-core team)
• Setting up the testing infrastructure for pipelines via Github Actions (+ local runner) to be triggered on dev/main branch release/merges #3
• Coming up with the Nextflow config for nomad executor

Long term maintenance

• Jorge and Abhinav invested as developers
• CMGG invested in maintaining the nf-core / Nextflow and Nomad infrastructure