nextflow-io / nf-hello

Nextflow plugin implementation skeleton

nf-hello plugin

This project contains a simple Nextflow plugin called nf-hello which provides examples of different plugin extensions:

• A custom trace observer that prints a message when the workflow starts and when the workflow completes
• A custom channel factory called reverse
• A custom operator called goodbye
• A custom function called randomString

NOTE: If you want to use this project as a starting point for a custom plugin, you must rename the plugins/nf-hello folder and update settings.gradle with your plugin name.

See the Nextflow documentation for more information about developing plugins.

Plugin structure

• settings.gradle

  Gradle project settings.

• plugins/nf-hello

  The plugin implementation base directory.

• plugins/nf-hello/build.gradle

  Plugin Gradle build file. Project dependencies should be added here.

• plugins/nf-hello/src/resources/META-INF/MANIFEST.MF

  Manifest file defining the plugin attributes e.g. name, version, etc. The attribute Plugin-Class declares the plugin main class. This class should extend the base class nextflow.plugin.BasePlugin e.g. nextflow.hello.HelloPlugin.

• plugins/nf-hello/src/resources/META-INF/extensions.idx

  This file declares one or more extension classes provided by the plugin. Each line should contain the fully qualified name of a Java class that implements the org.pf4j.ExtensionPoint interface (or a sub-interface).

• plugins/nf-hello/src/main

  The plugin implementation sources.

• plugins/nf-hello/src/test

  The plugin unit tests.

Plugin classes

• HelloConfig: shows how to handle options from the Nextflow configuration

• HelloExtension: shows how to create custom channel factories, operators, and fuctions that can be included into pipeline scripts

• HelloFactory and HelloObserver: shows how to react to workflow events with custom behavior

• HelloPlugin: the plugin entry point

Unit testing

To run your unit tests, run the following command in the project root directory (ie. where the file settings.gradle is located):

./gradlew check

Testing and debugging

To build and test the plugin during development, configure a local Nextflow build with the following steps:

1. Clone the Nextflow repository in your computer into a sibling directory:

   git clone --depth 1 https://github.com/nextflow-io/nextflow ../nextflow

2. Configure the plugin build to use the local Nextflow code:

   echo "includeBuild('../nextflow')" >> settings.gradle

   (Make sure to not add it more than once!)

3. Compile the plugin alongside the Nextflow code:

   make assemble

4. Run Nextflow with the plugin, using ./launch.sh as a drop-in replacement for the nextflow command, and adding the option -plugins nf-hello to load the plugin:

   ./launch.sh run nextflow-io/hello -plugins nf-hello

Testing without Nextflow build

The plugin can be tested without using a local Nextflow build using the following steps:

1. Build the plugin: make buildPlugins
2. Copy build/plugins/<your-plugin> to $HOME/.nextflow/plugins
3. Create a pipeline that uses your plugin and run it: nextflow run ./my-pipeline-script.nf

Package, upload, and publish

The project should be hosted in a GitHub repository whose name matches the name of the plugin, that is the name of the directory in the plugins folder (e.g. nf-hello).

Follow these steps to package, upload and publish the plugin:

1. Create a file named gradle.properties in the project root containing the following attributes (this file should not be committed to Git):

   • github_organization: the GitHub organisation where the plugin repository is hosted.
   • github_username: The GitHub username granting access to the plugin repository.
   • github_access_token: The GitHub access token required to upload and commit changes to the plugin repository.
   • github_commit_email: The email address associated with your GitHub account.

2. Use the following command to package and create a release for your plugin on GitHub:

   ./gradlew :plugins:nf-hello:upload

3. Create a pull request against nextflow-io/plugins to make the plugin accessible to Nextflow.

nf-hello's Issues

create a channel extension point

in order to have an example of how to extend ChannelFactory create some ChannelExtensionPoint with use DataflowWriteChannel and DataflowReadChannel

Update plugin to nextflow 22.08.0-edge

The plugin should be updated to make it compatible with APIs introduced by Nextflow 22.08.0-edge

Update plugin build template using latest best practice

The plugin build template should follow the same template used in the nextflow core plugins .

See nextflow-io/nextflow@00dd143

Add `make install` rule for easier testing

The current process for building and testing a plugin requires you to build Nextflow locally, include it in the plugin build, and use ./launch.sh to test the plugin with Nextflow in dev mode. This process is cumbersome and prone to errors,

An alternative which I first tried in the nf-prov plugin and is also being added to nf-validation here is to build and install the plugin directly to the plugins directory. While it may overwrite an existing plugin installation, it allows you to test the plugin without having to build Nextflow. It also avoids certain limitations of Nextflow dev mode (mainly that other non-core plugins cannot be used).

Cookie cutter template for Nextflow plugins

Create a cookie cutter template based on nf-hello, with a command-line tool that allows the user to enter things like plugin name, target nextflow version, etc. Should make it easier to start a new plugin instead of starting from nf-hello.

Tagging @ewels because I will probably ask you for advice on this at some point

CI integration test

We should add a basic integration test to the CI that simply runs a nextflow pipeline with the plugin enabled.

For nf-hello it would probably just be a local pipeline script showing the customer channel factory, operator, etc, but different plugins will have different use cases. The important thing is to create an example here and then encourage plugin developers to do the same for their plugins (and we would do the same for nf-prov and nf-sqldb).

This will give people better confidence that plugin releases actually work with some version of Nextflow.

Nextflow `23.05.0-edge` and later don't work with nf-hello

When I try to build the nf-hello plugin against Nextflow 23.05.0-edge or later, I get this error:

Could not determine the dependencies of task ':plugins:nf-hello:test'.
> Could not resolve all task dependencies for configuration ':plugins:nf-hello:testRuntimeClasspath'.
   > Could not resolve io.nextflow:nextflow:22.10.0.
     Required by:
         project :plugins:nf-hello
      > No matching variant of project :nextflow:nextflow was found. The consumer was configured to find a runtime of a library compatible with Java 17, packaged as a jar, preferably optimized for standard JVMs, and its dependencies declared externally but:
          - Variant 'apiElements' capability io.nextflow:nextflow:23.05.0-edge declares a library compatible with Java 11, packaged as a jar, and its dependencies declared externally:
              - Incompatible because this component declares an API of a component and the consumer needed a runtime of a component
              - Other compatible attribute:
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
          - Variant 'runtimeElements' capability io.nextflow:nextflow:23.05.0-edge declares a runtime of a library compatible with Java 11, packaged as a jar, and its dependencies declared externally:
              - Other compatible attribute:
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
          - Variant 'testFixturesApiElements' capability io.nextflow:nextflow-test-fixtures:23.05.0-edge declares a library, packaged as a jar, and its dependencies declared externally:
              - Incompatible because this component declares an API of a component compatible with Java 19 and the consumer needed a runtime of a component compatible with Java 17
              - Other compatible attribute:
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
          - Variant 'testFixturesRuntimeElements' capability io.nextflow:nextflow-test-fixtures:23.05.0-edge declares a runtime of a library, packaged as a jar, and its dependencies declared externally:
              - Incompatible because this component declares a component compatible with Java 19 and the consumer needed a component compatible with Java 17
              - Other compatible attribute:
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)

To be precise, this error happens during ./gradlew check. It builds fine, but then the plugin simply doesn't work at runtime:

// main.nf
include { reverse } from 'plugin/nf-hello'
Channel.reverse('hi!') 

$ make compile
$ ./launch.sh run main.nf -plugins nf-hello
N E X T F L O W  ~  version 23.05.0-edge
Launching `main.nf` [pensive_becquerel] DSL2 - revision: 0124bb0ec6
Plugin 'nf-hello' does not implement any extension point

I think the root cause is this commit in which the Nextflow build was upgraded from Java 17 to Java 19. I understand that it's supposed to be backwards compatible but something isn't working with the plugins. Maybe we need to update the plugin build script as well?

make compile : Cannot locate tasks that match ':nextflow:exportClasspath' as project 'nextflow' not found in root project 'nf-hello'.

Hi all,
there is a "compile" target in the Makefile but it raises the following error:

$ make compile
./gradlew :nextflow:exportClasspath compileGroovy

FAILURE: Build failed with an exception.

* What went wrong:
Cannot locate tasks that match ':nextflow:exportClasspath' as project 'nextflow' not found in root project 'nf-hello'.

Task 'exportClasspath' not found in root project 'nf-hello' and its subprojects.

As per the README

4. Run Nextflow with the plugin, using ./launch.sh as a drop-in replacement for the nextflow command, and adding the option -plugins nf-hello to load the plugin:

./launch.sh run nextflow-io/hello -plugins nf-hello

I get the error

Missing '.launch.classpath' file -- create it by running: ./gradlew exportClasspath

and if I try to run the command suggested, I get another error

./gradlew exportClasspath

FAILURE: Build failed with an exception.

* What went wrong:
Task 'exportClasspath' not found in root project 'nf-hello' and its subprojects.

Ideas?

$ gradle --version

------------------------------------------------------------
Gradle 7.6
------------------------------------------------------------


$ java --version
openjdk 11.0.22 2024-01-16 LTS
OpenJDK Runtime Environment Corretto-11.0.22.7.1 (build 11.0.22+7-LTS)
OpenJDK 64-Bit Server VM Corretto-11.0.22.7.1 (build 11.0.22+7-LTS, mixed mode)

running against Nextflow version 567f5334ff0d225cd95dc7b5239c4dbd64696a9a

./gradlew check : java.util.ServiceConfigurationError: Locale provider adapter "CLDR"cannot be instantiated.

cross-posted on slack: https://nextflow.slack.com/archives/C02T98BKE2K/p1709894909502359

I tried to compile the hello plugin but I got the following error/stacktrace:

$ ./gradlew check

> Task :plugins:nf-hello:compileGroovy FAILED
java.util.ServiceConfigurationError: Locale provider adapter "CLDR"cannot be instantiated.
        at java.base/sun.util.locale.provider.LocaleProviderAdapter.forType(LocaleProviderAdapter.java:200)
  (...)
        ... 12 more
Caused by: java.lang.IllegalArgumentException: newLimit > capacity: (55346993 > 27910144)
        at java.base/java.nio.Buffer.createLimitException(Buffer.java:398)
        at java.base/java.nio.Buffer.limit(Buffer.java:372)
        at java.base/java.nio.ByteBuffer.limit(ByteBuffer.java:1539)
        at java.base/java.nio.MappedByteBuffer.limit(MappedByteBuffer.java:331)
        at java.base/java.nio.MappedByteBuffer.limit(MappedByteBuffer.java:73)
        at java.base/jdk.internal.jimage.BasicImageReader.slice(BasicImageReader.java:213)
        at java.base/jdk.internal.jimage.BasicImageReader.readBuffer(BasicImageReader.java:372)
        at java.base/jdk.internal.jimage.BasicImageReader.getResourceBuffer(BasicImageReader.java:438)
        at java.base/jdk.internal.jimage.ImageReader.getResourceBuffer(ImageReader.java:199)
        at java.base/jdk.internal.module.SystemModuleFinders$SystemModuleReader.read(SystemModuleFinders.java:485)
        at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:809)
        at java.base/jdk.internal.loader.BuiltinClassLoader.findClassInModuleOrNull(BuiltinClassLoader.java:741)
        at java.base/jdk.internal.loader.BuiltinClassLoader.findClass(BuiltinClassLoader.java:621)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:633)
        at java.base/java.lang.Class.forName(Class.java:584)
        at java.base/java.lang.Class.forName(Class.java:559)
        at java.base/java.util.ServiceLoader.loadProvider(ServiceLoader.java:859)
        at java.base/java.util.ServiceLoader$ModuleServicesLookupIterator.hasNext(ServiceLoader.java:1084)
        at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1309)
        at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1393)
        at java.base/sun.util.cldr.CLDRLocaleProviderAdapter.lambda$new$0(CLDRLocaleProviderAdapter.java:84)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:569)
        at java.base/sun.util.cldr.CLDRLocaleProviderAdapter.<init>(CLDRLocaleProviderAdapter.java:83)
        at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:67)
        ... 15 more

FAILURE: Build failed with an exception.

I googled the error message but I don't really understand what I saw. My local javac version is avac 17.0.3.1.
Any idea ?

Super class does not exist

The file HelloExtension.groovy extends PluginExtensionPoint, which no longer exists in Nextflow.