elyz081 / nf-scil

SCIL Nextflow Toolbox

Welcome to nf-scil ! A Nextflow modules and workflows repository for neuroimaging maintained by the SCIL team. The primary focus of the library is to provide pre-built processes and processing sequences for diffusion Magnetic Resonance Imaging, optimized for Nextflow DLS2, based on open-source technologies and made easily available to pipeline's developers through the nf-core framework.

• Using modules from nf-scil
• Developing in nf-scil
  • Dependencies
  • Developer installation
  • Loading the project's environment
  • Working with VS Code
    • Manual configuration of the VS Code project :
    • Configuration via the devcontainer :
  • Contributing to nf-scil
    • Adding a new module to nf-scil
      • Generate the template
      • Edit the template
        • Editing the main
        • Editing the metadata
        • Editing the test cases
        • Editing the tests configuration
      • Run the tests to generate the test metadata file
      • Last safety test
      • Submit your PR
    • Defining processes optional parameters
    • Test data infrastructure
      • Using the .test_data directory
      • Using Scilpy Fetcher
  • Running tests

Using modules from nf-scil

To import modules from nf-scil, you first need to install nf-core on your system (can be done simply using pip install nf-core). Once done, nf-scil modules are imported using this command :

nf-core modules \
  --git-remote https://github.com/scilus/nf-scil.git \
  install <category>/<tool>

where you input the <tool> you want to import from the desired <category>. To get a list of the available modules, run :

nf-core modules \
  --git-remote https://github.com/scilus/nf-scil.git \
  list remote

Developing in nf-scil

Dependencies

• Python ≥ 3.8, < 3.13
• Java Runtime ≥ 11, ≤ 17
  • On Ubuntu, install openjdk-jre-<version> packages
• Nextflow ≥ 21.04.3
• Node ≥ 14 and Prettier (seel below)

Important

Nextflow might not detect the right Java virtual machine by default, more so if multiple versions of the runtime are installed. If so, you need to set the environment variable JAVA_HOME to target the right one.

• Linux : look in /usr/lib/jvm for a folder named java-<version>-openjdk-<platform> and use it as JAVA_HOME.
• MacOS : if the Java jvm is the preferential one, use JAVA_HOME=$(/usr/libexec/java_home). Else, look into /Library/Java/JavaVirtualMachines for the folder with the correct runtime version (named jdk<inner version>_1<runtime version>.jdk) and use the following : /Library/Java/JavaVirtualMachines/dk<inner version>_1<runtime version>.jdk/Contents/Home.

Developer installation

The project uses poetry to manage python dependencies. To install it using pipx, run the following commands :

pip install pipx
pipx ensurepath
pipx install poetry

Note

If the second command above fails, pipx cannot be found in the path. Prepend the second command with $(which python) -m and rerun the whole block.

Warning

Poetry doesn't like when other python environments are activated around it. Make sure to deactivate any before calling poetry commands.

Once done, install the project with :

poetry install

Loading the project's environment

Important

Make sure no python environment is activated before running commands !

The project scripts and dependencies can be accessed using :

poetry shell

which will activate the project's python environment in the current shell.

Note

You will know the poetry environment is activated by looking at your shell. The input line should be prefixed by : (nf-scil-tools-py<version>), with <version> being the actual Python version used in the environment.

To exit the environment, simply enter the exit command in the shell.

Important

Do not use traditional deactivation (calling deactivate), since it does not relinquish the environment gracefully, making it so you won't be able to reactivate it without exiting the shell.

Working with VS Code

Manual configuration of the VS Code project :

First install the nf-scil-extensions package. You can find it easily on the extension marketplace.

Configuration via the devcontainer :

The devcontainer definition for the project contains all required dependencies and setup steps are automatically executed. Open the cloned repository in VS Code and click on the arrow box in the lower left corner, to get a prompt to Reopen in container. The procedure will start a docker build, wait for a few minutes and enjoy your fully configured development environment.

• Available in the container :

  • nf-scil, nf-core all accessible through the terminal, which is configured to load the poetry environment in shells automatically
  • git, git-lfs, github-cli
  • curl, wget, apt-get
  • nextflow, docker, tmux

• Available in the VS Code IDE through extensions :

  • Docker images and containers management
  • Python and C++ linting, building and debugging tools
  • Github Pull Requests management
  • Github flavored markdown previewing

Contributing to nf-scil

If you want to propose a new module to the repository, follow the guidelines in the module creation documentation. we follow standards closely aligned with nf-core, with some exceptions on process atomicity and how test data is handled. Modules that don't abide to them won't be accepted and PR containing them will be closed automatically.

Running tests

Tests are run through nf-core, using the command :

nf-core modules \
  --git-remote https://github.com/scilus/nf-scil.git \
  test <category/tool>

The tool can be omitted to run tests for all modules in a category.

Installing Prettier

To install Prettier for the project, you need to have node and npm installed on your system to at least version 14. On Ubuntu, you can do it using snap :

sudo snap install node --classic

However, if you cannot install snap, or have another OS, refer to the official documentation for the installation procedure.

Under the current configuration for the Development Container, for this project, we use the following procedure, considering ${NODE_MAJOR} is at least 14 for Prettier :

curl -fsSL https://deb.nodesource.com/setup_${NODE_MAJOR}.x | bash - &&\
apt-get install -y nodejs

npm install --save-dev --save-exact prettier

echo "function prettier() { npm exec prettier $@; }" >> ~/.bashrc