nextflow-io / nf-co2footprint

[WIP] A Nextflow plugin to estimate the CO2 footprint of pipeline runs.

nf-co2footprint plugin [WIP]

A Nextflow plugin to estimate the CO₂ footprint of pipeline runs.

📚 Docs 👉🏻 https://nextflow-io.github.io/nf-co2footprint

Introduction

The nf-co2footprint plugin estimates the energy consumption for each pipeline task based on the Nextflow resource usage metrics and information about the power consumption of the underlying compute system. The carbon intensity of the energy production is then used to estimate the respective CO₂ emission.

The calculation is based on the carbon footprint computation method developed in the Green Algorithms project: www.green-algorithms.org

Green Algorithms: Quantifying the Carbon Footprint of Computation.

Lannelongue, L., Grealey, J., Inouye, M.,

Adv. Sci. 2021, 2100707. https://doi.org/10.1002/advs.202100707

The nf-co2footprint plugin generates a detailed TXT carbon footprint report containing the energy consumption, the estimated CO₂ emission and other relevant metrics for each task. Additionally, an HTML report is generated with information about the carbon footprint of the whole pipeline run and containing plots showing, for instance, an overview of the CO₂ emissions for the different processes.

Quick Start

Declare the plugin in your Nextflow pipeline configuration file:

plugins {
  id '[email protected]'
}

This is all that is needed. Run your pipeline with the usual command.

You can find more information on plugins in the Nextflow documentation.

Note

To test the plugin prior to its first release, refer to the contributing documentation.

Credits

The nf-co2footprint plugin is mainly developed and maintained by Sabrina Krakau and Júlia Mir-Pedrol at QBiC.

We thank the following people for their extensive assistance in the development of this pipeline:

• Phil Ewels
• Paolo Di Tommaso

nf-co2footprint's Issues

Inclusion of the carbon footprint of cached processes

This is to discuss how to present the carbon footprint of cached processes.

Simple example: 3 processes [P1] [P2] [P3]

• First run of the pipeline, all three are run
• Second run of the pipeline, cached [P1] is used, [P2] and [P3] are run

When presenting the carbon footprint of run (2), we can either:

• Take the carbon footprint of [P1] for run (1) and add the carbon footprints of [P2] and [P3] from run (2).
• Or only add the carbon footprints of [P2] and [P3] and ignore the impact of [P1].

Option 1 gives a better estimate of the total carbon footprint of the pipeline if we were to run it again from start to finish on new data let's say. But option 2 gives a more accurate estimate of the true carbon footprint of running step (2). And if adding run (1) + run (2), option 2 should be used (otherwise the footprint of [P1] would get double counted even though it was only run once).

It seems to depend a lot on what users want to do with this information, so perhaps best to give both information in the report so that users can decide what to do?

Include CPU TDP data from Green algorithms

Include CPU TDP data from Green algorithms

Add missing metrics to the task-specific txt output file

Add missing metrics to the task-specific txt output file, e.g. cpu_model. Which other metrics are missing?

task_id in report does not match task_id in txt output

the task_id in report does not always match task_id in txt output, at least the reported CO2 values do not match

Report all used plugin parameters

Currently, not all plugin parameters are given out, including the ones that are used for the actual computation but are not task specific, such as CI and PUE.
They should be listed in the HTML and TXT summary output.

Restructure code

Some parts of the code became a bit chaotic. Restructure code when main functionality is ready.

Add parameter to disable use of cpu_model

Add parameter to disable use of cpu_model and use default power draw or provided CPU core power draw value. Add parameter for custom power draw value.

Convert report plots to one with two y-axes

Convert report CO2e and energy consumption plots to one plot with two y-axes (on the left side).
The plots are identical, just with different units.

Handle numeric types consistently

• Handle numeric types: double, Double, float, Float, BigDecimal consistently
• change type for values for which float would be sufficient, such as cpu_usage, powerdraw (adjust List, maybe when anyway re-structuring)

Use scientific notation for equivalence values in html report

Use scientific notation for equivalence values in html report to display also smaller values.

Include CI data from Green Algorithms project

Include location-specific CI data from Green Algorithms project
https://github.com/GreenAlgorithms/green-algorithms-tool/tree/master/data

and potentially adjust processing.

Handle summary txt output

Add the summary txt output to the docs and think about how to make this nicer.

Map CPU model names from LINUX file to the `TDP_CPU` file

This is a tricky one and more long term. As it is now, it is likely that the CPU model name returned by LINUX doesn't match exactly the model name in the TDP table.

One option could be to try and find the standardised model names returned my LINUX and have a mapping table. Or some sophisticated regex to map CPU model names. Open topic!

Sorting in report table does not account for units

The sorting in the report table does currently not account for units, e.g. 998 mg is considered > 9g.