openomics / chrom-seek

An awesome set of epigenetic pipelines for bulk cfChip-seq, ChIP-seq, and ATAC-seq

chrom-seek 🔬

An awesome set of epigenetic pipelines

This is the home of the pipeline, chrom-seek. Its long-term goals: to accurately call and annotate peaks, to infer cell types in cell-free samples, and to boldly quantify diferential binding or accessibility like no pipeline before!

Overview

Welcome to chrom-seek! Before getting started, we highly recommend reading through chrom-seek's documentation.

The ./chrom-seek pipeline is composed of several interrelated sub-commands to set up and run the pipeline across different systems. Each of the available sub-commands performs different functions:

• chrom-seek run: Run the chrom-seek pipeline with your input files.
• chrom-seek unlock: Unlocks a previous runs output directory.
• chrom-seek install: Download reference files locally.
• chrom-seek cache: Cache remote resources locally, coming soon!

chrom-seek is an awesome set of pipelines designed specifically for cell-free ChIP-seq, bulk ChIP-seq, and bulk ATAC-seq sequencing data. It relies on technologies like Singularity¹ to maintain the highest level of reproducibility. The pipeline consists of a series of data processing and quality-control steps orchestrated by Snakemake², a flexible and scalable workflow management system, to submit jobs to a cluster.

The pipeline is compatible with data generated from Illumina short-read sequencing technologies. As input, it accepts a set of FastQ files and can be run locally on a compute instance or on-premise using a cluster. A user can define the method or mode of execution. The pipeline can submit jobs to a cluster using a job scheduler like SLURM (more coming soon!). A hybrid approach ensures the pipeline is accessible to all users.

Before getting started, we highly recommend reading through the usage section of each available sub-command.

For more information about issues or troubleshooting a problem, please check out our FAQ prior to opening an issue on Github.

Dependencies

Requires: singularity>=3.5 snakemake>=6.0

At the current moment, the pipeline uses a mixture of environment modules and docker images; however, this will be changing soon! In the very near future, the pipeline will only use docker images. With that being said, snakemake and singularity must be installed on the target system. Snakemake orchestrates the execution of each step in the pipeline. To guarantee the highest level of reproducibility, each step of the pipeline will rely on versioned images from DockerHub. Snakemake uses singularity to pull these images onto the local filesystem prior to job execution, and as so, snakemake and singularity will be the only two dependencies in the future.

Installation

Please clone this repository to your local filesystem using the following command:

# Clone Repository from Github
git clone https://github.com/OpenOmics/chrom-seek.git
# Change your working directory
cd chrom-seek/
# Add dependencies to $PATH
# Biowulf users should run
module load snakemake singularity
# Get usage information
./chrom-seek -h

Contribute

This site is a living document, created for and by members like you. chrom-seek is maintained by the members of OpenOmics and is improved by continuous feedback! We encourage you to contribute new content and make improvements to existing content via pull requests to our GitHub repository.

Cite

If you use this software, please cite our methods paper:

@BibText

@article {Jange202302003,
	author = {Moon Kyoo Jang and Tovah E Markowitz and Temesgen E Andargie and Zainab Apalara and Skyler Kuhn and Sean Agbor-Enoh},
	title = {Cell-free chromatin immunoprecipitation to detect molecular pathways in heart transplantation},
	volume = {6},
	number = {12},
	elocation-id = {e202302003},
	year = {2023},
	doi = {10.26508/lsa.202302003},
	publisher = {Life Science Alliance},
	abstract = {Existing monitoring approaches in heart transplantation lack the sensitivity to provide deep molecular assessments to guide management, or require endomyocardial biopsy, an invasive and blind procedure that lacks the precision to reliably obtain biopsy samples from diseased sites. This study examined plasma cell-free DNA chromatin immunoprecipitation sequencing (cfChIP-seq) as a noninvasive proxy to define molecular gene sets and sources of tissue injury in heart transplant patients. In healthy controls and in heart transplant patients, cfChIP-seq reliably detected housekeeping genes. cfChIP-seq identified differential gene signals of relevant immune and nonimmune molecular pathways that were predominantly down-regulated in immunosuppressed heart transplant patients compared with healthy controls. cfChIP-seq also identified cell-free DNA tissue sources. Compared with healthy controls, heart transplant patients demonstrated greater cell-free DNA from tissue types associated with heart transplant complications, including the heart, hematopoietic cells, lungs, liver, and vascular endothelium. cfChIP-seq may therefore be a reliable approach to profile dynamic assessments of molecular pathways and sources of tissue injury in heart transplant patients.},
	URL = {https://www.life-science-alliance.org/content/6/12/e202302003},
	eprint = {https://www.life-science-alliance.org/content/6/12/e202302003.full.pdf},
	journal = {Life Science Alliance}
}

@APA

Jang, M. K., Markowitz, T. E., Andargie, T. E., Apalara, Z., Kuhn, S., & Agbor-Enoh, S. (2023). Cell-free chromatin immunoprecipitation to detect molecular pathways in heart transplantation. Life Science Alliance, 6(12), e202302003. https://doi.org/10.26508/lsa.202302003 

References

<sup>1. Kurtzer GM, Sochat V, Bauer MW (2017). Singularity: Scientific containers for mobility of compute. PLoS ONE 12(5): e0177459.
2. Koster, J. and S. Rahmann (2018). "Snakemake-a scalable bioinformatics workflow engine." Bioinformatics 34(20): 3600.</sup>

chrom-seek's Issues

ppqt rule needs improved fix

Currently using this code as a fix for ppqt rule to run the chrom-seek pipeline, but need to find a more straightforward solution.

 if paired_end:
    extensionsPPQT={"sorted": "bam", "Q5DD":"bam"}
else:
    extensionsPPQT= {"sorted": "bam", "tagAlign": "gz", "Q5DD": "tagAlign.gz"}

Update behavior --peakcall parser and validation

About

Make the following changes to how the --peakcall PEAKCALL file is parsed and validated:

1. Update the names of the following columns:
   a. "ChIP" becomes "Sample"
   b. "Input" becomes "InputControl"
2. InputControl becomes an optional column.
   a. Only "Sample" and "Group" should be required
3. Documentation on the CLI and wiki should be updated to include the changes and made to encompass both ATAC and ChIP

Write script for new genome reference builds

ATAC Conditional Rules

For ATAC,

The following can be considered to probably be conditional when running the pipeline:

1. Between macs2 and Genrich- Genrich should be default, macs2 should be conditional
2. Between edgeR and deseq2 in the Diffbind rules - Deseq2 should be default, edgeR should be conditional.
3. Motif analysis (MEME/Homer? rules) could probably be conditional for ATAC.
4. Between genes, prot, protsec, protTSS - I think only prot needs to run by default. This should be a selection as the number of total jobs is ballooning in certain circumstances.

These are suggestions not demands,

Paul

Additional functionality for cfChIP

The new features to be added to the cfChIP assay include:

1. Run a DiffBind R script on all "ChIP" samples to:
   • Identify consensus peaks, produce bed file
   • Get normalized counts per sample
   • Get UMAP coordinates based upon this information
2. Annotate consensus peaks with uropa
3. Run Amos's specialized differential RMD
4. If only one group per sample and all samples have a group, create UMAP plot

add FRiP rule

Add Homer_motif rule

update DiffBind rule to work

Completed with 487945f and de0906b commits

test ChIP-seq assay with mm10 references

update UROPA rule to work for cfChIP assay: basic functional cfChIP pipeline

This is step completes basic functional pipeline for cfChIP.

Build docker image for pipeline

At the current moment, the pipeline is using environment modules. We need to build docker images for all the software dependencies of the pipeline.

Tasks

Beta Give feedback

No tasks being tracked yet.

test ChIP-seq assay with hg38 references

comment/document pipeline rules

Adjust pipeline to work with single-end sequencing data for ChIP

Empty test files

Hi,

I'm trying to test the pipeline using the test files but all files on https://github.com/OpenOmics/chrom-seek/tree/main/.tests are empty except the the peakcall.tsv and the contrasts.tsv. Can you have some working test files in the github?

Best

add genrich to ATAC assay

Missing R package due to singularity sharing filesystems from host

About

The following issue was discovered while running run_spp.R from ppqt:

The error indicates that the R package, caTools, was missing from the docker image. Initially, I could not reproduce this behavior, and from my side, it appeared that caTools was installed:

$ module purge
$ ml singularity
$ singularity exec -B $PWD /data/OpenOmics/SIFs/ppqt_v0.2.0.sif Rscript -e 'library(caTools); sessionInfo()'

R version 4.1.2 (2021-11-01)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 22.04.3 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.10.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.10.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] caTools_1.18.2

loaded via a namespace (and not attached):
[1] compiler_4.1.2 bitops_1.0-7  

After more debugging, it appeared that I could reproduce the error if I module load R before running singularity:

$ module purge
$ module load R
$ ml singularity
$ singularity exec -B $PWD /data/OpenOmics/SIFs/ppqt_v0.2.0.sif Rscript -e 'library(caTools); sessionInfo()'

Error in library(caTools) : there is no package called ‘caTools’
Execution halted

It appears the following environment variable, R_LIBS_SITE, is exported when R is module loaded into a user's $PATH. If that environment variable is set, that will update what .libPaths R internally uses to find R packages. This can cause issues with how the R installation within the docker image finds/loads packages.

The following can be used to fix the problem.

$ module purge
$ module load R
$ ml singularity
$ R_LIBS_SITE='' singularity exec -B $PWD /data/OpenOmics/SIFs/ppqt_v0.2.0.sif Rscript -e 'library(caTools); sessionInfo()'
...

However, it would be better to run singularity with the --containall, -C option. This would prevent this problem in the future and other issues that can occur from sharing filesystems and env variables with the host.

DiffBind block

DiffBind with blocking currently errors out. This causes both DiffBind block and regular DiffBind to fail within the pipeline. Error seems to be that the csv file being used does not contain the blocking information.

ATAC ppqt

For ATAC part,

ppqt can fail causing a lot of downstream issues which can lead to bigwigs not being made. If this not necessary for bigwig formation, this should be removed.

Paul

set up pipeline to work with RheMac10 references

Peak calling with samples lacking input controls

Sicer kills the dry-run when there are samples lacking input controls.

SICER2 does not support rheMac10 out of the box

SICER2 has information pertaining to genome chromosome lengths hard-coded in the following file:
https://github.com/zanglab/SICER2/blob/master/sicer/lib/GenomeData.py

To add support for a new reference genome, a user must do the following:

I emailed the author of SICER2 asking if it would be possible to implement any of the suggestions here. It is worth also noting, if one were to make these changes, one should also update the dtype set for chr. At the current moment, it is set to U6 or a unicode string of length 6, which may not be long enough in some edge-cases (could cause a string to get truncated and cause collisions).

For the time being, we will turn off sicer2 when the reference genome is rheMac10; however, we should explore other solutions if the author cannot implement this on their side:

• Fork and add functionality described in issue 23
• Fork, update sicer/lib/GenomeData.py with rheMac10 information and build docker image

split DiffBind and DiffBind block into two separate runs so Diffbind block errors don't make the DiffBind results disappear

add a few extra cfChIP rules

Add rerun triggers to run sub command

Add --rerun-trigger to run sub command

This adds a mechanism to control how snakemake decides what rules to re-run. By default, this option is set to the following: ['mtime', 'params', 'input', 'software-env', 'code'] which can cause upstream rules to re-run unexpectedly.

Add something to the docs stating if this occurs, then this should be set to --rerun-trigger mtime. This allows for the user to decide how/why rules are re-run.

test out bulk ChIP-seq assay part of the pipeline

Add FRIP heatmap plot

FRIP_plot.txt

Incorporate Subrata's code into the pipeline

add other peakcallers: macsBroad and sicer

Script to build new reference files

It would be awesome if we had a script to automate the process of building reference files for different/new organisms, something where we could give it a minimal set of inputs (such as a genomic fasta file, annotation, and an optional blacklist, etc.) and it would handle the rest.