will-rowe / genome-sketching

online notebooks for a review of genome sketching

Genome Sketching

a review of sketching algorithms for genomic data

---

---

Overview

This repository accompanies my review of sketching algorithms for genomic data:

When The Levee Breaks: a practical guide to sketching algorithms for processing the flood of genomic data

The notebooks correspond to the sections in the paper:

• Section 3: Sketching algorithms
  • 3.1 Set similarity with MinHash
  • 3.2 Set membership with Bloom filter
• Section 4: Example workflows for genomics
  • Overview
  • Workflow 1: Sample QC
  • Workflow 2: Resistome profiling
  • Workflow 3: Outbreak surveillance
  • Workflow 4: Data mining

I've also included a notebook to cover some basic concepts (e.g. k-mer decomposition). This notebook is called Background and we save ourselves some time later by importing the functions from this notebook into the other notebooks (e.g. for reading sequence data files).

If you've not used a Jupyter notebook before, just hit "shift + enter" to run a cell. Here is a good guide to get you started with Jupyter.

Running the workbooks

To run the workbooks interactively from your browser, use Binder:

Running the workbooks locally

If you want to run the workbooks locally, you'll need conda and jupyter.

Once you have these, follow these steps:

1. clone the repo

git clone https://github.com/will-rowe/genome-sketching

cd genome-sketching

2. create the environment

conda env create --file=binder/environment.yml

source activate genome-sketching

3. run the notebook server

jupyter notebook --notebook-dir=./notebooks

See also

As well as the notebooks, this repository also contains a curated list of bioinformatics tools that use sketching. Please let me know if tools need adding, or file a pull request!

Notes

I'd like to thank Titus Brown and Luiz Irber, who not only got me onto Binder, but also inspired me to review the current sketching methods for genomics (thanks to their excellent sourmash documentation and examples of MinHash for genomics).

Thanks to Francesco Mosconi for showing how to use Travis CI for build testing the notebooks.

genome-sketching's Issues

Notebook 3.3. "Element frequency with Count-Min sketch" is empty

Thanks a ton for putting this repo together. I am working through the notebooks on Binder and noticed notebook 3.3 only has a one markdown cell. There doesn't appear to be any code. I'm not sure if this is just my binder instance or if the notebook is empty.