opengene / defq

Please switch to https://github.com/OpenGene/defastq

defq

Ultra-fast Multi-threaded FASTQ Demultiplexing

introduction

Use either index1 or index2 to demultiplex a single FASTQ file to multiple FASTQ files. This tool is developed in C++ with multi-threading supported.

simple usage

# read1 and read2 are precessed separately
# -o specifies the output folder, and current working directory is used by default
# -f specifies the filename suffix so that R1 / R2 can have different file name
# -s specifies a samplesheet to defq how to demux
defq -i in.R1.fq.gz -o demux_out_dir -s samplesheet.csv -f .R1.fq.gz
defq -i in.R2.fq.gz -o demux_out_dir -s samplesheet.csv -f .R2.fq.gz

Where samplesheet.csv is 3-column (filename, index1, index2) CSV file. You can use index1 for all samples, or use index2 for all samples, but you cannot use both index1 and index2. If the indexes have different length, defq will match the longer index first, and then match the shorter ones if the read is still not matched. A sample sheet example:

#filename, #index1, #index2
file1, ATTCAGAA,
file2, ATTACT,
file3, GAGATTCC,
file4, CGCTCATT,

You will then have following output:

demux_out_dir/
├── Undetermined.R1.fq.gz
├── Undetermined.R2.fq.gz
├── file1.R1.fq.gz
├── file1.R2.fq.gz
├── file2.R1.fq.gz
├── file2.R2.fq.gz
├── file3.R1.fq.gz
├── file3.R2.fq.gz
├── file4.R1.fq.gz
└── file4.R2.fq.gz

get defq

download binary

This binary is only for Linux systems: http://opengene.org/defq/defq

# this binary was compiled on CentOS, and tested on CentOS/Ubuntu
wget http://opengene.org/defq/defq
chmod a+x ./defq

or compile from source

# get source (you can also use browser to download from master or releases)
git clone https://github.com/OpenGene/defq.git

# build
cd defq
make

# Install
sudo make install

gzip compression

When the output file name is ended with .gz, the output will be compressed with gzip.

• If you specify file name suffix by -f or --suffix1, add .gz into the suffix.
• If you don't specify file name suffix by -f or --suffix1, add .gz into the file name column in samplesheet.csv.

all options

usage: defq --in1=string --sample_sheet=string [options] ... 
options:
  -i, --in1             input file name (string)
  -s, --sample_sheet    a CSV file contains three columns (filename, index1, index2) (string)
  -o, --out_folder      output folder, default is current working directory (string [=.])
  -f, --suffix1         the suffix to be appended to the output file name given in sample sheet, default is none (string [=])
  -u, --undetermined    the file name of undetermined data, default is Undetermined (string [=Undetermined])
  -z, --compression     compression level for gzip output (1 ~ 9). 1 is fastest, 9 is smallest, default is 2. (int [=2])
  -?, --help            print this message

defq's Issues

cannot demultiplex

Hi, I don't understand what is wrong ...

I have a fastq file with the following sequences. (My barcode is under bracket):

@MN00591:192:000H332YK:1:11102:20930:2477 1:N:0:1
GCCCCCAAAATCAGCGAAATGGGCGCCATTGGTGATGACACCAGGCATCAGAATAAATGTTGTCGGCGAGATATTTGTCGGTCCCAGACTTTCGCTGAACGGCCACGGAGATAGGGCTAGGAGTAACCAGAATGGAGAACGC  [ATATCATTAACAGAG]  GGTACGCTCATCAGAATTAACC
+
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF

My barcode file contains the barcode :

#filename,#index1,#index2
A11,ATATCATTAACAGAG,  
H10,ATTAATAGTGGTATG,

But the following command keep creating 0 bytes files.

defq -i file.fastq -o demux_out_dir -s index.csv -f .fastq -z 1

Enhancement Memory Control

When I extract the fq.gz from size 30G data(fq.gz), the memory will be large, was killed by the system.

defq.sh: line 2: 95033 Killed /Bioinfo/tools/defq --in1 Undetermined_S0_R1_001.fastq.gz -s /Task/OUT/index2.sheet -o /Task/OUT/P5index -f _R1.fastq.gz

15024 fastq 20 0 70.4g 70g 1420 S 2368.2 55.7 456:49.89 defq

deadlock may happen when writing is too slow

What is the difference between defq and defastq software

I downloaded the defastq software through this link：
http://opengene.org/defastq/defastq

What is the difference between defq and defastq software?