hildebra / sdm

Fast tool for sequence related tasks

sdm

Simple demultiplexing This project started as a simple sequence demultiplexer, because I needed a fast and memory efficient program for this. Since then (5 years ago) it has grown in scope and complexity and is by now part of most of my pipelines, as the initial filtering step, that is able to handle a variety of standard sequence inputs (fasta / fastq all versions / gz of these), detecting inputs and even handling corrupt files (till the point where they are corrupted). Thus this tool is a basic bioinformatic program, written in C++11 for fast and efficient functionality. It's main purposes are:

• parsing between DNA sequence formats (fasta / fastq)
• demultiplexing multi sample sequence files (based on DNA sequence or header keywords)
• removing sequencing / PCR primer from reads
• filtering DNA sequences based on a multitude of quality parameters
• fast extraction of a subset of DNA sequences, based on header names (also works for paired end reads)
• combining several inputs into one (filtered) sequence file

sdm's Issues

Can't compile on two different systems

Hi there,

I tried to compile on centos 7:

$ make
g++ -D__STDC_CONSTANT_MACROS -O3 -std=c++17 -lrt -I./  -c -o Demultipl.o Demultipl.cpp
In file included from gzstream.h:35,
                 from DNAconsts.h:92,
                 from InputStream.h:27,
                 from containers.h:22,
                 from IO.h:23,
                 from Demultipl.cpp:21:
./zlib.h:1420:9: error: ‘z_size_t’ does not name a type; did you mean ‘ssize_t’?
 1420 | ZEXTERN z_size_t ZEXPORT gzfread OF((voidp buf, z_size_t size, z_size_t nitems,
      |         ^~~~~~~~
      |         ssize_t
./zlib.h:1454:9: error: ‘z_size_t’ does not name a type; did you mean ‘ssize_t’?
 1454 | ZEXTERN z_size_t ZEXPORT gzfwrite OF((voidpc buf, z_size_t size,
      |         ^~~~~~~~
      |         ssize_t
In file included from ./zlib.h:34,
                 from gzstream.h:35,
                 from DNAconsts.h:92,
                 from InputStream.h:27,
                 from containers.h:22,
                 from IO.h:23,
                 from Demultipl.cpp:21:
./zlib.h:1708:37: error: ‘z_size_t’ has not been declared
 1708 |                                     z_size_t len));
      |                                     ^~~~~~~~
./zlib.h:1743:35: error: ‘z_size_t’ has not been declared
 1743 |                                   z_size_t len));
      |                                   ^~~~~~~~
make: *** [<builtin>: Demultipl.o] Error 1

and that didn't work so I tried on archlinux with the 5.12.15-arch1-1 kernel:

> make
g++ -D__STDC_CONSTANT_MACROS -O3 -std=c++17 -lrt -I./  -c -o containers.o containers.cpp
containers.cpp: In constructor ‘OutputStreamer::OutputStreamer(Filters*, OptContainer&, std::ios_base::openmode, std::shared_ptr<ReadSubset>, int, std::string, int)’:
containers.cpp:238:30: warning: passing NULL to non-pointer argument 1 of ‘std::vector<_Tp, _Alloc>::vector(std::vector<_Tp, _Alloc>::size_type, const allocator_type&) [with _Tp = ofbufstream*; _Alloc = std::allocator<ofbufstream*>; std::vector<_Tp, _Alloc>::size_type = long unsigned int; std::vector<_Tp, _Alloc>::allocator_type = std::allocator<ofbufstream*>]’ [-Wconversion-null]
  238 |                 of_merged_fq(NULL),
      |                              ^~~~
In file included from /usr/include/c++/11.1.0/vector:67,
                 from DNAconsts.h:59,
                 from InputStream.h:27,
                 from containers.h:22,
                 from containers.cpp:21:
/usr/include/c++/11.1.0/bits/stl_vector.h:510:24: note:   declared here
  510 |       vector(size_type __n, const allocator_type& __a = allocator_type())
      |              ~~~~~~~~~~^~~
g++ -D__STDC_CONSTANT_MACROS -O3 -std=c++17 -lrt -I./  -c -o Demultipl.o Demultipl.cpp
g++ -D__STDC_CONSTANT_MACROS -O3 -std=c++17 -lrt -I./  -c -o DNAconsts.o DNAconsts.cpp
g++ -D__STDC_CONSTANT_MACROS -O3 -std=c++17 -lrt -I./  -c -o FastxReader.o FastxReader.cpp
g++ -D__STDC_CONSTANT_MACROS -O3 -std=c++17 -lrt -I./  -c -o gzipstream.o gzipstream.cpp
g++ -D__STDC_CONSTANT_MACROS -O3 -std=c++17 -lrt -I./  -c -o InputStream.o InputStream.cpp
g++ -D__STDC_CONSTANT_MACROS -O3 -std=c++17 -lrt -I./  -c -o IO.o IO.cpp
g++ -D__STDC_CONSTANT_MACROS -O3 -std=c++17 -lrt -I./  -c -o ReadMerger.o ReadMerger.cpp
In file included from ReadMerger.h:12,
                 from ReadMerger.cpp:1:
include/robin_hood.h: In member function ‘size_t robin_hood::detail::Table<IsFlat, MaxLoadFactor100, Key, T, Hash, KeyEqual>::calcMaxNumElementsAllowed(size_t) const’:
include/robin_hood.h:2027:52: error: ‘numeric_limits’ is not a member of ‘std’
 2027 |         if (ROBIN_HOOD_LIKELY(maxElements <= (std::numeric_limits<size_t>::max)() / 100)) {
      |                                                    ^~~~~~~~~~~~~~
include/robin_hood.h:180:59: note: in definition of macro ‘ROBIN_HOOD_LIKELY’
  180 | #    define ROBIN_HOOD_LIKELY(condition) __builtin_expect(condition, 1)
      |                                                           ^~~~~~~~~
include/robin_hood.h:2027:73: error: expected primary-expression before ‘>’ token
 2027 |         if (ROBIN_HOOD_LIKELY(maxElements <= (std::numeric_limits<size_t>::max)() / 100)) {
      |                                                                         ^
include/robin_hood.h:180:59: note: in definition of macro ‘ROBIN_HOOD_LIKELY’
  180 | #    define ROBIN_HOOD_LIKELY(condition) __builtin_expect(condition, 1)
      |                                                           ^~~~~~~~~
include/robin_hood.h:2027:76: error: ‘::max’ has not been declared; did you mean ‘std::max’?
 2027 |         if (ROBIN_HOOD_LIKELY(maxElements <= (std::numeric_limits<size_t>::max)() / 100)) {
      |                                                                            ^~~
include/robin_hood.h:180:59: note: in definition of macro ‘ROBIN_HOOD_LIKELY’
  180 | #    define ROBIN_HOOD_LIKELY(condition) __builtin_expect(condition, 1)
      |                                                           ^~~~~~~~~
In file included from /usr/include/c++/11.1.0/algorithm:62,
                 from include/robin_hood.h:41,
                 from ReadMerger.h:12,
                 from ReadMerger.cpp:1:
/usr/include/c++/11.1.0/bits/stl_algo.h:3467:5: note: ‘std::max’ declared here
 3467 |     max(initializer_list<_Tp> __l, _Compare __comp)
      |     ^~~
make: *** [<builtin>: ReadMerger.o] Error 1

Any thoughts on how to get it compiled successfully?

Thanks!

include NovaSeq

Dear Falk,
could you at some point please provide a config file for NovaSeq data?