SE adapter trimmer about htstream

[msettles]

Adapters can only be found on the 3' end and can be 1bp of the adapter (not expected to identify) to the full adapter, plus bogus sequence afterward. You should actually know this and now I'm asking about it at your defense, so study up ;)

Look at the description/algorithm in sickle http://bioinformatics.ucdavis.edu/research-computing/software/ and other papers.

I would add it to the existing adapter-removal application (for SE input only), you can verify using the PE approach, so a R1 trimmed of adapter should produce the same results as R1/R2 trimmed of adapter using overlap method.

On 5' primers, I would do something different and maybe we can put in as a feature request later. I have an algorithm for this that I use in dbcAmplicons that can be used as a starting point, but this is not a priority.

We are doing alot of Quantseq libraries right now, which are SE reads, preprocessing includes 5' trim, adapter removal and polyA/T removal. At the moment, it would seem SE adapter removal is the only thing missing from HTStream

[samhunter]

A few thoughts,

1. Adapter trimming for SE reads would be nice, but alternatively, maybe tell the lab to size select if you plan to do SE reads? Or.. sequence shorter reads? Some exploratory testing I did with pattern-based adapter trimming suggested that it became very difficult to accurately remove them when the length got below a certain threshold. Probably a more sophisticated method could do better, but I don't think anyone has great results. Also, I have read recently that the new pseudo-mappers are much less negatively impacted by poly-A and adapter sequences, so this might be somewhat of a moot point (reads with full-length adapters may not map uniquely anyway).

2. This is probably a wet-lab problem.... Just tell the lab to size select your libraries better. What are they doing sequencing such short fragments? That short junk is probably mostly primer dimers anyway!!!

3. Weird, why do you have a 5' adapter on your Quantseq libraries? Are you doing the 3' mRNA-Seq FWD kit, or the REV kit? The FWD kit suggests only sequencing toward the poly-A end of the transcript, so your reads should only have poly-A and adapter on the 3' end correct? Or do you mean that you are trimming the reads on the 5' end for quality or something else?

[msettles]

My supportive/counter thoughts 1. I hate SE generally, and only receive these datasets when a) they didn’t first talk to me, I grumble and make due, usually ignoring adapter trimming and most preprocessing b) quantseq, or other 3’ biased technique like 10x. 2. You always get a certain portion of adapters doesn’t matter how well you cut, from a QA/QC perspective, I find the adapter trimming result most useful for this and 100% any effect on mapping is likely to be negligible and/or focused to some edge cases. Are you telling me you never have ‘outies’ anymore in any your datasets ;) 3. TRUE on the Quantseq FWD with is most what we use, I first followed the bbduk protocol for preprocessing as I get a better feel for these (actually I first didn’t do any preprocessing and it worked great) these which does 3 things 5’ trim 11bp (primer), then provide adapter AND polyA/T and trim 3’ end, then quality trim 3’ end. In a perfect work, if you trim the polyA the adapter should go do, is HTStream able to trim an internal polyA? Would be quick to take out the adapter, leave in the polyA/T see what happens. 4. Its more for completeness of HTStream, for good reasons or not, many still feel its needed and I like it for QA comparability there are already so few things to measure on SE to judge whether you have a comparable dataset, or diagnose outlier/issues, I don’t think there is any expectation that we can get EVERY basepair, that’s just impossible without trimming off a lot of non-adapter sequence, which is why the overlap method is SO superior and additional excuse to go PE. The result however from PE overlap can be used to quickly validate, parameterize the SE trimmer. From: Sam Hunter <[email protected]> Reply-To: ibest/HTStream <[email protected]> Date: Friday, August 4, 2017 at 9:38 AM To: ibest/HTStream <[email protected]> Cc: Matt Settles <[email protected]>, Assign <[email protected]> Subject: Re: [ibest/HTStream] SE adapter trimmer (#87) A few thoughts, 1. Adapter trimming for SE reads would be nice, but alternatively, maybe tell the lab to size select if you plan to do SE reads? Or.. sequence shorter reads? Some exploratory testing I did with pattern-based adapter trimming suggested that it became very difficult to accurately remove them when the length got below a certain threshold. Probably a more sophisticated method could do better, but I don't think anyone has great results. Also, I have read recently that the new pseudo-mappers are much less negatively impacted by poly-A and adapter sequences, so this might be somewhat of a moot point (reads with full-length adapters may not map uniquely anyway). 2. This is probably a wet-lab problem.... Just tell the lab to size select your libraries better. What are they doing sequencing such short fragments? That short junk is probably mostly primer dimers anyway!!! 3. Weird, why do you have a 5' adapter on your Quantseq libraries? Are you doing the 3' mRNA-Seq FWD kit, or the REV kit? The FWD kit suggests only sequencing toward the poly-A end of the transcript, so your reads should only have poly-A and adapter on the 3' end correct? Or do you mean that you are trimming the reads on the 5' end for quality or something else? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or mute the thread.

[samhunter]

Ahh, so trimming off the random primer for the Quantseq libs. I guess that sort of makes sense, but, the point is just to count transcripts, so it would be interesting to see if there are real impacts.

David and I chatted about this a bit. We like the idea of having a pattern-based 3' or 5' trimmer that could be used for adapters, primers, or whatever else. It will almost certainly be a different application from any of the current trimmers, and might not get implemented right away (due to other more pressing priorities), but would add a nice tool to the tool set.

[msettles]

Agreed, general 3’/5’ patterned trimmer would be good, though there are differences between 5’/3’ so when the time comes, would be good to talk about scope/types Matt From: Sam Hunter <[email protected]> Reply-To: ibest/HTStream <[email protected]> Date: Friday, August 4, 2017 at 10:08 AM To: ibest/HTStream <[email protected]> Cc: Matt Settles <[email protected]>, Assign <[email protected]> Subject: Re: [ibest/HTStream] SE adapter trimmer (#87) Ahh, so trimming off the random primer for the Quantseq libs. I guess that sort of makes sense, but, the point is just to count transcripts, so it would be interesting to see if there are real impacts. David and I chatted about this a bit. We like the idea of having a pattern-based 3' or 5' trimmer that could be used for adapters, primers, or whatever else. It will almost certainly be a different application from any of the current trimmers, and might not get implemented right away (due to other more pressing priorities), but would add a nice tool to the tool set. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or mute the thread.

[dstreett]

@msettles - Just an FYI, they are called LIns (Long INserts) and SIns (Short INserts) now. :) (Biggies and smalls was rejected due to some minor issues with trademarks.)

LIns and SIns better ways to describe Illumina overlaps.

[msettles]

Much better From: David Streett <[email protected]> Reply-To: ibest/HTStream <[email protected]> Date: Friday, August 4, 2017 at 10:14 AM To: ibest/HTStream <[email protected]> Cc: Matt Settles <[email protected]>, Mention <[email protected]> Subject: Re: [ibest/HTStream] SE adapter trimmer (#87) @msettles - Just an FYI, they are called LIns (Long INserts) and SIns (Short INserts) now. :) (Biggies and smalls was rejected due to some minor issues with trademarks.) LIns and SIns better ways to describe Illumina overlaps. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[msettles]

Well shouldn't it be overlaps instead of inserts? really as far as inserts go there are 3 classes, 'short' < length of read and contain adapter, 'medium' they are > length of reads but < length of 2x read, and 'long' > length of both reads (results in a pair)

[msettles]

addressed in #122