test 'gt repfind small' fails for some seeds about genometools

[Garonenur]

we should look out for tests that fail on travis with return code 2, these are probable assertions that fail.

In both cases if compiled with assert=no valgrind produces some errors, see http://pastebin.com/bugfGLCZ (link is only good for 1 month)

[satta]

Well, the original error seems to be:

gt_assert(queryrep != NULL && pos < queryrep->length), function gt_mmsearch_accessquery, file src/match/esa-mmsearch.c, line 55

for a pos value of 2 and the following values of queryrep:

 {sequence = 0xd3dc70 "\001\001\376\003\003\003\376\003\376", 
  reversecopy = false, encseq = 0x0, 
  readmode = GT_READMODE_FORWARD, startpos = 0, 
  length = 2}

when the seed is 71207745 and a pos value of 4 and queryrep content of

{sequence = 0xd3dc70 "\003\376\001\001\003\003\003\376\376\001\376\001\003\003\376\376\376\003\376\376", 
 reversecopy = false, encseq = 0x0, 
 readmode = GT_READMODE_FORWARD, startpos = 0, length = 4}

when the seed is 471897537

Obviously when assertions are disabled, the program does not terminate here and continues to read into undefined memory, causing the valgrind issues mentioned above. Hence I conclude these problems are related.

Who will address this?

[stefan-kurtz]

Has this error already been in the version before the introduction of
GtUword and friends?
If so, I can address it. Otherwise, someone else has to do it.

Bye,

Stefan

On 09/17/2013 19:44 , Sascha Steinbiss wrote:

Well, the original error seems to be:

|gt_assert(queryrep != NULL && pos < queryrep->length);
|

for a |pos| value of 2 and the following values of queryrep:

| {sequence = 0xd3dc70 "\001\001\376\003\003\003\376\003\376",
reversecopy = false, encseq = 0x0,
readmode = GT_READMODE_FORWARD, startpos = 0,
length = 2}
|

when the seed is 71207745 and a |pos| value of 4 and queryrep content of

|{sequence = 0xd3dc70 "\003\376\001\001\003\003\003\376\376\001\376\001\003\003\376\376\376\003\376\376",
reversecopy = false, encseq = 0x0,
readmode = GT_READMODE_FORWARD, startpos = 0, length = 4}
|

when the seed is 471897537

Obviously when assertions are disabled, the program does not terminate
here and continues to read into undefined memory, causing the valgrind
issues mentioned above. Hence I conclude these problems are related.

Who will address this?

—
Reply to this email directly or view it on GitHub
#128 (comment).

Prof. Dr. Stefan Kurtz
Zentrum fuer Bioinformatik
Universitaet Hamburg
Bundesstrasse 43
20146 Hamburg
Germany

Email: [email protected]
URL: http://www.zbh.uni-hamburg.de/kurtz
Phone: [phone]
FAX: [phone]

[satta]

Both v1.5.1 stable and commit b7a6b59 (prior to the Windows-related merges) are not affected, implying that this bug must indeed have been introduced later. I will bisect to the exact commit introducing the regression and report back.

[satta]

Commit 6fb4ea5 seems to be the culprit. I traced the source of the problem to the insertion of the line

#include "core/yarandom.h"

into src/match/test-maxpairs.c. If I comment out this line in the current master, everything still compiles fine and the repfind errors for both faulty seeds finish without problems.

I can only speculate that the use of different implementations of random() (the yarandom and the native one) leads to the problem, and that we should make sure that the same one is used everywhere. Any comments @gordon? What was the original reason for including these lines, given that everything compiled fine before?

[gordon]

Very interesting bug! The reason for the inclusion of yarandom.h was that the random() function from stdlib.h is POSIX and is not available on Windows (try running scripts/build_windows.sh without including yarandom.h). Furthermore, the results from the POSIX random() are not reproducible (they are not effected by our seed) and system dependent. Therefore, it makes sense you use our own (reproducible and system independent) random() implementation everywhere.

Furthermore, the bug was there before the inclusion of yarandom.h, it was just very rarely triggered. If one uses the the value 1144606435 instead of the random() call the bug can be reproduced (1144606435 is a valid return value of the POSIX random()).

[satta]

I see. @stefan-kurtz has agreed to look into this.

[stefan-kurtz]

This problem is solved now. The problem was due to the fact, that
sometimes the sampled query sequences contained special characters
which are not allowed in a query. The special character in a query are
now replaced by random characters. The fixing commits (and a few other
commits) can be found in my sk-branch on github. Hope this helps.

On 09/17/2013 19:44 , Sascha Steinbiss wrote:

Well, the original error seems to be:

|gt_assert(queryrep != NULL && pos < queryrep->length);
|

for a |pos| value of 2 and the following values of queryrep:

| {sequence = 0xd3dc70 "\001\001\376\003\003\003\376\003\376",
reversecopy = false, encseq = 0x0,
readmode = GT_READMODE_FORWARD, startpos = 0,
length = 2}
|

when the seed is 71207745 and a |pos| value of 4 and queryrep content of

|{sequence = 0xd3dc70 "\003\376\001\001\003\003\003\376\376\001\376\001\003\003\376\376\376\003\376\376",
reversecopy = false, encseq = 0x0,
readmode = GT_READMODE_FORWARD, startpos = 0, length = 4}
|

when the seed is 471897537

Obviously when assertions are disabled, the program does not terminate
here and continues to read into undefined memory, causing the valgrind
issues mentioned above. Hence I conclude these problems are related.

Who will address this?

—
Reply to this email directly or view it on GitHub
#128 (comment).

Prof. Dr. Stefan Kurtz
Zentrum fuer Bioinformatik
Universitaet Hamburg
Bundesstrasse 43
20146 Hamburg
Germany

Email: [email protected]
URL: http://www.zbh.uni-hamburg.de/kurtz
Phone: [phone]
FAX: [phone]

[gordon]

Thanks Stefan! Could you send a pull request? Just click on "Pull Request" on https://github.com/stefan-kurtz/genometools/tree/sk

This allows a maintainer to merge it into the master once the tests pass.

[stefan-kurtz]

Done.

On 09/20/2013 16:33 , Gordon Gremme wrote:

Thanks Stefan! Could you send a pull request? Just click on "Pull
Request" on https://github.com/stefan-kurtz/genometools/tree/sk

This allows a maintainer to merge it into the master once the tests pass.

—
Reply to this email directly or view it on GitHub
#128 (comment).

Prof. Dr. Stefan Kurtz
Zentrum fuer Bioinformatik
Universitaet Hamburg
Bundesstrasse 43
20146 Hamburg
Germany

Email: [email protected]
URL: http://www.zbh.uni-hamburg.de/kurtz
Phone: [phone]
FAX: [phone]

[satta]

Hmm, I do not see it... are you sure you sent it?

[stefan-kurtz]

You are right. The stuff does not appear, when I check the github-website
and my genometools branch sk.

What I did:

---

roma [1002] git branch
clustering
hashprefixes
kaerkkaeinen
massmatch
master
mergemappedstr
sa2shulen

• sk
  sk-20130626-d8079eb1e6b24975a832c5c93ac47c5bcd04ff8c
  roma [1003] git log
  roma [1004] git push
  Password for 'https://[email protected]':
  Everything up-to-date

  What did I do wrong?

Bye, Stefan

On 09/20/2013 18:55 , Sascha Steinbiss wrote:

Hmm, I do not see it... are you sure you sent it?

—
Reply to this email directly or view it on GitHub
#128 (comment).

Prof. Dr. Stefan Kurtz
Zentrum fuer Bioinformatik
Universitaet Hamburg
Bundesstrasse 43
20146 Hamburg
Germany

Email: [email protected]
URL: http://www.zbh.uni-hamburg.de/kurtz
Phone: [phone]
FAX: [phone]

[satta]

I see your changes in your sk branch on github. The commits are there on https://github.com/stefan-kurtz/genometools/commits/sk. All that is missing is the pull request. Logged into GitHub, go to https://github.com/stefan-kurtz/genometools/tree/sk and click on "Pull request" there:

Then just confirm sending the pull request with the green button on the next page.