error when trying to run hsp.py about picrust2

[gavinmdouglas]

Hi Igor,

My pleasure, thanks for trying it out!

I haven't seen this error before, but I think it may have something to do with your study sequence ids. Would you mind sending me your tree (and other input files if possible) to my email used for the google group?

Also, do all of the test commands complete successfully when you run pytest in the main picrust2 directory?

Thanks,

Gavin

[itiago]

Hi Gavin
all testt went well after installing, and even now, again. but stil the command does not.
I'm sending you the sequs.
One other question, is it imperative that I use more than one sample data or can I run picrust2 in one single data (despite the fact that until metagenome prediction part where the biom enters that is not relevant.

thanks for the help..
Best

(picrust2-dev) igor@ubuntu:~/picrust2-master$ pytest
============================= test session starts ==============================
platform linux -- Python 3.6.5, pytest-3.5.1, py-1.5.3, pluggy-0.6.0
rootdir: /home/igor/picrust2-master, inifile:
plugins: cov-2.5.1
collected 27 items

tests/test_hsp.py ....... [ 25%]
tests/test_metagenome_pipeline.py ...... [ 48%]
tests/test_place_seqs.py ...... [ 70%]
tests/test_run_minpath.py .. [ 77%]
tests/test_util.py .... [ 92%]
tests/test_workflow.py .. [100%]

========================== 27 passed in 15.72 seconds ==========================
(picrust2-dev) igor@ubuntu:~/picrust2-master$ hsp.py -i 16S -t /home/igor/Desktop/Alfaguara/Picrust2/alfaguara.tre -o /home/igor/Desktop/Alfaguara/Picrust2/16S_Alfaguara_predicted -p 10 -n

*** caught segfault ***
address 0x556a3, cause 'memory not mapped'

Traceback:
1: .Call(treeBuild, x)
2: FUN(X[[i]], ...)
3: lapply(STRING, .treeBuild)
4: read.tree(Args[1])
An irrecoverable exception occurred. R is aborting now ...
Error running this command:
Rscript /home/igor/picrust2-master/picrust2/Rscripts/castor_hsp.R /home/igor/Desktop/Alfaguara/Picrust2/alfaguara.tre /home/igor/picrust2-master/precalculated/prokaryotic/16S_counts_mean_round_var.txt mp TRUE FALSE FALSE 10 /tmp/wrmm6mzo /tmp/1_418els None None FALSE

[itiago]

What is you email on google?
sorry...

[gavinmdouglas]

Hi Igor,

I have received your files, thanks. The first thing I noticed was that your study sequences have very long names and seem to be already aligned - I'm not sure if this would have caused issues in the produced tree or not. I am re-running your data now after removing gap characters, but it's taking a while since there are so many (>60,000). How were these sequences clustered? In general your data analysis will likely be improved if you can filter out sequences that are extremely rare, which I think could make a difference for your data.

I'll get back to you when I have a chance to troubleshoot your data more.

Best,

Gavin

[itiago]

Hi Gavin
Thanks for that.
These are already clustered and filtered seqs in mothur. All singletons and doubletons were removed from the data set. But Yes I could more, say below 2%?

I'll wait for your analyses results,
Best
Igor

[gavinmdouglas]

Hi Igor,

I think I've made the required fix. The "read.tree" function from the ape R package was crashing when reading in your study tree. I have switched to instead read in the tree using "read_tree" from the castor R package. You can download the modified picrust2 version at this branch: https://github.com/picrust/picrust2/tree/read_tree

Please let me know if you run into any other problems and if this works for you!

Gavin

[itiago]

Hi Gavin So, what I did was download the branch that youve sent, instaled again. I runned the 16Ss command and it worked, the EC command gave an error: (picrust2-dev) igor@ubuntu:~/Desktop/Alfaguara/Picrust2$ hsp.py -i EC -t alfaguara.tre -o EC_predicted -p 10 Error in serialize(what, NULL, xdr = FALSE) : cannot allocate buffer Calls: mclapply -> lapply -> FUN -> sendMaster -> serialize Error in serialize(what, NULL, xdr = FALSE) : cannot allocate buffer Calls: mclapply -> lapply -> FUN -> sendMaster -> serialize Error in mcfork() : unable to fork, possible reason: Cannot allocate memory Calls: mclapply -> lapply -> FUN -> mcfork In addition: Warning message: In mclapply(trait_states_mapped, hsp_max_parsimony, tree = full_tree, : scheduled cores 7, 3, 10 encountered errors in user code, all values of the jobs will be affected Execution halted Warning message: system call failed: Cannot allocate memory Error running this command: Rscript /home/igor/picrust2-read_tree/picrust2/Rscripts/castor_hsp.R alfaguara.tre /home/igor/picrust2-read_tree/precalculated/prokaryotic/ec_counts_mean_round_var.txt mp FALSE FALSE FALSE 10 /tmp/gireyiut /tmp/6omtl6hr None None FALSE (picrust2-dev) igor@ubuntu:~/Desktop/Alfaguara/Picrust2$ thanks for any help Best

…

On Thu, May 17, 2018 at 7:30 PM, Gavin Douglas ***@***.***> wrote: Hi Igor, I think I've made the required fix. The "read.tree" function from the ape R package was crashing when reading in your study tree. I have switched to instead read in the tree using "read_tree" from the castor R package. You can download the modified picrust2 version at this branch: https://github.com/picrust/picrust2/tree/read_tree Please let me know if you run into any other problems and if this works for you! Gavin — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#11 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AQAOmSxvqWDfmKd-lMa5pqLrruoFbSO6ks5tzcGpgaJpZM4UDRB6> .

-- Igor Tiago Researcher Universidade de Coimbra Laboratório Microbiologia Edificio Patronato Rua da Matemática Nº 49 3000-276 Coimbra, Portugal

[gavinmdouglas]

Hi Igor,

How much memory do you have on your machine? If you use fewer processes the job will run slower, but take less memory. Also, does the job die right away or after a while. If it takes a while can you see the memory usage creep up if you watch the activity with the top command?

[itiago]

I have 24GB in a virtual machine. I will lower the threads, since it runs little while then goes down. Ill check with top. Thanks A sexta, 18/05/2018, 13:13, Gavin Douglas <[email protected]> escreveu:

…

Hi Igor, How much memory do you have on your machine? If you use fewer processes the job will run slower, but take less memory. Also, does the job die right away or after a while. If it takes a while can you see the memory usage creep up if you watch the activity with the top command? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#11 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AQAOmUgZNbVlxI5Qyh8aoXUfKf-CS1oJks5tzrr0gaJpZM4UDRB6> .

[itiago]

Hi Gavinn
I lowered the threads to 2 and still the same error:

(picrust2-dev) igor@ubuntu:/Desktop/Alfaguara/Picrust2$ hsp.py -i EC -t alfaguara.tre -o EC_predicted -p 2
Error in serialize(what, NULL, xdr = FALSE) : cannot allocate buffer
Calls: mclapply -> lapply -> FUN -> sendMaster -> serialize
Warning messages:
1: In mclapply(trait_states_mapped, hsp_max_parsimony, tree = full_tree, :
scheduled core 1 encountered error in user code, all values of the job will be affected
2: In mclapply(names(hsp_out_models), function(x) { :
all scheduled cores encountered errors in user code
Error in $<-.data.frame(*tmp*, sequence, value = c("M01028_125_000000000-AN36D_1_1111_16866_17721", :
replacement has 65410 rows, data has 0
Calls: $&lt;- -&gt; $&lt;-.data.frame
Execution halted
Error running this command:
Rscript /home/igor/picrust2-read_tree/picrust2/Rscripts/castor_hsp.R alfaguara.tre /home/igor/picrust2-read_tree/precalculated/prokaryotic/ec_counts_mean_round_var.txt mp FALSE FALSE FALSE 2 /tmp/4ljnwko_ /tmp/vmu2wrax None None FALSE
(picrust2-dev) igor@ubuntu:/Desktop/Alfaguara/Picrust2$

Any help?
Thanks

[gavinmdouglas]

Hi Igor,

I re-wrote how the HSP parallelization step works (see the PR here: #13) and you shouldn't have this problem anymore if you update to the latest version. Note that there is now a new option for hsp.py script (--chunk_size), which specified how many gene families should be read into a given R environment at a time (and that will be run by 1 processor). There are ~3000 gene families in the E.C. table so you could set this to be 300 if you wanted to run 10 processes for instance. Note that if you set this option to be 3000 and set 10 processes that it would still only run on 1 CPU since there would only be one chunk of input data.

Best,

Gavin

[itiago]

Hi Gavin I'm out of office for a meeting, but I thank you in advance for your update, I'm eager to test it and see if it works for my data. Next week I hope I have good news. Best Igor A terça, 22/05/2018, 16:57, Gavin Douglas <[email protected]> escreveu:

…

Hi Igor, I re-wrote how the HSP parallelization step works (see the PR here: #13 <#13>) and you shouldn't have this problem anymore if you update to the latest version. Note that there is now a new option for hsp.py script (--chunk_size), which specified how many gene families should be read into a given R environment at a time (and that will be run by 1 processor). There are ~3000 gene families in the E.C. table so you could set this to be 300 if you wanted to run 10 processes for instance. Note that if you set this option to be 3000 and set 10 processes that it would still only run on 1 CPU since there would only be one chunk of input data. Best, Gavin — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#11 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AQAOmZIcwsCn8_JJxff3sOwmWIdZoe8Rks5t1CdAgaJpZM4UDRB6> .

[itiago]

Hi Gavin so, the hsp.py worked fine (thanks for that), now the problem is the next step: Metagenome prediction. I don't understand this step, once the biom file from mothur only provides OTUs and relative abundance, and how it will relate to the later results obtained from picrust2. I did run the make.biom file from mothur and have a biom file (please remember that I only have one sample, so I did this with a biomfile for just one sample) then I run the command and got this error result: (picrust2-dev) igor@ubuntu:~/Desktop/Alfaguara/Picrust2$ metagenome_pipeline.py -i shared.0.03.biom -m 16S_predicted.tsv -f EC_predicted.tsv -p 4 -o metagenome_prediction Traceback (most recent call last): File "/home/igor/miniconda2/envs/picrust2-dev/bin/metagenome_pipeline.py", line 6, in <module> exec(compile(open(__file__).read(), __file__, 'exec')) File "/home/igor/picrust2/scripts/metagenome_pipeline.py", line 76, in <module> main() File "/home/igor/picrust2/scripts/metagenome_pipeline.py", line 64, in main output_normfile=True) File "/home/igor/picrust2/picrust2/metagenome_pipeline.py", line 46, in run_metagenome_pipeline pred_marker) File "/home/igor/picrust2/picrust2/util.py", line 246, in three_df_index_overlap_sort "input files.") ValueError: No sequence ids overlap between all three of the input files. Thanks for any help Best Igor

…

On Thu, May 24, 2018 at 2:25 PM, Igor Tiago ***@***.***> wrote: Hi Gavin I'm out of office for a meeting, but I thank you in advance for your update, I'm eager to test it and see if it works for my data. Next week I hope I have good news. Best Igor A terça, 22/05/2018, 16:57, Gavin Douglas ***@***.***> escreveu: > Hi Igor, > > I re-wrote how the HSP parallelization step works (see the PR here: #13 > <#13>) and you shouldn't have > this problem anymore if you update to the latest version. Note that there > is now a new option for hsp.py script (--chunk_size), which specified > how many gene families should be read into a given R environment at a time > (and that will be run by 1 processor). There are ~3000 gene families in the > E.C. table so you could set this to be 300 if you wanted to run 10 > processes for instance. Note that if you set this option to be 3000 and set > 10 processes that it would still only run on 1 CPU since there would only > be one chunk of input data. > > Best, > > Gavin > > — > You are receiving this because you modified the open/close state. > Reply to this email directly, view it on GitHub > <#11 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AQAOmZIcwsCn8_JJxff3sOwmWIdZoe8Rks5t1CdAgaJpZM4UDRB6> > . >

-- Igor Tiago Researcher Universidade de Coimbra Laboratório Microbiologia Edificio Patronato Rua da Matemática Nº 49 3000-276 Coimbra, Portugal

[gavinmdouglas]

Hi Igor,

I'm glad that the HSP step is runnable for you now. I'm going to close this issue now so that it will be easier to keep them straight in the future. I posted your new issue here: #16 where I'll try to address it!

Best,

Gavin