Development of Cloud Compute Images for Microbial Ecology
Cloud computing offers many advantages for processing metagenomics data:
- No need to purchase and maintain expensive local compute infrastructure. This is especially important for small groups with limited budgets.
- No scheduler. No queue. Saves costs and time while debugging pipelines and code.
- Compute images for cloud community can be a community resource, saving time installing and updating software and databases.
- Reproducibility. Others can replicate your compute environment exactly by re-using your image.
- Infinite scalability enabling you to address bigger questions.
- Rapid access to NCBI data, but this only applies to Google Cloud and Amazon Web Services.
- (done) Create a custom arch linux image on gcloud. Check out this wiki page.
- (done) Install programs and databases. See wiki - Check out this wiki page.
- (done) Share images with microbial ecology community. Check out this wiki page.
- (done) Test installed programs.
You can use the python script "cloud_bio_installs.py" to install a set of programs I find useful. Most installations will likely work out of the box. The script depends on the python module "wget". If you just want to install a few programs, you can also just have a look at the shell commands in the script and copy/run those that you need. To run the script:
python -m pip install --upgrade wget python path/to/cloud_bio_installs.py --root [path_to_where_you_want_the_programs_to_be - default: /bio/bin]
Before you attempt to run the installed programs:
source [path_to_where_you_installed_the_programs]/profile
And before running python programs:
source [path_to_where_you_want_the_programs_to_be]/python-env/bin/activate
You will also need databases. To install those:
if [ ! -d /bio/databases ] then sudo mkdir /bio/databases sudo chmod -R a+wX /bio/databases fi
cd /bio/databases
- mkdir metaamp
- cd metaamp
- wget https://mothur.s3.us-east-2.amazonaws.com/wiki/silva.nr_v138.1.tgz
- tar -xf silva.nr_v138.1.tgz
- rm silva.nr_v138.1.tgz
- ln -s silva.nr_v138_1.align silva.nr.fasta
- ln -s silva.nr_v138_1.tax silva.nr.tax
- ln -s /bio/databases/metaamp /bio/bin/ribosomal/metaamp/bin/database
- cd ..
Metaamp needs code edits to run. Best to do manually:
- In file get_diversity.pl, comment out the line $projDir =~ s//bin//;
- In file global.pl, comment out the line $projDir =~ s//bin//;
- getOTUTaxonTable.relabund.pl, change line 96 to if(exists $table{"$prefix_$otuid"}->{$sample} && $totals{$sample}>0){
- mkdir phyloflash
- cd phyloflash
- source /bio/bin/profile
- phyloFlash_makedb.pl --remote
- cd ..
- mkdir gtdbtk
- cd gtdbtk
- wget https://data.gtdb.ecogenomic.org/releases/latest/auxillary_files/gtdbtk_data.tar.gz
- tar -xf gtdbtk_data.tar.gz
- rm gtdbtk_data.tar.gz
- cd ..
- mkdir checkm
- cd checkm
- wget "https://data.ace.uq.edu.au/public/CheckM_databases/checkm_data_2015_01_16.tar.gz"
- tar -xf checkm_data_2015_01_16.tar.gz
- rm checkm_data_2015_01_16.tar.gz
- cd ..
- source /bio/bin/python-env/bin/activate
- checkm data setRoot /bio/databases/checkm
- deactivate
- cd /bio/databases
- wget https://osf.io/v46sc/download
- tar -xf download
- mv db virsorter
- source /bio/bin/python-env/bin/activate
- virsorter config --init-source --db-dir=./virsorter/
- deactivate
-
source /bio/bin/antismash-env/bin/activate
-
download-antismash-databases
-
deactivate
-
mkdir /bio/databases/antismash
-
cp -rf /bio/bin/antismash-env/lib/python3.10/site-packages/antismash/databases/* /bio/databases/antismash
-
rm -rf /bio/bin/antismash-env/lib/python3.10/site-packages/antismash/databases
-
ln -sf /bio/databases/antismash /bio/bin/antismash-env/lib/python3.10/site-packages/antismash/databases
-
To enable running antismash with python-env activated, create a file "antismash" in /bio/bin with the following content:
#!/bin/sh
CURRENT_VIRTUAL_ENV=$VIRTUAL_ENV
source /bio/bin/antismash-env/bin/activate
antismash "$@"
deactivate
source $CURRENT_VIRTUAL_ENV/bin/activate
To configure repeatmasker:
- source /bio/bin/profile if not already sourced
- cd /bio/bin/repeatMasker
- ./configure (accept all default options, set search engine to rmblast)
To complete installation:
- cd /bio/bin/BMGE/src
- echo Main-Class: BMGE > MANIFEST.MF
- jar -cmvf MANIFEST.MF BMGE.jar BMGE.class bmge/*.class
- rm MANIFEST.MF BMGE.class bmge/*.class