Issue with running docker image about deepvariant

[pichuan]

Hi @Kwondo87 ,
I think this might be similar to #640

If I remember correctly, the issue was that Singularity uses the Numpy from your computer (instead of the docker image), and in your setting it's incompatible.

Please look at the discussion in #640 and see if it's helpful for you.

[Kwondo87]

Thank you for the reply, @pichuan

I took a look at the discussion. It looks like the issue was not resolved for deepvariant 1.6.0. Also, the numpy version on my computer doesn't matter, as the numpy version inside the singularity image is being used as you've shown in discussion #640.

1.4.0 image worked on setting but I would like to use the latest version of singularity image if possible.

Thanks!

[pichuan]

@Kwondo87 I wasn't able to reproduce the Singularity issue on my side, so I'll need more information from you.

I also noticed that your original post said you used https://github.com/google/deepvariant/blob/r0.10/docs/deepvariant-quick-start.md . Can you use https://github.com/google/deepvariant/blob/r1.6/docs/deepvariant-quick-start.md instead in case there's differences? (Although, I just quickly checked and they don't look different on the Singularity section)

@Kwondo87 given that I'm not able to reproduce the error, can you give me your step-by-step, including how you get the image, how you run singularity, etc. What your OS is, etc. Maybe check the numpy versions in anywhere that might be relevant and just print them?

I'll be able to help more when I can reproduce on my side. (Something similar to what I did in #745 (comment) will be really helpful. Although, in that case I also wasn't able to reproduce the issue)

[pichuan]

Btw, I have confirmed on a CPU machine , I was able to run the Quick Start with Singularity without any issues.

pichuan@pichuan-cpu:~$ singularity --version
singularity version 3.7.0
pichuan@pichuan-cpu:~$ uname -a
Linux pichuan-cpu 5.15.0-1047-gcp #55~20.04.1-Ubuntu SMP Wed Nov 15 11:38:25 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

And a few different version checks:

pichuan@pichuan-cpu:~$ singularity exec deepvariant_1.6.0.sif pip freeze | grep numpy
numpy==1.24.4
pichuan@pichuan-cpu:~$ singularity shell deepvariant_1.6.0.sif
Singularity> python -c "import numpy; print(numpy.__version__)"
1.24.4
Singularity> python3 -c "import numpy; print(numpy.__version__)"
1.24.4
pichuan@pichuan-cpu:~$ singularity exec --bind "${INPUT_DIR}":"/input","${OUTPUT_DIR}":"/output",/usr/lib/locale/:/usr/lib/locale/ deepvariant_1.6.0.sif python -c "import numpy; print(numpy.__version__)"
1.24.4

[Kwondo87]

Thanks for the quick reply, @pichuan

First of all, I followed the instructions in https://github.com/google/deepvariant/blob/r1.6/docs/deepvariant-quick-start.md, but it gives the same error.

1. I got the image from "docker://google/deepvariant:1.6.0" I tried both: locally download the image and use the image without downloading. But they gave the same error.
2. Follow the instructions in the link above to run the program.

Here is the script that I used

#!/bin/bash

BIN_VERSION="1.6.0"

INPUT_DIR="${PWD}/quickstart-testdata"
DATA_HTTP_DIR="https://storage.googleapis.com/deepvariant/quickstart-testdata"

mkdir -p ${INPUT_DIR}
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/NA12878_S1.chr20.10_10p1mb.bam
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/NA12878_S1.chr20.10_10p1mb.bam.bai
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/test_nist.b37_chr20_100kbp_at_10mb.bed
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/test_nist.b37_chr20_100kbp_at_10mb.vcf.gz
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/test_nist.b37_chr20_100kbp_at_10mb.vcf.gz.tbi
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/ucsc.hg19.chr20.unittest.fasta
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/ucsc.hg19.chr20.unittest.fasta.fai
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/ucsc.hg19.chr20.unittest.fasta.gz
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/ucsc.hg19.chr20.unittest.fasta.gz.fai
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/ucsc.hg19.chr20.unittest.fasta.gz.gzi

OUTPUT_DIR="${PWD}/quickstart-output"
mkdir -p "${OUTPUT_DIR}"

# Pull the image.
singularity pull docker://google/deepvariant:"${BIN_VERSION}"

# Run DeepVariant.
singularity run -B /usr/lib/locale/:/usr/lib/locale/ \
  docker://google/deepvariant:"${BIN_VERSION}" \
  /opt/deepvariant/bin/run_deepvariant \
  --model_type=WGS \
  --ref="${INPUT_DIR}"/ucsc.hg19.chr20.unittest.fasta \
  --reads="${INPUT_DIR}"/NA12878_S1.chr20.10_10p1mb.bam \
  --regions "chr20:10,000,000-10,010,000" \
  --output_vcf="${OUTPUT_DIR}"/output.vcf.gz \
  --output_gvcf="${OUTPUT_DIR}"/output.g.vcf.gz \
  --intermediate_results_dir "${OUTPUT_DIR}/intermediate_results_dir" \
  --num_shards=1

And I did same things as you to check the versions.

singularity --version
singularity version 3.8.5-2.el7

uname -a
Linux sumner098 3.10.0-1062.1.2.el7.x86_64 #1 SMP Mon Sep 30 14:19:46 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

singularity exec deepvariant_1.6.0.sif pip freeze | grep numpy
numpy==1.22.1

singularity shell -B /usr/lib/locale/:/usr/lib/locale/ docker://google/deepvariant:1.6.0
Singularity> python -c "import numpy; print(numpy.__version__)"
1.22.1
Singularity> python3 -c "import numpy; print(numpy.__version__)"
1.22.1

singularity exec --bind ${PWD}/quickstart-testdata/,${PWD}/quickstart-output/,/usr/lib/locale/:/usr/lib/locale/ deepvariant_1.6.0.sif python -c "import numpy; print(numpy.__version__)"
1.22.1

Thanks a lot!

[pichuan]

Hi @Kwondo87 ,
Can you try updating your numpy to 1.24.4 and see if it changes anything?

[Kwondo87]

Hi, @pichuan

I upgraded numpy from 1.22.1 to 1.24.4, and it worked!!
But I'm a bit confused as I upgraded the numpy in one of the containers using "pip install numpy --upgrade", and it worked in the other container. Does the container image use numpy from outside of the container?

Thanks a million anyway!

Singularity deepvariant_1.6.0.sif:> pip freeze | grep numpy
numpy==1.24.4

[pichuan]

Glad to hear that it worked.

I might have mentioned this before (maybe in the previous thread) - I'm not super familiar with Singularity myself, so I won't be able to help more on that. Given that your use case works, I'll close this issue!