pyBigWig using 30+ cores about pybigwig

[dpryan79]

30! It shouldn't ever use more than 1. Can you post an example where this happens?

[goodez]

30! It shouldn't ever use more than 1. Can you post an example where this happens?

Without posting a bunch of code I think I can describe my simple script:

First I load in a bed file of ~20,000 annotated genes. I then open up a single bigwig file:
bw=pyBigWig.open(args.bigwig)

Then I loop through each line of the bed file. For each entry in the bed, I evaluate the coverage of my bigwig. These are all of the functions I use in total:

               tss_covs=bw.values(chrom, tss_pos-n, tss_pos+n, numpy=True)
               pA_covs=bw.values(chrom, pA_pos+1-n, pA_pos+n+2, numpy=True)
               mid_covs=bw.values(chrom, tss_pos+n, pA_pos+1-n, numpy=True)[indexes.astype(int)]
                covs=np.concatenate((tss_covs,mid_covs,pA_covs))
                all_covs=np.vstack((all_covs,covs))

I am also using the modules numpy and argparse. Maybe numpy is causing the issue?

[goodez]

Ah I have just looked into numpy a bit more. I think the issue is there and I can solve it this way:

export MKL_NUM_THREADS=1
export NUMEXPR_NUM_THREADS=1
export OMP_NUM_THREADS=1

I am sorry for blaming your wonderful project! Thank you for such a quick response.

[dpryan79]

It's quite likely that numpy is causing this, particularly if you've installed it via conda. At the beginning of your script, try adding the following:

import os

# This should force numpy to run single threaded. See issue #697
# This module MUST be imported before numpy
# Note that these environment variables are internal to deepTools (they won't exist on the shell after the command completes)
if 'MKL_NUM_THREADS' not in os.environ:
    os.environ['MKL_NUM_THREADS'] = 'sequential'
if 'NUMEXPR_NUM_THREADS' not in os.environ:
    os.environ['NUMEXPR_NUM_THREADS'] = '1'
if 'OMP_NUM_THREADS' not in os.environ:
    os.environ['OMP_NUM_THREADS'] = '1'
if 'VECLIB_MAXIMUM_THREADS' not in os.environ:
    os.environ['VECLIB_MAXIMUM_THREADS'] = '1

I've placed that code at the very beginning of all the deepTools executables to prevent similar issues with numpy.

[goodez]

Thanks that's even better! Woo!