nn-classification module uses all available CPUs instead of the limit set by --threads about genomad

[apcamargo]

Hi @jamesck2

Which version are you using? Since version 1.5.1, the nn-classification module allows setting the number of threads.

[jamesck2]

Executing genomad --version shows that I have genomad 1.5.2. Strange. I'm going to try running the nn-classification module by itself and see if I get the same issue. I will report back.

[jamesck2]

The same issue happened when I ran nn-classification by itself. Here's the screen in case it helps (I terminated the run once I saw all cores being used again:

`genomad nn-classification --threads 30 /storage2/scratch/kosmopoulos/peatland_viruses/assemblies/CGEBMP337A_MEGAHITCOA_justAssemblies/BAr1A1B1C_final_assembly.fasta /storage2/scratch/kosmopoulos/peatland_viruses/genomad_out/BAr1A1B1C/
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Executing geNomad nn-classification (v1.5.2). This will classify the input sequences into chromosome, plasmid, or virus based on the nucleotide sequence. │
│ ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── │
│ Outputs: │
│ /storage2/scratch/kosmopoulos/peatland_viruses/genomad_out/BAr1A1B1C/BAr1A1B1C_final_assembly_nn_classification │
│ ├── BAr1A1B1C_final_assembly_nn_classification.json (execution parameters) │
│ ├── BAr1A1B1C_final_assembly_encoded_sequences (directory containing encoded sequence data) │
│ ├── BAr1A1B1C_final_assembly_nn_classification.tsv (contig classification: tabular format) │
│ ├── BAr1A1B1C_final_assembly_nn_classification.npz (contig classification: binary format) │
│ ├── BAr1A1B1C_final_assembly_encoded_proviruses (directory containing encoded sequence data) │
│ ├── BAr1A1B1C_final_assembly_provirus_nn_classification.tsv (provirus classification: tabular format) │
│ └── BAr1A1B1C_final_assembly_provirus_nn_classification.npz (provirus classification: binary format) │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
[08:30:44] Executing genomad nn-classification.
[08:30:46] Previous execution detected. Steps will be skipped unless their outputs are not found. Use the --restart option to force the execution of all the steps again.
[08:30:48] BAr1A1B1C_final_assembly_encoded_sequences was found. Skipping sequence encoding.
[08:30:48] BAr1A1B1C_final_assembly_encoded_proviruses was found. Skipping provirus encoding.
Classifying sequences ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.2% | 3:42:27

Aborted.
^CException ignored in: <module 'threading' from '/storage2/scratch/kosmopoulos/miniconda3/envs/genomad/lib/python3.11/threading.py'>
Traceback (most recent call last):
File "/storage2/scratch/kosmopoulos/miniconda3/envs/genomad/lib/python3.11/threading.py", line 1534, in _shutdown
def _shutdown():

KeyboardInterrupt:`

[apcamargo]

I'll investigate it. In my installations, --threads is working as expected (including in the end-to-end command). Can you check which version of TensorFlow you have installed? Maybe they changed something.

>>> import tensorflow
>>> tensorflow.__version__
'2.11.0'

[jamesck2]

>>> import tensorflow
2023-07-28 09:24:49.356172: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE4.1 SSE4.2 AVX AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
>>> tensorflow.__version__
'2.12.0'

Interesting... I'll downgrade to 2.11.0 and see what happens

[jamesck2]

Ok, I uninstalled genomad 1.5.2 and installed genomad 1.5.1 This did not cause any problems at the nn-classification step and no extra cores were used. I'm not sure what the differences are between 1.5.1 and 1.5.2 that caused this problem for me, but at least I can use nn-classification now 😃

[apcamargo]

That's strange. This shouldn't have been affected. Did the TensorFlow version change?

[jamesck2]

It changed to 2.12.1

[apcamargo]

That shouldn't be the issue, then. I just tested a new 1.5.2 installation and couldn't reproduce the issue. Maybe it was something in your environment?

Since you solved the problem on your end and I can't reproduce it for now, I'll close the issue. I'll re-open it if I someone else reports the same problem.