Is txt input eligible? about sigprofilerclusters

[zyh4482]

I found your wiki here. The issue is solved.

[zyh4482]

After testing SigProfilerClusters with txt input, I got an error log as below :
Full command line:

sigSim.SigProfilerSimulator("tcga", "/home/tomas/project/sigprofiler/tcga", "GRCh38", contexts = ['288'], chrom_based=True, simulations=100)
hp.analysis("tcga", "GRCh38", "96", ["288"], "/home/tomas/project/sigprofiler/tcga", analysis="all", sortSims=True, subClassify=True, includedVAFs=False, includedCCFs=False, correction=True, calculateIMD=True, max_cpu=8, TCGA=False, sanger=False)

Error log:

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/tomas/anaconda3/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/tomas/anaconda3/lib/python3.8/site-packages/SigProfilerClusters/hotspot.py", line 797, in calculateSampleIMDs
    y2s[sample], bincenters2s[sample], q_values[sample], interval_lines[sample], orig_mutations_samps[sample], avg_simCounts[sample], std_simCounts[sample], imds[sample], lower_CIs[sample], upper_CIs[sample], lower_CIs_refined[sample], upper_CIs_refined[sample], avg_bin_counts_samp[sample] = first_run(overall_distances_all, distances_orig_all_samps, distances_orig_all, vcf_path_clust, vcf_path_nonClust, sample, original, sim_count, project, genome, clustering_vaf, correctionData, correction, regions, imds_corrected, windowSize, chromLengths)
  File "/home/tomas/anaconda3/lib/python3.8/site-packages/SigProfilerClusters/hotspot.py", line 417, in first_run
    distance_cut, q_val, avgSim, stdevSim, upper_CI_refined, lower_CI_refined = refineIMD (distances, distances_orig, y2, avg_bin_counts, interval_line, bincenters2[interval_line], bincenters2[interval_line+1], CI, lower_CI, upper_CI, sigValue)
IndexError: index 1 is out of bounds for axis 0 with size 1
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "workflow.py", line 7, in <module>
    hp.analysis("tcga", "GRCh38", "96", ["288"], "/home/tomas/project/sigprofiler/tcga", analysis="all", sortSims=True, subClassify=True, includedVAFs=False, includedCCFs=False, correction=True, calculateIMD=True, max_cpu=8, TCGA=False, sanger=False)
  File "/home/tomas/anaconda3/lib/python3.8/site-packages/SigProfilerClusters/SigProfilerClusters.py", line 669, in analysis
    regions, imds = hotspot.hotSpotAnalysis(project, genome, contexts, simContext, ref_dir, windowSize, processors, plotIMDfigure, exome, chromLengths, binsDensity, original, signature, percentage, firstRun, clustering_vaf, calculateIMD, chrom_based, correction)
  File "/home/tomas/anaconda3/lib/python3.8/site-packages/SigProfilerClusters/hotspot.py", line 1059, in hotSpotAnalysis
    r.get()
  File "/home/tomas/anaconda3/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
IndexError: index 1 is out of bounds for axis 0 with size 1

The output folder does not contain clustered, nonclustered folders.

The input txt exactly follows the structure of your example text.

How could I deal with it? Thank you.

[ebergstr]

Hi,

Thank you for the comment, and apologies for the delay! Can you please try to rerun by simply adding a backslash to your project path? For example: "/home/tomas/project/sigprofiler/tcga/"

We are adding this automatic path check in our next update to the code, but this should provide a quick fix. Please let us know if this solves your issue.

Best,
Erik

[zyh4482]

According to my latest error log created by SigProfilerCluster at 2022/05/09:

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/tomas/anaconda3/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/tomas/anaconda3/lib/python3.8/site-packages/SigProfilerClusters/hotspot.py", line 797, in calculateSampleIMDs
    y2s[sample], bincenters2s[sample], q_values[sample], interval_lines[sample], orig_mutations_samps[sample], avg_simCounts[sample], std_simCounts[sample], imds[sample], lower_CIs[sample], upper_CIs[sample], lower_CIs_refined[sample], upper_CIs_refined[sample], avg_bin_counts_samp[sample] = first_run(overall_distances_all, distances_orig_all_samps, distances_orig_all, vcf_path_clust, vcf_path_nonClust, sample, original, sim_count, project, genome, clustering_vaf, correctionData, correction, regions, imds_corrected, windowSize, chromLengths)
  File "/home/tomas/anaconda3/lib/python3.8/site-packages/SigProfilerClusters/hotspot.py", line 417, in first_run
    distance_cut, q_val, avgSim, stdevSim, upper_CI_refined, lower_CI_refined = refineIMD (distances, distances_orig, y2, avg_bin_counts, interval_line, bincenters2[interval_line], bincenters2[interval_line+1], CI, lower_CI, upper_CI, sigValue)
IndexError: index 1 is out of bounds for axis 0 with size 1
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "workflow.py", line 7, in <module>
    hp.analysis("tcga", "GRCh38", "96", ["288"], "/home/tomas/project/sigprofiler/tcga/", analysis="all", sortSims=True, subClassify=True, includedVAFs=False, includedCCFs=False, correction=True, calculateIMD=True, max_cpu=8, TCGA=False, sanger=False, standardVC=False)
  File "/home/tomas/anaconda3/lib/python3.8/site-packages/SigProfilerClusters/SigProfilerClusters.py", line 669, in analysis
    regions, imds = hotspot.hotSpotAnalysis(project, genome, contexts, simContext, ref_dir, windowSize, processors, plotIMDfigure, exome, chromLengths, binsDensity, original, signature, percentage, firstRun, clustering_vaf, calculateIMD, chrom_based, correction)
  File "/home/tomas/anaconda3/lib/python3.8/site-packages/SigProfilerClusters/hotspot.py", line 1059, in hotSpotAnalysis
    r.get()
  File "/home/tomas/anaconda3/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
IndexError: index 1 is out of bounds for axis 0 with size 1

I think I should have added the backslash to my path. I'll try it again asap and update this issue later.

Here is an example of my input data. Anything wrong?

Project	Sample	ID	Genome	mut_type	chrom	pos_start	pos_end	ref	alt	Type
ACC	TCGA-P6-A5OH-01A	chr1_27410171_G_A	GRCh38	SNV	1	27410171	27410171	G	A	SOMATIC
ACC	TCGA-P6-A5OH-01A	chr1_39455094_T_A	GRCh38	SNV	1	39455094	39455094	T	A	SOMATIC
ACC	TCGA-P6-A5OH-01A	chr1_43669220_G_A	GRCh38	SNV	1	43669220	43669220	G	A	SOMATIC
ACC	TCGA-P6-A5OH-01A	chr1_61088433_T_A	GRCh38	SNV	1	61088433	61088433	T	A	SOMATIC
ACC	TCGA-P6-A5OH-01A	chr1_61088434_C_T	GRCh38	SNV	1	61088434	61088434	C	T	SOMATIC

[zyh4482]

Update:

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/tomas/anaconda3/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/tomas/anaconda3/lib/python3.8/site-packages/SigProfilerClusters/hotspot.py", line 797, in calculateSampleIMDs
    y2s[sample], bincenters2s[sample], q_values[sample], interval_lines[sample], orig_mutations_samps[sample], avg_simCounts[sample], std_simCounts[sample], imds[sample], lower_CIs[sample], upper_CIs[sample], lower_CIs_refined[sample], upper_CIs_refined[sample], avg_bin_counts_samp[sample] = first_run(overall_distances_all, distances_orig_all_samps, distances_orig_all, vcf_path_clust, vcf_path_nonClust, sample, original, sim_count, project, genome, clustering_vaf, correctionData, correction, regions, imds_corrected, windowSize, chromLengths)
  File "/home/tomas/anaconda3/lib/python3.8/site-packages/SigProfilerClusters/hotspot.py", line 417, in first_run
    distance_cut, q_val, avgSim, stdevSim, upper_CI_refined, lower_CI_refined = refineIMD (distances, distances_orig, y2, avg_bin_counts, interval_line, bincenters2[interval_line], bincenters2[interval_line+1], CI, lower_CI, upper_CI, sigValue)
IndexError: index 1 is out of bounds for axis 0 with size 1
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "workflow.py", line 7, in <module>
    hp.analysis("tcga", "GRCh38", "96", ["288"], "/home/tomas/project/sigprofiler/tcga/", analysis="all", sortSims=True, subClassify=True, includedVAFs=False, includedCCFs=False, correction=True, calculateIMD=True, max_cpu=8, TCGA=False, sanger=False, standardVC=False)
  File "/home/tomas/anaconda3/lib/python3.8/site-packages/SigProfilerClusters/SigProfilerClusters.py", line 669, in analysis
    regions, imds = hotspot.hotSpotAnalysis(project, genome, contexts, simContext, ref_dir, windowSize, processors, plotIMDfigure, exome, chromLengths, binsDensity, original, signature, percentage, firstRun, clustering_vaf, calculateIMD, chrom_based, correction)
  File "/home/tomas/anaconda3/lib/python3.8/site-packages/SigProfilerClusters/hotspot.py", line 1059, in hotSpotAnalysis
    r.get()
  File "/home/tomas/anaconda3/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
IndexError: index 1 is out of bounds for axis 0 with size 1

I'll print my workflow.py here:

from SigProfilerMatrixGenerator import install as genInstall
from SigProfilerSimulator import SigProfilerSimulator as sigSim
from SigProfilerClusters import SigProfilerClusters as hp

#genInstall.install('GRCh38', rsync=False, bash=True)
sigSim.SigProfilerSimulator("tcga", "/home/tomas/project/sigprofiler/tcga/", "GRCh38", contexts = ['288'], chrom_based=True, simulations=100)
hp.analysis("tcga", "GRCh38", "96", ["288"], "/home/tomas/project/sigprofiler/tcga/", analysis="all", sortSims=True, subClassify=True, includedVAFs=False, includedCCFs=False, correction=True, calculateIMD=True, max_cpu=8, TCGA=False, sanger=False, standardVC=False)

Could you please help me with it? Thanks.

[MousumyCSE]

Hi,
Sorry for the late response. I have tried to run with your example input, it has run successfully at my end. Can you please try to rerun the same code with max_cpu=2? If that does not work, please try to run without using the max_cpu parameter.

Please let us know if that resolve your problem.

Best,
Mousumy

[zyh4482]

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/tomas/anaconda3/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/tomas/anaconda3/lib/python3.8/site-packages/SigProfilerClusters/hotspot.py", line 797, in calculateSampleIMDs
    y2s[sample], bincenters2s[sample], q_values[sample], interval_lines[sample], orig_mutations_samps[sample], avg_simCounts[sample], std_simCounts[sample], imds[sample], lower_CIs[sample], upper_CIs[sample], lower_CIs_refined[sample], upper_CIs_refined[sample], avg_bin_counts_samp[sample] = first_run(overall_distances_all, distances_orig_all_samps, distances_orig_all, vcf_path_clust, vcf_path_nonClust, sample, original, sim_count, project, genome, clustering_vaf, correctionData, correction, regions, imds_corrected, windowSize, chromLengths)
  File "/home/tomas/anaconda3/lib/python3.8/site-packages/SigProfilerClusters/hotspot.py", line 417, in first_run
    distance_cut, q_val, avgSim, stdevSim, upper_CI_refined, lower_CI_refined = refineIMD (distances, distances_orig, y2, avg_bin_counts, interval_line, bincenters2[interval_line], bincenters2[interval_line+1], CI, lower_CI, upper_CI, sigValue)
IndexError: index 1 is out of bounds for axis 0 with size 1
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "workflow.py", line 7, in <module>
    hp.analysis("tcga", "GRCh38", "96", ["288"], "/home/tomas/project/sigprofiler/tcga/", analysis="all", sortSims=True, subClassify=True, includedVAFs=False, includedCCFs=False, correction=True, calculateIMD=True, TCGA=False, sanger=False, standardVC=False)
  File "/home/tomas/anaconda3/lib/python3.8/site-packages/SigProfilerClusters/SigProfilerClusters.py", line 669, in analysis
    regions, imds = hotspot.hotSpotAnalysis(project, genome, contexts, simContext, ref_dir, windowSize, processors, plotIMDfigure, exome, chromLengths, binsDensity, original, signature, percentage, firstRun, clustering_vaf, calculateIMD, chrom_based, correction)
  File "/home/tomas/anaconda3/lib/python3.8/site-packages/SigProfilerClusters/hotspot.py", line 1059, in hotSpotAnalysis
    r.get()
  File "/home/tomas/anaconda3/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
IndexError: index 1 is out of bounds for axis 0 with size 1

Still the same issue.

[zyh4482]

I found the input data from BLCA cohort produced following error message:

findfont: Font family ['Times New Roman'] not found. Falling back to DejaVu Sans.
findfont: Font family ['Arial'] not found. Falling back to DejaVu Sans.
findfont: Font family ['Courier New'] not found. Falling back to DejaVu Sans.
findfont: Font family ['Courier New'] not found. Falling back to DejaVu Sans.
findfont: Font family ['Times New Roman'] not found. Falling back to DejaVu Sans.
findfont: Font family ['Times New Roman'] not found. Falling back to DejaVu Sans.
Traceback (most recent call last):
  File "workflow.py", line 7, in <module>
    hp.analysis("BLCA", "GRCh38", "96", ["288"], "/home/tomas/project/sigprofiler/tcga/", analysis="all", sortSims=True, subClassify=True, includedVAFs=False, includedCCFs=False, correction=True, calculateIMD=True, TCGA=False, sanger=False, standardVC=False)
  File "/home/tomas/anaconda3/lib/python3.8/site-packages/SigProfilerClusters/SigProfilerClusters.py", line 706, in analysis
    plottingFunctions.rainfall(chrom_based, project, input_path, chrom_path, chromLengths, centromeres, contexts, includedVAFs, includedCCFs, correction, windowSize, bedRanges)
  File "/home/tomas/anaconda3/lib/python3.8/site-packages/SigProfilerClusters/plottingFunctions.py", line 1109, in rainfall
    newMutations = pd.read_csv(mutationsPath[i], sep="\t", names=["project", "samples","ID","genome","mutType","chr","start","end", "ref", "alt", "mutClass", "IMDplot", "IMD"], header=0, skiprows=[0], engine='python')
  File "/home/tomas/anaconda3/lib/python3.8/site-packages/pandas/io/parsers.py", line 610, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/home/tomas/anaconda3/lib/python3.8/site-packages/pandas/io/parsers.py", line 468, in _read
    return parser.read(nrows)
  File "/home/tomas/anaconda3/lib/python3.8/site-packages/pandas/io/parsers.py", line 1057, in read
    index, columns, col_dict = self._engine.read(nrows)
  File "/home/tomas/anaconda3/lib/python3.8/site-packages/pandas/io/parsers.py", line 2471, in read
    alldata = self._rows_to_cols(content)
  File "/home/tomas/anaconda3/lib/python3.8/site-packages/pandas/io/parsers.py", line 3164, in _rows_to_cols
    self._alert_malformed(msg, row_num + 1)
  File "/home/tomas/anaconda3/lib/python3.8/site-packages/pandas/io/parsers.py", line 2923, in _alert_malformed
    raise ParserError(msg)
pandas.errors.ParserError: Expected 13 fields in line 66942, saw 25

line 66942:

BLCA	TCGA-DK-A2I4-01A	chr14_55151921_C_G	GRCh38	SNV	14	55151921	55151921	C	G	SOMATIC

I don't know if it is the issue with this sample. I subset all variants found in this sample and paste them in the test.txt.

Could you please help me with it? Thanks.

test.txt

[MousumyCSE]

Hi,
Apologies for the late response. We have run the SigProfilerClusters tool with your test data and have not encountered any issues.
Can you please send us the original input files(if it is possible) and log files?

By this time, can I ask you to check if you are using the updated version of the tools or not! If so, please remove the previous log and the output files and rerun the SigProfilerClusters tool.

Best,
Mousumy