amptk filter --subtract functionality about amptk

[nextgenusfs]

So in the error result, is mockIM4p82redo the option passed to -b or is it mockIM4 as you said in the command before the error message? Maybe that combination of options is causing an error, give me a few minutes to take a look.

[devonorourke]

It's kind of odd. Because our compute cluster uses SLURM to submit jobs, I generate two .log files. The .amptk-filter.log file doesn't show any error, while the SLURM-generated .log file is the one that's throwing the Python error.

If I try to run the script with just the -b mockIM4p82redo flag (and no --mc info) the operation fails and the .amptk-filter.log generates an error stating that I have to include the --mc file.

So I have discovered how to make errors all over the place, ha!

I've also tried altering the --subtract flat to auto instead of an integer, but the same Python error is thrown.

[nextgenusfs]

So the SLURM job is erring out and not amptk filter?

[devonorourke]

Sorry. It appears that the information being generated in the SLURM log file is specific to the output from amptk... here's the whole shebang:

      /\^._.^/\     starting process: Sun Jan 28 15:46:55 EST 2018
[03:46:56 PM]: OS: linux2, 24 cores, ~ 131 GB RAM. Python: 2.7.14
[03:46:57 PM]: AMPtk v1.0.3, USEARCH v9.2.64, VSEARCH v2.6.2
[03:46:57 PM]: Loading OTU table: ../dropd.cluster.otu_table.txt
[03:46:57 PM]: OTU table contains 953 OTUs and 9,335,697 read counts
[03:46:57 PM]: Mapping OTUs to Mock Community (USEARCH)
[03:46:57 PM]: Sorting OTU table naturally
[03:46:57 PM]: Removing OTUs according to --min_reads_otu: (OTUs with less than 2 reads from all samples)
[03:46:57 PM]: Found 10 mock chimeras (only in mock sample and not mapped to mock sequences) excluding from index-bleed calculation
[03:46:57 PM]: Index bleed, mock into samples: 8.307496%.  Index bleed, samples into mock: 0.050620%.
[03:46:57 PM]: Overwriting auto detect index-bleed, setting to 1.000000%
[03:46:57 PM]: Auto subtract filter set to 433
[03:46:57 PM]: Subtracting 433 from OTU table
-------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/lustre/macmaneslab/devon/pkgs/amptk-1.0.3/bin/amptk-filter.py", line 484, in <module>
    mocks = final[args.mock_barcode]
  File "/mnt/lustre/software/linuxbrew/colsa/Cellar/python/2.7.14/lib/python2.7/site-packages/pandas/core/frame.py", line 2139, in __getitem__
    return self._getitem_column(key)
  File "/mnt/lustre/software/linuxbrew/colsa/Cellar/python/2.7.14/lib/python2.7/site-packages/pandas/core/frame.py", line 2146, in _getitem_column
    return self._get_item_cache(key)
  File "/mnt/lustre/software/linuxbrew/colsa/Cellar/python/2.7.14/lib/python2.7/site-packages/pandas/core/generic.py", line 1842, in _get_item_cache
    values = self._data.get(item)
  File "/mnt/lustre/software/linuxbrew/colsa/Cellar/python/2.7.14/lib/python2.7/site-packages/pandas/core/internals.py", line 3838, in get
    loc = self.items.get_loc(item)
  File "/mnt/lustre/software/linuxbrew/colsa/Cellar/python/2.7.14/lib/python2.7/site-packages/pandas/core/indexes/base.py", line 2524, in get_loc
    return self._engine.get_loc(self._maybe_cast_indexer(key))
  File "pandas/_libs/index.pyx", line 117, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 139, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1265, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1273, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'mockIM4p82redo'
      /\^._.^/\     ending process: Sun Jan 28 15:46:57 EST 2018

So from what I can tell, the process stops once amptk begins to subtract the values from the OTU table, with the trouble having something to do with the mock community name.

I hope you like my computer bats in the log file.

[nextgenusfs]

Yeah, okay, I think i just was able to get it to spit me the same error locally. I'll debug now

[nextgenusfs]

well, this a6b5499 should fix it. Not sure why I wrote that in the first place, I think it had something to do with when I was calculating error rate.

[devonorourke]

ok - should I edit the script to delete it myself, or do you pass a new version?
thanks for looking into it so quick

[nextgenusfs]

if you can edit your copy that would be probably better, then I won't rush a release before it has been run through my tests. I have a few other minor things I wanted to address.

[nextgenusfs]

I gather you can see how often I use the subtract filter.....

[devonorourke]

Yep, just edited the local version with that change and the script proceeded just fine.

Thank you !

[nextgenusfs]

Great, let me know when you run into more problems!

[devonorourke]

yeah, I'm a trouble maker.

[nextgenusfs]

Let's leave this issue open in case others have same problem. I will close when I do the next release.