lkremer / scbs

Python package with CLI for downstream analysis of single cell methylation data.

Hi there

I'm Lukas, a postdoctoral researcher in the labs of Simon Anders and Ana Martin-Villalba in Heidelberg, Germany. I share Python and R code related to bioinformatics, data science, omics and the occasional fun code project.

scbs's Issues

scbs matrix: Allow user to choose between long or wide matrix format

Currently, the output of scbs matrix is a long table, but for most purposes you need a regular cell x region matrix.

wrong repo

make "scbs filter" crash when you filter all cells

If you set your filtering thresholds too strict, scbs filter currently produces a weird directory with empty matrices. This is useless and confusing, instead it should just raise an exception when you try to filter 100% of cells.

Thanks @gleb-gavrish for the suggestion!

add more VMR info to the scbs scan output

Similar to #15, scbs scan should report more info about the resulting VMRs, such as the number of cells with coverage in this VMR, the number of CpG sites, etc.

scbs filter should preserve the run_info log file

scbs prepare writes a data_directory and a log file denoting the scbs version number and the command line parameters that were used.
scbs filter takes a data_directory and creates a new, filtered data_directory. But it doesn't copy the previous unfiltered run_info.txt file. It should copy at least copy the log file and maybe append one line to log the filter parameters.

feature request: merging files for memory issues?

Hi there--I'm an analyst in the Luo lab (associated with one of the single-cell methylomes datasets used in the preprint!). Thanks for developing this software-- I love the user-friendliness and the VMR concept in scbs, and running scbs on CpG methylation is quite smooth.

However, I'm running into some issues as I frequently need to work with non-CpG methylation (CH methylation), which can be associated with ~20-fold more loci. Using 64GB memory and ~2,000 cells, larger chromosomes will fail at the end of the scbs prepare step in what looks like the .coo to .npz conversion.

While I'm currently attempting to re-run with more memory, this is a relatively low cell count dataset for us. I'd be great to somehow merge sets of cells (so maybe I could run on 1,000 of the dataset at a time?) or somehow read/write each chromosome in blocks to use less memory, though I'm not sure either is possible with the details of the sparse format. Are there any ways to "recover" a run to convert the 1.coo file (which is successfully created) without re-running prepare, or any other recommendations?

Cheers!

Populating 57334996 x 2165 matrix for chromosome 19...
Converting from COO to CSR...
Writing to scbs_out_CH/19.npz ...
Populating 260521582 x 2165 matrix for chromosome 1...
Traceback (most recent call last):
  File "/u/home/c/lib/python3.8/site-packages/scbs/prepare.py", line 156, in _load_csr_from_coo
coo = pd.read_csv(coo_path, delimiter=",", header=None).values
File "/u/home/c/lib/python3.8/site-packages/pandas/util/_decorators.py", line 311, in wrapper
return func(*args, **kwargs)
File "/u/home/c/lib/python3.8/site-packages/pandas/io/parsers/readers.py", line 680, in read_csv
return _read(filepath_or_buffer, kwds)
File "/u/home/c/lib/python3.8/site-packages/pandas/io/parsers/readers.py", line 581, in _read
return parser.read(nrows)
File "/u/home/c/lib/python3.8/site-packages/pandas/io/parsers/readers.py", line 1269, in read
df = DataFrame(col_dict, columns=columns, index=index)
File "/u/home/c/lib/python3.8/site-packages/pandas/core/frame.py", line 636, in __init__
mgr = dict_to_mgr(data, index, columns, dtype=dtype, copy=copy, typ=manager)
File "/u/home/c/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 502, in dict_to_mgr
return arrays_to_mgr(arrays, columns, index, dtype=dtype, typ=typ, consolidate=copy)
File "/u/home/c/lib/python3.8/site-packages/pandas/core/internals/construction.py", line 156, in arrays_to_mgr
return create_block_manager_from_column_arrays(
  File "/u/home/c/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 1954, in create_block_manager_from_column_arrays
  blocks = _form_blocks(arrays, consolidate)
  File "/u/home/c/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 2028, in _form_blocks
  values, placement = _stack_arrays(list(tup_block), dtype)
  File "/u/home/c/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 2067, in _stack_arrays
  stacked = np.empty(shape, dtype=dtype)
  numpy.core._exceptions.MemoryError: Unable to allocate 51.7 GiB for an array with shape (3, 2313401180) and data type int64
  
  During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
    File "/u/home/c/bin/scbs", line 8, in <module>
    sys.exit(cli())
  File "/u/home/c/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
  return self.main(*args, **kwargs)
  File "/u/home/c/lib/python3.8/site-packages/click/core.py", line 1053, in main
  rv = self.invoke(ctx)
  File "/u/home/c/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
  return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/u/home/c/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
  return ctx.invoke(self.callback, **ctx.params)
  File "/u/home/c/lib/python3.8/site-packages/click/core.py", line 754, in invoke
  return __callback(*args, **kwargs)
  File "/u/home/c/lib/python3.8/site-packages/scbs/cli.py", line 157, in prepare_cli
  prepare(**kwargs)
  File "/u/home/c/lib/python3.8/site-packages/scbs/prepare.py", line 44, in prepare
  mat = _load_csr_from_coo(coo_path, chrom_size, n_cells)
  File "/u/home/c/lib/python3.8/site-packages/scbs/prepare.py", line 165, in _load_csr_from_coo
  raise type(exc)(f"{exc} (problematic file: {coo_path})").with_traceback(
    TypeError: __init__() missing 1 required positional argument: 'dtype'

allow `scbs filter` to overwrite the input directory

Currently, scbs filter creates a new directory containing the filtered data. Some users might want to filter "in place", which would mean that the unfiltered input directory is overwritten with the new, filtered directory. Any attempts to do this currently result in buggy behavior. So I should either implement this feature, or, if it's too complicated to do, explicitly forbid this behavior by raising an exception if the output and input directory are the same.

Add a method to allow for easy cell filtering (e.g. after QC)

Currently, you have do this manually by re-running scbs prepare on high-quality cells only. It would be nice if you could just filter low quality or uninteresting cells from DATA_DIR directly.

custom format specification

For option '--input-format', column 4 is defined as follows:

 4. The column number that contains either unmethylated counts (u) or the total
                       coverage (c) followed by either 'm' or 'c', e.g. '4c' to denote that the 4th column
                       contains the coverage

Why do I have to suffix the number with 'm' and not 'u'?

add more DMR information to the scbs diff output

It would be useful to add more information to the resulting DMR list. For instance, the mean methylation % in both cell groups, the number of CpG sites, the number of cells that have coverage in this DMR, and the raw p-value (for volcano plots etc.).