biocpy / scranpy

Single-cell data analysis, from C++ to Python

BiocPy

Installs all BiocPy packages. The package itself does not provide any functionality on its own. It provides an

• easier way to install all core packages
• an interface to access the base data representations or classes

Install

Package is published to PyPI

pip install biocpy

Usage

To import any of the data classes,

from biocpy.genomicranges import GenomicRanges
from biocpy.summarizedexperiment import SummarizedExperiment
from biocpy.singlecellexperiment import SingleCellExperiment 
from biocpy.multiassayexperiment import MultiAssayExperiment

gr = GenomicRange(...)
se = SummarizedExperiment(...)
sce = SingleCellExperiment(...)
mae = MultiAssayExperiment(...)

For more use cases, checkout the documentation.

Note

This project has been set up using PyScaffold 4.1.1. For details and usage information on PyScaffold see https://pyscaffold.org/.

scranpy's Issues

Code golf me!

Current minimal working example for the PBMC dataset:

# TODO: streamline the loader:
path = "pbmc4k-tenx.h5"
import h5py as h5
fhandle = h5.File(path)
import scipy.sparse as sp
mat = sp.csc_matrix((fhandle["matrix"]["data"], fhandle["matrix"]["indices"], fhandle["matrix"]["indptr"]), fhandle["matrix"]["shape"])
features = [x.decode("ascii") for x in fhandle["matrix"]["features"]["name"]]

# Performing QC.
import scranpy.quality_control as qc
metrics = qc.per_cell_rna_qc_metrics(mat, { "mito": qc.guess_mito_from_symbols(features) })
filters = qc.suggest_rna_qc_filters(metrics)

import numpy as np
overall = np.logical_and(
    np.logical_and(
        metrics.column("sums") >= filters.column("sums")[0],
        metrics.column("detected") >= filters.column("detected")[0]
    ),
    metrics.column("subset_proportions").column("mito") <= filters.column("subset_proportions").column("mito")[0]
)

# MISSING: filtering.

import mattress
ptr = mattress.tatamize(mat)
import scranpy.normalization as norm
normed = norm.log_norm_counts(ptr)

import scranpy.feature_selection as feat 
varstats = feat.model_gene_variances(normed)
resids = varstats.column("residuals")
cutoff = np.sort(resids)[-2000]
selected = []
for i in range(len(resids)):
    if resids[i] >= cutoff:
        selected.append(i)

import scranpy.dimensionality_reduction as dimred
pca = dimred.run_pca(normed, rank=20, subset=selected)

# TODO: run these all at once.
tsne = dimred.run_tsne(pca.principal_components)
umap = dimred.run_umap(pca.principal_components)

import scranpy.clustering as clust
g = clust.build_snn_graph(pca.principal_components)
clusters = g.community_multilevel().membership

import scranpy.marker_detection as mark
markers = mark.score_markers(normed, clusters)

better exception handling

set output numpy vectors to zeros

concurrent.futures error on M2 Mac

Hi there,

Getting an error from concurrent.futures when trying to run the 'Quick Start' example from the readme specifically on my M2 mac. The identical code raises no errors on Ubuntu 22.04 LTS. I get the same results across multiple major python versions. Any ideas for how to fix this?

import singlecellexperiment
import scranpy

sce = singlecellexperiment.read_tenx_h5("./pbmc4k-tenx.h5")
options = scranpy.AnalyzeOptions()
options.per_cell_rna_qc_metrics_options.subsets = {
    "mito" : scranpy.guess_mito_from_symbols(sce.row_data['name'], 'mt-')
}
results = scranpy.analyze_sce(sce, options=options)

Full trace:

/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/site-packages/genomicranges/SeqInfo.py:348: UserWarning: 'seqnames' is deprecated, use 'get_seqnames' instead
  warn("'seqnames' is deprecated, use 'get_seqnames' instead", UserWarning)
/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/site-packages/biocframe/BiocFrame.py:833: UserWarning: This method performs an in-place operation, use 'set_column' instead
  warn(
/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/site-packages/genomicranges/SeqInfo.py:348: UserWarning: 'seqnames' is deprecated, use 'get_seqnames' instead
  warn("'seqnames' is deprecated, use 'get_seqnames' instead", UserWarning)
/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/site-packages/biocframe/BiocFrame.py:833: UserWarning: This method performs an in-place operation, use 'set_column' instead
  warn(
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/multiprocessing/spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/multiprocessing/spawn.py", line 131, in _main
    prepare(preparation_data)
  File "/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/multiprocessing/spawn.py", line 246, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/multiprocessing/spawn.py", line 297, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen runpy>", line 286, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/griffen/scratch/scranpy/process.py", line 9, in <module>
    results = scranpy.analyze_sce(sce, options=options)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/site-packages/scranpy/analyze/analyze.py", line 209, in analyze_sce
    return analyze_se(
           ^^^^^^^^^^^
  File "/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/site-packages/scranpy/analyze/analyze.py", line 132, in analyze_se
    return analyze(
           ^^^^^^^^
  File "/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/site-packages/scranpy/analyze/analyze.py", line 70, in analyze
    return live_analyze(
           ^^^^^^^^^^^^^
  File "/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/site-packages/scranpy/analyze/live_analyze.py", line 297, in live_analyze
    get_tsne, get_umap, graph, remaining_threads = run_neighbor_suite(
                                                   ^^^^^^^^^^^^^^^^^^^
  File "/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/site-packages/scranpy/analyze/run_neighbor_suite.py", line 100, in run_neighbor_suite
    executor.submit(
  File "/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/concurrent/futures/process.py", line 835, in submit
    self._adjust_process_count()
  File "/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/concurrent/futures/process.py", line 794, in _adjust_process_count
    self._spawn_process()
  File "/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/concurrent/futures/process.py", line 812, in _spawn_process
    p.start()
  File "/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/multiprocessing/context.py", line 289, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/multiprocessing/popen_spawn_posix.py", line 42, in _launch
    prep_data = spawn.get_preparation_data(process_obj._name)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/multiprocessing/spawn.py", line 164, in get_preparation_data
    _check_not_importing_main()
  File "/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/multiprocessing/spawn.py", line 140, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

        To fix this issue, refer to the "Safe importing of main module"
        section in https://docs.python.org/3/library/multiprocessing.html

Traceback (most recent call last):
  File "/Users/griffen/scratch/scranpy/process.py", line 9, in <module>
    results = scranpy.analyze_sce(sce, options=options)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/site-packages/scranpy/analyze/analyze.py", line 209, in analyze_sce
    return analyze_se(
           ^^^^^^^^^^^
  File "/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/site-packages/scranpy/analyze/analyze.py", line 132, in analyze_se
    return analyze(
           ^^^^^^^^
  File "/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/site-packages/scranpy/analyze/analyze.py", line 70, in analyze
    return live_analyze(
           ^^^^^^^^^^^^^
  File "/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/site-packages/scranpy/analyze/live_analyze.py", line 348, in live_analyze
    results.tsne = get_tsne()
                   ^^^^^^^^^^
  File "/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/site-packages/scranpy/analyze/run_neighbor_suite.py", line 125, in get_tsne
    return _tasks[0].result()
           ^^^^^^^^^^^^^^^^^^
  File "/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/griffen/mambaforge/envs/biocpy/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Versions:

# Name                    Version                   Build  Channel
appnope                   0.1.3              pyhd8ed1ab_0    conda-forge
assorthead                0.0.11                   pypi_0    pypi
asttokens                 2.4.1              pyhd8ed1ab_0    conda-forge
biocframe                 0.5.9                    pypi_0    pypi
biocutils                 0.1.5                    pypi_0    pypi
bzip2                     1.0.8                h93a5062_5    conda-forge
c-ares                    1.26.0               h93a5062_0    conda-forge
ca-certificates           2023.11.17           hf0a4a13_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
comm                      0.2.1              pyhd8ed1ab_0    conda-forge
debugpy                   1.8.0           py312h9f69965_1    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
delayedarray              0.5.0                    pypi_0    pypi
exceptiongroup            1.2.0              pyhd8ed1ab_2    conda-forge
executing                 2.0.1              pyhd8ed1ab_0    conda-forge
genomicranges             0.4.12                   pypi_0    pypi
h5py                      3.10.0          nompi_py312hac0f6fd_101    conda-forge
hdf5                      1.14.3          nompi_h5bb55e9_100    conda-forge
igraph                    0.11.3                   pypi_0    pypi
importlib-metadata        7.0.1              pyha770c72_0    conda-forge
importlib_metadata        7.0.1                hd8ed1ab_0    conda-forge
ipykernel                 6.29.0             pyh3cd1d5f_0    conda-forge
ipython                   8.21.0             pyh707e725_0    conda-forge
iranges                   0.2.3                    pypi_0    pypi
jedi                      0.19.1             pyhd8ed1ab_0    conda-forge
jupyter_client            8.6.0              pyhd8ed1ab_0    conda-forge
jupyter_core              5.7.1           py312h81bd7bf_0    conda-forge
krb5                      1.21.2               h92f50d5_0    conda-forge
libaec                    1.1.2                h13dd4ca_1    conda-forge
libblas                   3.9.0           21_osxarm64_openblas    conda-forge
libcblas                  3.9.0           21_osxarm64_openblas    conda-forge
libcurl                   8.5.0                h2d989ff_0    conda-forge
libcxx                    16.0.6               h4653b0c_0    conda-forge
libedit                   3.1.20191231         hc8eb9b7_2    conda-forge
libev                     4.33                 h93a5062_2    conda-forge
libexpat                  2.5.0                hb7217d7_1    conda-forge
libffi                    3.4.2                h3422bc3_5    conda-forge
libgfortran               5.0.0           13_2_0_hd922786_2    conda-forge
libgfortran5              13.2.0               hf226fd6_2    conda-forge
liblapack                 3.9.0           21_osxarm64_openblas    conda-forge
libnghttp2                1.58.0               ha4dd798_1    conda-forge
libopenblas               0.3.26          openmp_h6c19121_0    conda-forge
libsodium                 1.0.18               h27ca646_1    conda-forge
libsqlite                 3.44.2               h091b4b1_0    conda-forge
libssh2                   1.11.0               h7a5bd25_0    conda-forge
libzlib                   1.2.13               h53f4e23_5    conda-forge
llvm-openmp               17.0.6               hcd81f8e_0    conda-forge
matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
mattress                  0.1.4                    pypi_0    pypi
ncls                      0.0.68                   pypi_0    pypi
ncurses                   6.4                  h463b476_2    conda-forge
nest-asyncio              1.6.0              pyhd8ed1ab_0    conda-forge
numpy                     1.26.3          py312h8442bc7_0    conda-forge
openssl                   3.2.1                h0d3ecfb_0    conda-forge
packaging                 23.2               pyhd8ed1ab_0    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
pexpect                   4.9.0              pyhd8ed1ab_0    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pip                       23.3.2             pyhd8ed1ab_0    conda-forge
platformdirs              4.2.0              pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.42             pyha770c72_0    conda-forge
psutil                    5.9.8           py312he37b823_0    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
pygments                  2.17.2             pyhd8ed1ab_0    conda-forge
python                    3.12.1          hdf0ec26_1_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python_abi                3.12                    4_cp312    conda-forge
pyzmq                     25.1.2          py312h1edf716_0    conda-forge
readline                  8.2                  h92ec313_1    conda-forge
scipy                     1.12.0                   pypi_0    pypi
scranpy                   0.1.1                    pypi_0    pypi
setuptools                69.0.3             pyhd8ed1ab_0    conda-forge
singlecellexperiment      0.4.3                    pypi_0    pypi
six                       1.16.0             pyh6c4a22f_0    conda-forge
stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
summarizedexperiment      0.4.2                    pypi_0    pypi
texttable                 1.7.0                    pypi_0    pypi
tk                        8.6.13               h5083fa2_1    conda-forge
tornado                   6.3.3           py312h02f2b3b_1    conda-forge
traitlets                 5.14.1             pyhd8ed1ab_0    conda-forge
typing_extensions         4.9.0              pyha770c72_0    conda-forge
tzdata                    2023d                h0c530f3_0    conda-forge
wcwidth                   0.2.13             pyhd8ed1ab_0    conda-forge
wheel                     0.42.0             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h57fd34a_0    conda-forge
zeromq                    4.3.5                h965bd2d_0    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
zstd                      1.5.5                h4f39d0f_0    conda-forge

Documentation for anndata, and running log norm functions

• also get normalized counts without the log scaling

add verbose to each step

notes to self

• Need to transpose the PCs because C++ has it in column-major but NumPy will treat it as row major.

Block needs to be sliced after QC in the analyze function.

So it's not really something that we can set at option time.

set random seed for igraph

pypy builds in github action