RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm` about scnym

[nagendraKU]

@cartal @sruthi-hub Since the repo seems to be inactive, I am pasting here the session info from a working local conda installation of scNym. Maybe this is useful for you to get scNym running locally.

You will also need CUDA toolkit 10.2.89 & cudnn 8.2.4.15 for cuda 10.2. My conda env also has gcc 11.1.0 but I am not sure if this is strictly needed (my HPC system needs a bunch of stuff by default).

absl-py==1.0.0
anndata==0.7.4
anyio==3.5.0
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
asttokens==2.0.5
attrs==21.4.0
Babel==2.10.1
backcall==0.2.0
beautifulsoup4==4.11.1
bleach==5.0.0
cachetools==4.2.4
certifi==2022.9.24
cffi==1.15.0
chardet==3.0.4
charset-normalizer==2.1.1
ConfigArgParse==1.1
cycler==0.11.0
debugpy==1.6.0
decorator==5.1.1
defusedxml==0.7.1
dunamai==1.11.1
entrypoints==0.4
executing==0.8.3
fastjsonschema==2.15.3
fonttools==4.32.0
get_version==3.5.4
google-auth==1.35.0
google-auth-oauthlib==0.4.6
grpcio==1.44.0
h5py==2.10.0
idna==3.4
igraph==0.9.10
importlib-metadata==4.11.3
importlib-resources==5.7.1
ipykernel==6.13.0
ipython==8.2.0
ipython-genutils==0.2.0
jedi==0.18.1
Jinja2==3.1.1
joblib==1.1.0
json5==0.9.6
jsonschema==4.4.0
jupyter-client==7.2.2
jupyter-core==4.10.0
jupyter-server==1.16.0
jupyterlab==3.3.4
jupyterlab-pygments==0.2.2
jupyterlab-server==2.13.0
kiwisolver==1.4.2
legacy-api-wrap==1.2
leidenalg==0.8.0
llvmlite==0.32.1
louvain==0.7.0
Markdown==3.3.6
MarkupSafe==2.1.1
matplotlib==3.5.1
matplotlib-inline==0.1.3
mistune==0.8.4
more-itertools==8.12.0
natsort==8.1.0
nbclassic==0.3.7
nbclient==0.6.0
nbconvert==6.5.0
nbformat==5.3.0
nest-asyncio==1.5.5
networkx==2.8
notebook==6.4.11
notebook-shim==0.1.0
numba==0.49.1
numexpr==2.8.1
numpy==1.23.3
numpy-groupies==0.9.13
oauthlib==3.2.0
packaging==21.3
pandas==1.4.2
pandocfilters==1.5.0
parso==0.8.3
patsy==0.5.2
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.2.0
pluggy==0.13.1
prometheus-client==0.14.1
prompt-toolkit==3.0.29
protobuf==3.20.0
psutil==5.9.0
ptyprocess==0.7.0
pure-eval==0.2.2
py==1.11.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.21
Pygments==2.11.2
pynndescent==0.5.6
pyparsing==3.0.8
pyrsistent==0.18.1
pytest==5.4.1
python-dateutil==2.8.1
python-igraph==0.9.10
pytz==2022.1
PyYAML==5.3.1
pyzmq==22.3.0
requests==2.28.1
requests-cache==0.5.2
requests-oauthlib==1.3.0
requests-toolbelt==0.9.1
rsa==4.8
scanpy==1.6.0
scikit-learn==0.22.2.post1
scikit-misc==0.1.3
scipy==1.8.0
scnym==0.3.2
seaborn==0.11.2
Send2Trash==1.8.0
setuptools-scm==6.4.2
sinfo==0.3.4
six==1.14.0
sniffio==1.2.0
soupsieve==2.3.2.post1
stack-data==0.2.0
statsmodels==0.13.2
stdlib-list==0.8.0
tables==3.6.1
tensorboard==2.2.1
tensorboard-plugin-wit==1.6.0.post2
tensorboardX==2.1
terminado==0.13.3
texttable==1.6.4
tinycss2==1.1.1
tomli==2.0.1
torch==1.11.0
torchvision==0.12.0
tornado==6.1
tqdm==4.44.1
traitlets==5.1.1
typing_extensions==4.3.0
umap-learn==0.3.10
urllib3==1.26.12
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==1.3.2
Werkzeug==2.1.1
zipp==3.8.0

[nagendraKU]

I run into the same issue when attempting to run the training on Google Colab (high mem VM with A100 GPU). Installed scNym from github using pip.

Any help is appreciated!

[cartal]

@jacobkimmel any help/advice with this would be much appreciated! Thanks

[sruthi-hub]

@cartal @nagendraKU Were you able to fix this? I have the same error.