Remove dependencies on neural-specific modules but issue a warning if a function is trying to be used that requires it about teneto

[wiheto]

Closing this as the PR is ready to be merged.

[wiheto]

Sounds like a good idea (happy non-neural people are using the package!).

Do you want to submit a PR? If not, I'll add this to the next version myself.

[wiheto]

Actually, I just realized it's going to require a tad of reorganization that I'll have to think about. So I will try and get it in.

First, I think this would be a good time to create a new module like: teneto.neuro for all the neuroimaging related functions to be in. Then this would make your suggestion much easier to implement.

[wiheto]

So what I suggested what an awful suggestion at the end of the day. I think keeping it simple makes a lot of sense.

How about this:

Option 1 (you submit a PR).

1. Create a new file in teneto/utils. Something like: teneto/utils/importchecker.py and place assert_installed_packages in there.

2. Submit PR.

3. Then I will add new installation instructions, remove the dependencies, and I will add @assert_installed_packages(...) in the relevant places.

Option 2 (I'll do it).

Basically the same as 1, but I'll do the steps. The reason for doing option 1 is if you want the credit for the idea.

[jolespin]

I haven't dove too deep into the package but (in the future) having teneto.neuro doesn't sound like a bad idea at all. However, that is probably a considerable amount of work on your end.

I'd like to take a shot at this. I'll let you know if I can't figure it out. I've been writing a paper non-stop lately so I could use a break haha. I've been meaning to learn how to properly implement decorators (you can see my failed attempt above) so this is the perfect tangent.

btw, I do bioinformatics with an emphasis in microbial ecology (both environmental and human). teneto seems to be the only legit temporal network analysis package in Python. I actually reference your 2017 paper quite a bit in a mini-review paper I'm writing right now. The way you describe a lot of these concepts was really helpful for both me and my advisor.

[wiheto]

The way you describe a lot of these concepts was really helpful for both me and my advisor.

Happy to hear that.

I'd like to take a shot at this. I'll let you know if I can't figure it out

Only do it if you have time and want to contribute to the package. Otherwise I don't mind adding it myself. Your call.

I haven't dove too deep into the package but (in the future) having teneto.neuro doesn't sound like a bad idea at all. However, that is probably a considerable amount of work on your end.

It is probably a good idea to do teneto.neruo in the longterm. But I realized it needs a little bit of thinking to do and would still needs to add your solution unless some subpackages are not imported directly (like sklearn).

[jolespin]

I think I've implemented everything correctly (almost).

Is there a test command that I should use?

There's an error that I received from import bids Do you get this error with the current version? It's a little confusing here because the script is bids.py and it's importing a bids package. Was this supposed to be pybids?

Was this class in another script at one point?

(teneto_env) jespinozlt2-osx:GitHub jespinoz$ pip install ./teneto/
Processing ./teneto
Collecting statsmodels>=0.8.0
  Downloading statsmodels-0.11.1-cp36-cp36m-macosx_10_13_x86_64.whl (8.4 MB)
     |████████████████████████████████| 8.4 MB 1.5 MB/s
Collecting networkx>=2.0
  Downloading networkx-2.4-py3-none-any.whl (1.6 MB)
     |████████████████████████████████| 1.6 MB 86.6 MB/s
Collecting python-louvain>=0.13
  Using cached python-louvain-0.14.tar.gz (19 kB)
Collecting pandas>=0.21
  Downloading pandas-1.0.3-cp36-cp36m-macosx_10_9_x86_64.whl (10.2 MB)
     |████████████████████████████████| 10.2 MB 31.3 MB/s
Collecting scipy>=1.4.1
  Downloading scipy-1.4.1-cp36-cp36m-macosx_10_6_intel.whl (28.5 MB)
     |████████████████████████████████| 28.5 MB 16.6 MB/s
Collecting numpy>=1.16.1
  Downloading numpy-1.18.3-cp36-cp36m-macosx_10_9_x86_64.whl (15.2 MB)
     |████████████████████████████████| 15.2 MB 3.9 MB/s
Collecting patsy>=0.5
  Downloading patsy-0.5.1-py2.py3-none-any.whl (231 kB)
     |████████████████████████████████| 231 kB 76.3 MB/s
Collecting decorator>=4.3.0
  Using cached decorator-4.4.2-py2.py3-none-any.whl (9.2 kB)
Collecting python-dateutil>=2.6.1
  Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting pytz>=2017.2
  Downloading pytz-2020.1-py2.py3-none-any.whl (510 kB)
     |████████████████████████████████| 510 kB 78.1 MB/s
Collecting six
  Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Building wheels for collected packages: teneto, python-louvain
  Building wheel for teneto (setup.py) ... done
  Created wheel for teneto: filename=teneto-0.5.1-py3-none-any.whl size=15573800 sha256=341a55f2d5b49a39db583a68ee0b910b97fd1112e7eb680f73c019ef0a7c2af9
  Stored in directory: /private/var/folders/6z/5vbtz_gmkr76ftgc3149dvtr0003c0/T/pip-ephem-wheel-cache-fsw41ubv/wheels/f9/00/78/2359861b121b145bdff9a5b676a6ce3c60b5ef2ee3a3462d93
  Building wheel for python-louvain (setup.py) ... done
  Created wheel for python-louvain: filename=python_louvain-0.14-py3-none-any.whl size=8292 sha256=0473a40f0120f860888b654b97688db298ea56c2a5e09803a92e6e9c79537fcf
  Stored in directory: /Users/jespinoz/Library/Caches/pip/wheels/d5/54/b7/9a703ffb0734ba1cdce4b827b2f07ed05e3108d76b32b9f510
Successfully built teneto python-louvain
ERROR: tf-nightly 1.9.0.dev20180605 requires absl-py>=0.1.6, which is not installed.
ERROR: tf-nightly 1.9.0.dev20180605 requires astor>=0.6.0, which is not installed.
ERROR: tf-nightly 1.9.0.dev20180605 requires gast>=0.2.0, which is not installed.
ERROR: tf-nightly 1.9.0.dev20180605 requires protobuf>=3.4.0, which is not installed.
ERROR: tf-nightly 1.9.0.dev20180605 requires tb-nightly<1.10.0a0,>=1.9.0a0, which is not installed.
ERROR: tf-nightly 1.9.0.dev20180605 requires termcolor>=1.1.0, which is not installed.
Installing collected packages: numpy, scipy, six, patsy, python-dateutil, pytz, pandas, statsmodels, decorator, networkx, python-louvain, teneto
Successfully installed decorator-4.4.2 networkx-2.4 numpy-1.18.3 pandas-1.0.3 patsy-0.5.1 python-dateutil-2.8.1 python-louvain-0.14 pytz-2020.1 scipy-1.4.1 six-1.14.0 statsmodels-0.11.1 teneto-0.5.1
(teneto_env) jespinozlt2-osx:GitHub jespinoz$ python
Python 3.6.10 | packaged by conda-forge | (default, Apr 24 2020, 16:27:41)
[GCC Clang 9.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
im>>> import teneto
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jespinoz/anaconda3/envs/teneto_env/lib/python3.6/site-packages/teneto/__init__.py", line 9, in <module>
    from . import networkmeasures
  File "/Users/jespinoz/anaconda3/envs/teneto_env/lib/python3.6/site-packages/teneto/networkmeasures/__init__.py", line 3, in <module>
    from .temporal_degree_centrality import temporal_degree_centrality
  File "/Users/jespinoz/anaconda3/envs/teneto_env/lib/python3.6/site-packages/teneto/networkmeasures/temporal_degree_centrality.py", line 4, in <module>
    from ..utils import process_input
  File "/Users/jespinoz/anaconda3/envs/teneto_env/lib/python3.6/site-packages/teneto/utils/__init__.py", line 3, in <module>
    from .utils import check_packages, graphlet2contact, contact2graphlet,\
  File "/Users/jespinoz/anaconda3/envs/teneto_env/lib/python3.6/site-packages/teneto/utils/utils.py", line 8, in <module>
    from ..classes import TemporalNetwork
  File "/Users/jespinoz/anaconda3/envs/teneto_env/lib/python3.6/site-packages/teneto/classes/__init__.py", line 3, in <module>
    from .bids import TenetoBIDS
  File "/Users/jespinoz/anaconda3/envs/teneto_env/lib/python3.6/site-packages/teneto/classes/bids.py", line 3, in <module>
    import bids
ModuleNotFoundError: No module named 'bids'

[wiheto]

Fair point that that is confusing. "import bids" is importing pybids.

Is there a test command that I should use?

I think the best was to test would be to use: teneto/classes/bids.py. The only neuroimaging module it is importing is pybids (i.e. import bids).

then you could do from teneto import TenetoBIDS to make sure it still imports.

[jolespin]

Thanks, for some reason I'm having trouble importing the new function that I added in utils. Can you spot the error?

>>> import teneto
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jespinoz/Google/GitHub/teneto/teneto/__init__.py", line 9, in <module>
    from . import networkmeasures
  File "/Users/jespinoz/Google/GitHub/teneto/teneto/networkmeasures/__init__.py", line 3, in <module>
    from .temporal_degree_centrality import temporal_degree_centrality
  File "/Users/jespinoz/Google/GitHub/teneto/teneto/networkmeasures/temporal_degree_centrality.py", line 4, in <module>
    from ..utils import process_input
  File "/Users/jespinoz/Google/GitHub/teneto/teneto/utils/__init__.py", line 3, in <module>
    from .utils import check_packages, graphlet2contact, contact2graphlet,\
  File "/Users/jespinoz/Google/GitHub/teneto/teneto/utils/utils.py", line 8, in <module>
    from ..classes import TemporalNetwork
  File "/Users/jespinoz/Google/GitHub/teneto/teneto/classes/__init__.py", line 3, in <module>
    from .bids import TenetoBIDS
  File "/Users/jespinoz/Google/GitHub/teneto/teneto/classes/bids.py", line 6, in <module>
    from ..neuroimagingtools import load_tabular_file, get_sidecar
  File "/Users/jespinoz/Google/GitHub/teneto/teneto/neuroimagingtools/__init__.py", line 3, in <module>
    from .fmriutils import make_parcellation
  File "/Users/jespinoz/Google/GitHub/teneto/teneto/neuroimagingtools/fmriutils.py", line 3, in <module>
    from ..utils import check_packages
ImportError: cannot import name 'check_packages'

I thought from ..utils import check_packages would be correct.
I also made sure to add it to the utils/__init__.py file.

[wiheto]

If you submit a PR, it will be easier for me to look at the code/suggest changes. Hard to debug without seeing the code.

[jolespin]

It's been a minute since I've issued a pull request.

Here's my repository:
https://github.com/jolespin/teneto

The pull request failed all tests as expect b/c of issues importing but I think once the import works it should pass the tests:
#66

When you get a chance, can you check out the imports of check_packages? I thought that from ..utils import check_packages would work in fmriutils.py

[wiheto]

So spent some time looking at the problem this weekend. There is a circular import bug going on which I previously solved with a couple of absolute instead of relative imports (and the reason teneto was a little slow to import). It was basically all being held together with duck tape and something I sould have fixed a while ago. Now it is time to fix it.

So I will solve this over the next few days cause there needs to be a little bit of restructuring of the code. And then we'll deal with your PR.

[jolespin]

Oh ok, I know how that is and those problems are the worst because it takes some significant rewrites. In the past what I've done was either brought everything into a separate utils package (e.g. teneto_utils) or brought everything in the util subpackage (e.g. teneto.utils). Good luck.

[wiheto]

Problems solved (mainly).
But I then had to reject your PR. You had committed a lot of files to it however, not just the changes you made for this issues. Please do the following:

1. Update to the latest version on the master branch.
2. Change utils/utils.py and utils/__init__.py as you had done before.
3. Commit it.
4. Then submit a PR that contains changes to those two files.

I will then go about and remove the dependencies and update documentation. I already tested the code, and it works fine. It just needs to be submitted correctly now.
And, if it is not clear, the reason I think you should submit this is for the credit, since it it your code.

[jolespin]

Sounds good! I'll try to get this done this week.

I found the "optional" package error from pandas when trying to read excel files using xlrd that I was originally referring to with this functionality:

~/anaconda3/envs/soothsayer4_env/lib/python3.8/site-packages/pandas/io/excel/_base.py in read_excel(io, sheet_name, header, names, index_col, usecols, squeeze, dtype, engine, converters, true_values, false_values, skiprows, nrows, na_values, keep_default_na, verbose, parse_dates, date_parser, thousands, comment, skipfooter, convert_float, mangle_dupe_cols, **kwds)
    302 
    303     if not isinstance(io, ExcelFile):
--> 304         io = ExcelFile(io, engine=engine)
    305     elif engine and engine != io.engine:
    306         raise ValueError(

~/anaconda3/envs/soothsayer4_env/lib/python3.8/site-packages/pandas/io/excel/_base.py in __init__(self, io, engine)
    822         self._io = stringify_path(io)
    823 
--> 824         self._reader = self._engines[engine](self._io)
    825 
    826     def __fspath__(self):

~/anaconda3/envs/soothsayer4_env/lib/python3.8/site-packages/pandas/io/excel/_xlrd.py in __init__(self, filepath_or_buffer)
     18         """
     19         err_msg = "Install xlrd >= 1.0.0 for Excel support"
---> 20         import_optional_dependency("xlrd", extra=err_msg)
     21         super().__init__(filepath_or_buffer)
     22 

~/anaconda3/envs/soothsayer4_env/lib/python3.8/site-packages/pandas/compat/_optional.py in import_optional_dependency(name, extra, raise_on_missing, on_version)
     90     except ImportError:
     91         if raise_on_missing:
---> 92             raise ImportError(msg) from None
     93         else:
     94             return None

ImportError: Missing optional dependency 'xlrd'. Install xlrd >= 1.0.0 for Excel support Use pip or conda to install xlrd.

[jolespin]

Just got it to work! Uploading now and issuing a pull request. Sorry for the delay. I had some papers come back that have been in review for a minute.

Creating a new conda environment for teneto

(base) jespinozlt2-osx:GitHub jespinoz$ conda env remove --name teneto_env && conda create --name teneto_env python=3 && conda activate teneto_env

Remove all packages in environment /Users/jespinoz/anaconda3/envs/teneto_env:

Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /Users/jespinoz/anaconda3/envs/teneto_env

  added / updated specs:
    - python=3


The following NEW packages will be INSTALLED:

  ca-certificates    conda-forge/osx-64::ca-certificates-2020.4.5.1-hecc5488_0
  certifi            conda-forge/osx-64::certifi-2020.4.5.1-py38h32f6830_0
  libcxx             conda-forge/osx-64::libcxx-10.0.0-h1af66ff_2
  libffi             bioconda/osx-64::libffi-3.2.1-1
  ncurses            conda-forge/osx-64::ncurses-6.1-h0a44026_1002
  openssl            conda-forge/osx-64::openssl-1.1.1g-h0b31af3_0
  pip                conda-forge/noarch::pip-20.1-pyh9f0ad1d_0
  python             conda-forge/osx-64::python-3.8.2-hd5f0129_7_cpython
  python_abi         conda-forge/osx-64::python_abi-3.8-1_cp38
  readline           conda-forge/osx-64::readline-8.0-hcfe32e1_0
  setuptools         conda-forge/osx-64::setuptools-46.4.0-py38h32f6830_0
  sqlite             conda-forge/osx-64::sqlite-3.30.1-h93121df_0
  tk                 conda-forge/osx-64::tk-8.6.10-hbbe82c9_0
  wheel              conda-forge/noarch::wheel-0.34.2-py_1
  xz                 conda-forge/osx-64::xz-5.2.5-h0b31af3_0
  zlib               conda-forge/osx-64::zlib-1.2.11-h0b31af3_1006


Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate teneto_env
#
# To deactivate an active environment, use
#
#     $ conda deactivate

Installin local version of teneto after changes

(teneto_env) jespinozlt2-osx:GitHub jespinoz$ pip install teneto/
Processing ./teneto
Collecting statsmodels>=0.8.0
  Using cached statsmodels-0.11.1-cp38-cp38-macosx_10_13_x86_64.whl (8.5 MB)
Collecting networkx>=2.0
  Using cached networkx-2.4-py3-none-any.whl (1.6 MB)
Processing /Users/jespinoz/Library/Caches/pip/wheels/22/f9/ce/591ffa9b16851da50ca337c9ecfd44d79a7b87fcbd2a7a0021/python_louvain-0.14-py3-none-any.whl
Collecting pandas>=0.21
  Using cached pandas-1.0.3-cp38-cp38-macosx_10_9_x86_64.whl (10.2 MB)
Collecting scipy>=1.4.1
  Using cached scipy-1.4.1-cp38-cp38-macosx_10_9_x86_64.whl (28.8 MB)
Collecting numpy>=1.16.1
  Using cached numpy-1.18.4-cp38-cp38-macosx_10_9_x86_64.whl (15.2 MB)
Collecting matplotlib>=3.1
  Using cached matplotlib-3.2.1-cp38-cp38-macosx_10_9_x86_64.whl (12.4 MB)
Collecting patsy>=0.5
  Using cached patsy-0.5.1-py2.py3-none-any.whl (231 kB)
Collecting decorator>=4.3.0
  Using cached decorator-4.4.2-py2.py3-none-any.whl (9.2 kB)
Collecting pytz>=2017.2
  Using cached pytz-2020.1-py2.py3-none-any.whl (510 kB)
Collecting python-dateutil>=2.6.1
  Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting cycler>=0.10
  Using cached cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)
Collecting kiwisolver>=1.0.1
  Using cached kiwisolver-1.2.0-cp38-cp38-macosx_10_9_x86_64.whl (60 kB)
Collecting six
  Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Building wheels for collected packages: teneto
  Building wheel for teneto (setup.py) ... done
  Created wheel for teneto: filename=teneto-0.5.2_dev_c-py3-none-any.whl size=15594489 sha256=8c78f35abd5da17094b848244eac4ca378e871959e06ba986265b1c313baaad4
  Stored in directory: /private/var/folders/6z/5vbtz_gmkr76ftgc3149dvtr0003c0/T/pip-ephem-wheel-cache-2qaog5d1/wheels/11/be/1a/3f25bc328dd14e960aa1056278e5652b5f98cc9a70c7584ef1
Successfully built teneto
Installing collected packages: six, numpy, patsy, pytz, python-dateutil, pandas, scipy, statsmodels, decorator, networkx, python-louvain, pyparsing, cycler, kiwisolver, matplotlib, teneto
Successfully installed cycler-0.10.0 decorator-4.4.2 kiwisolver-1.2.0 matplotlib-3.2.1 networkx-2.4 numpy-1.18.4 pandas-1.0.3 patsy-0.5.1 pyparsing-2.4.7 python-dateutil-2.8.1 python-louvain-0.14 pytz-2020.1 scipy-1.4.1 six-1.14.0 statsmodels-0.11.1 teneto-0.5.2-dev-c

Making sure teneto imports

(teneto_env) jespinozlt2-osx:GitHub jespinoz$ python
Python 3.8.2 | packaged by conda-forge | (default, Apr 24 2020, 07:56:27)
[Clang 9.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import teneto
>>> teneto
<module 'teneto' from '/Users/jespinoz/anaconda3/envs/teneto_env/lib/python3.8/site-packages/teneto/__init__.py'>
>>> exit()

Checking packages

(teneto_env) jespinozlt2-osx:GitHub jespinoz$ conda list
# packages in environment at /Users/jespinoz/anaconda3/envs/teneto_env:
#
# Name                    Version                   Build  Channel
ca-certificates           2020.4.5.1           hecc5488_0    conda-forge
certifi                   2020.4.5.1       py38h32f6830_0    conda-forge
cycler                    0.10.0                   pypi_0    pypi
decorator                 4.4.2                    pypi_0    pypi
kiwisolver                1.2.0                    pypi_0    pypi
libcxx                    10.0.0               h1af66ff_2    conda-forge
libffi                    3.2.1                         1    bioconda
matplotlib                3.2.1                    pypi_0    pypi
ncurses                   6.1               h0a44026_1002    conda-forge
networkx                  2.4                      pypi_0    pypi
numpy                     1.18.4                   pypi_0    pypi
openssl                   1.1.1g               h0b31af3_0    conda-forge
pandas                    1.0.3                    pypi_0    pypi
patsy                     0.5.1                    pypi_0    pypi
pip                       20.1               pyh9f0ad1d_0    conda-forge
pyparsing                 2.4.7                    pypi_0    pypi
python                    3.8.2           hd5f0129_7_cpython    conda-forge
python-dateutil           2.8.1                    pypi_0    pypi
python-louvain            0.14                     pypi_0    pypi
python_abi                3.8                      1_cp38    conda-forge
pytz                      2020.1                   pypi_0    pypi
readline                  8.0                  hcfe32e1_0    conda-forge
scipy                     1.4.1                    pypi_0    pypi
setuptools                46.4.0           py38h32f6830_0    conda-forge
six                       1.14.0                   pypi_0    pypi
sqlite                    3.30.1               h93121df_0    conda-forge
statsmodels               0.11.1                   pypi_0    pypi
teneto                    0.5.2-dev-c              pypi_0    pypi
tk                        8.6.10               hbbe82c9_0    conda-forge
wheel                     0.34.2                     py_1    conda-forge
xz                        5.2.5                h0b31af3_0    conda-forge
zlib                      1.2.11            h0b31af3_1006    conda-forge

[jolespin]

Is there a test command I can run for teneto to make sure everything is working correctly?

[wiheto]

Don’t worry. Should be fine. I’ll take care of this this evening. Sorry for it taking some time to get to. /William

…

27 maj 2020 kl. 17:59 skrev Josh L. Espinoza ***@***.***>:  Is there a test command I can run for teneto to make sure eerything is working correctly? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or unsubscribe.

[wiheto]

As noted on the PR, after merging, this code has been removed. The reason were several:

1. some easy to fix errors was import importlibs instead of from importlibs import import_module - easy to fix.
2. the way that globals() was being updated inside a wrapper did not seem to import the modules in globals outside of the wrapper. I do not know the solution to this. The code executed, but the packages were never imported when needed.
3. This then required that imports be moved to inside of functions (which was the case for some of the functions anyway), which is non-pythonic way of doing things (and I still couldn't get it to fully work doing it this way either with check_packages only checking if the package existed. TenetoBIDS kept failing to import bids regardless). 4
4. Some other modules were removed that are still needed (sklearn and tables).

Maybe you have an answer to these (primarily problem 2)? If yes, great! If you want to submit a new PR, please also change the .travis.yml file and add "pip install" commands for the neuropackages to the .travis.yml. This way we can run testing on the PR and not on the master branch like I did now (that is what I should have asked you to do at the start, sorry about that).

[jolespin]

Thanks for looking into this. I will see if there any adjustments I can make.

Are there any tests I can run by hand to make sure everything imports and runs correctly before submitting the PR?

[wiheto]

The tests run automatically. It failed before cause of travis. But if you update the .travis.yml file with pip installs, then the tests will run /William

…

1 juni 2020 kl. 17:43 skrev Josh L. Espinoza ***@***.***>:  Thanks for looking into this. I will see if there any adjustments I can make. Are there any tests I can run by hand to make sure everything imports and runs correctly before submitting the PR? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or unsubscribe.