Dependencies & compilations on Google colab about coref

[stefanhgm]

Hi @jkkummerfeld

Thanks for putting together the Colab notebook for running the SpanBERT model: https://colab.research.google.com/drive/1SlERO9Uc9541qv6yH26LJz5IM9j7YVra#scrollTo=w5n-XspBFHjD.

We tried to copy and run it today and ran into a new issue when running the pip install -r requirements.txt --log install-log.txt -q. Any idea what have changed from the previous year's version? Could this maybe be a python version issue?

Found existing installation: tensorflow 2.8.0+zzzcolab20220506162203
Uninstalling tensorflow-2.8.0+zzzcolab20220506162203:
  Successfully uninstalled tensorflow-2.8.0+zzzcolab20220506162203
     |████████████████████████████████| 99 kB 9.4 MB/s
     |████████████████████████████████| 1.2 MB 54.1 MB/s
     |████████████████████████████████| 157 kB 57.6 MB/s
     |████████████████████████████████| 6.6 MB 39.7 MB/s
     |████████████████████████████████| 543 kB 55.4 MB/s
     |████████████████████████████████| 52 kB 808 kB/s
     |████████████████████████████████| 2.2 MB 52.9 MB/s
     |████████████████████████████████| 264 kB 61.0 MB/s
     |████████████████████████████████| 889 kB 53.5 MB/s
     |████████████████████████████████| 126 kB 51.8 MB/s
     |████████████████████████████████| 147 kB 54.2 MB/s
     |████████████████████████████████| 50 kB 6.3 MB/s
     |████████████████████████████████| 41 kB 716 kB/s
     |████████████████████████████████| 87 kB 7.0 MB/s
     |████████████████████████████████| 20.3 MB 1.2 MB/s
     |████████████████████████████████| 2.1 MB 31.7 MB/s
     |████████████████████████████████| 1.2 MB 56.5 MB/s
     |████████████████████████████████| 425 kB 29.7 MB/s
     |████████████████████████████████| 71 kB 9.5 MB/s
     |████████████████████████████████| 245 kB 71.4 MB/s
     |████████████████████████████████| 70 kB 7.5 MB/s
     |████████████████████████████████| 56 kB 3.4 MB/s
     |████████████████████████████████| 194 kB 49.8 MB/s
     |████████████████████████████████| 511 kB 53.9 MB/s
     |████████████████████████████████| 59 kB 6.0 MB/s
     |████████████████████████████████| 6.7 MB 48.3 MB/s
     |████████████████████████████████| 27.4 MB 1.2 MB/s
     |████████████████████████████████| 3.1 MB 55.0 MB/s
     |████████████████████████████████| 488 kB 73.8 MB/s
     |████████████████████████████████| 377.1 MB 9.0 kB/s
     |████████████████████████████████| 748.9 MB 587 bytes/s
     |████████████████████████████████| 8.8 MB 61.3 MB/s
     |████████████████████████████████| 322 kB 51.1 MB/s
     |████████████████████████████████| 4.1 MB 52.2 MB/s
     |████████████████████████████████| 46 kB 4.7 MB/s
     |████████████████████████████████| 375 kB 50.4 MB/s
  Building wheel for absl-py (setup.py) ... done
  Building wheel for gast (setup.py) ... done
  Building wheel for h5py (setup.py) ... done
  Building wheel for html5lib (setup.py) ... done
  Building wheel for JPype1 (setup.py) ... done
  Building wheel for mmh3 (setup.py) ... done
  Building wheel for msgpack-python (setup.py) ... done
  Building wheel for psycopg2 (setup.py) ... done
  Building wheel for pycparser (setup.py) ... done
  Building wheel for pyhocon (setup.py) ... done
  Building wheel for wrapt (setup.py) ... done
  Building wheel for PyYAML (setup.py) ... done
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
kapre 0.3.7 requires tensorflow>=2.0.0, which is not installed.
yellowbrick 1.4 requires scikit-learn>=1.0.0, but you have scikit-learn 0.21.0 which is incompatible.
torchtext 0.12.0 requires torch==1.11.0, but you have torch 1.2.0 which is incompatible.
torchaudio 0.11.0+cu113 requires torch==1.11.0, but you have torch 1.2.0 which is incompatible.
tensorflow-probability 0.16.0 requires gast>=0.3.2, but you have gast 0.2.2 which is incompatible.
tensorflow-metadata 1.8.0 requires absl-py<2.0.0,>=0.9, but you have absl-py 0.7.1 which is incompatible.
tensorflow-metadata 1.8.0 requires protobuf<4,>=3.13, but you have protobuf 3.9.1 which is incompatible.
tables 3.7.0 requires numpy>=1.19.0, but you have numpy 1.17.0 which is incompatible.
pywavelets 1.3.0 requires numpy>=1.17.3, but you have numpy 1.17.0 which is incompatible.
pyasn1-modules 0.2.8 requires pyasn1<0.5.0,>=0.4.6, but you have pyasn1 0.4.2 which is incompatible.
pandas 1.3.5 requires numpy>=1.17.3; platform_machine != "aarch64" and platform_machine != "arm64" and python_version < "3.10", but you have numpy 1.17.0 which is incompatible.
pandas 1.3.5 requires python-dateutil>=2.7.3, but you have python-dateutil 2.6.1 which is incompatible.
kapre 0.3.7 requires numpy>=1.18.5, but you have numpy 1.17.0 which is incompatible.
jaxlib 0.3.7+cuda11.cudnn805 requires numpy>=1.19, but you have numpy 1.17.0 which is incompatible.
jax 0.3.8 requires numpy>=1.19, but you have numpy 1.17.0 which is incompatible.
imbalanced-learn 0.8.1 requires scikit-learn>=0.24, but you have scikit-learn 0.21.0 which is incompatible.
googleapis-common-protos 1.56.1 requires protobuf>=3.15.0, but you have protobuf 3.9.1 which is incompatible.
google-colab 1.0.0 requires astor~=0.8.1, but you have astor 0.8.0 which is incompatible.
google-colab 1.0.0 requires six~=1.15.0, but you have six 1.12.0 which is incompatible.
google-api-python-client 1.12.11 requires six<2dev,>=1.13.0, but you have six 1.12.0 which is incompatible.
google-api-core 1.31.5 requires protobuf>=3.12.0; python_version > "3", but you have protobuf 3.9.1 which is incompatible.
google-api-core 1.31.5 requires six>=1.13.0, but you have six 1.12.0 which is incompatible.
flask 1.1.4 requires Jinja2<3.0,>=2.10.1, but you have jinja2 2.10 which is incompatible.
flask 1.1.4 requires Werkzeug<2.0,>=0.15, but you have werkzeug 0.14.1 which is incompatible.
fbprophet 0.7.1 requires python-dateutil>=2.8.0, but you have python-dateutil 2.6.1 which is incompatible.
datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.
bokeh 2.3.3 requires pillow>=7.1.0, but you have pillow 6.1.0 which is incompatible.
albumentations 0.1.12 requires imgaug<0.2.7,>=0.2.5, but you have imgaug 0.2.9 which is incompatible.
/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/__init__.py", line 28, in <module>
    from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/__init__.py", line 83, in <module>
    from tensorflow.python import keras
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/__init__.py", line 26, in <module>
    from tensorflow.python.keras import activations
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/activations.py", line 24, in <module>
    from tensorflow.python.keras.utils.generic_utils import deserialize_keras_object
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/utils/__init__.py", line 34, in <module>
    from tensorflow.python.keras.utils.io_utils import HDF5Matrix
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/utils/io_utils.py", line 29, in <module>
    import h5py
  File "/usr/local/lib/python3.7/dist-packages/h5py/__init__.py", line 36, in <module>
    from ._conv import register_converters as _register_converters
  File "h5py/_conv.pyx", line 1, in init h5py._conv
  File "h5py/h5t.pyx", line 1, in init h5py.h5t
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject
/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/__init__.py", line 28, in <module>
    from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/__init__.py", line 83, in <module>
    from tensorflow.python import keras
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/__init__.py", line 26, in <module>
    from tensorflow.python.keras import activations
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/activations.py", line 24, in <module>
    from tensorflow.python.keras.utils.generic_utils import deserialize_keras_object
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/utils/__init__.py", line 34, in <module>
    from tensorflow.python.keras.utils.io_utils import HDF5Matrix
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/utils/io_utils.py", line 29, in <module>
    import h5py
  File "/usr/local/lib/python3.7/dist-packages/h5py/__init__.py", line 36, in <module>
    from ._conv import register_converters as _register_converters
  File "h5py/_conv.pyx", line 1, in init h5py._conv
  File "h5py/h5t.pyx", line 1, in init h5py.h5t
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject
coref_kernels.cc:3:10: fatal error: tensorflow/core/framework/op.h: No such file or directory
 #include "tensorflow/core/framework/op.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

[jkkummerfeld]

Some sort of version compatibility does seem like the issue, but I don't think it's a python issue as those errors all show python 3.7, which is from 2018.

I don't have time to dig into this at the moment, but in the meantime if you find a solution please share it so I can update the colab!

[jkkummerfeld]

Thanks for mentioning this! I've updated the colab.

[LifeIsStrange]

The colab appears to be broken :/

On local python 3.7 I have psycopg2 dependency error btw

[jkkummerfeld]

I'm happy to incorporate a fix if you are able to identify one!

[LifeIsStrange]

Sorry I'm currently trying to work with alternative libraries such as wl coref or coref hoi