training: Cannot convert a symbolic Tensor to a numpy array about calamari

[bertsky]

Ah, found the culprit: It seems that TF has a soft dependency on certain Numpy versions. In my case, downgrading to numpy==1.19.5 for tensorflow==2.4.1 helped avoid the above error.

(I have now worked myself up to a problem with warmstarting... coming up.)

Thanks for your feedback!

[bertsky]

For the sake of completeness:

Here is the SO issue that brought me on the right track.

And this is what I had to do ultimately:

pip install --no-use-pep517 numpy==1.19.5

[bertsky]

Note: I was trying to warmstart from calamari_models_experimental/deep3_fraktur19/0.ckpt.json.

Is there a known workaround?

[andbue]

Sorry, I've never encountered it myself. Since it disappeared in #314 after a reinstall, could you provide some more information on how you installed calamari?

Here is a dockerfile that I use on our hpc cluster, maybe that helps:

FROM nvidia/cuda:11.5.1-cudnn8-runtime-ubuntu18.04
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && \
    apt-get install -y --no-install-recommends software-properties-common && \
    add-apt-repository -y ppa:ubuntu-toolchain-r/ppa && \
    apt-get purge -y --auto-remove software-properties-common && \
    apt-get install -y --no-install-recommends build-essential && \
    apt-get install -y --no-install-recommends python3.7 python3.7-dev python3-pip python3-setuptools python3-wheel python3-distutils && \
    apt clean && \
    python3.7 -m pip --no-cache-dir install --upgrade pip && \
    python3.7 -m pip --no-cache-dir install https://github.com/Calamari-OCR/calamari/tarball/master && \
    apt-get purge -y --auto-remove build-essential && \
    rm -rf /var/lib/apt/lists/*

[bertsky]

I installed from PyPI. Just tried --force-reinstall to no avail. Also tried updating, which gave me 2.2.2 intead of 2.2.1, but still no change (the error persists).

pip log

pip install -U calamari-ocr
Requirement already satisfied: calamari-ocr in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (2.2.1)
Collecting calamari-ocr
  Using cached calamari_ocr-2.2.2-py3-none-any.whl (198 kB)
Requirement already satisfied: python-bidi in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from calamari-ocr) (0.4.2)
Requirement already satisfied: tensorflow>=2.4.0 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from calamari-ocr) (2.4.1)
Requirement already satisfied: lxml in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from calamari-ocr) (4.8.0)
Requirement already satisfied: h5py in /software/ml/h5py/2.10.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages/h5py-2.10.0-py3.7-linux-ppc64le.egg (from calamari-ocr) (2.10.0)
Requirement already satisfied: edit-distance in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from calamari-ocr) (1.0.4)
Requirement already satisfied: tfaip==1.2.6 in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from calamari-ocr) (1.2.6)
Requirement already satisfied: xlsxwriter in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from calamari-ocr) (3.0.3)
Requirement already satisfied: pooch==1.4.0 in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (1.4.0)
Requirement already satisfied: GitPython in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (3.1.27)
Requirement already satisfied: python-Levenshtein in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (0.12.2)
Requirement already satisfied: prettytable in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (3.2.0)
Requirement already satisfied: pandas in /software/ml/SciPy-bundle/2019.10-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (0.25.3)
Requirement already satisfied: dataclasses-json in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (0.5.5)
Requirement already satisfied: imageio in /software/ml/scikit-image/0.16.2-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (2.6.1)
Requirement already satisfied: opencv-python-headless in /home/h1/rosa992c/.local/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (4.2.0+d608d69)
Requirement already satisfied: openpyxl in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (3.0.9)
Requirement already satisfied: nptyping in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (2.0.0)
Requirement already satisfied: xlrd==1.2.0 in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (1.2.0)
Requirement already satisfied: paiargparse==1.1.2 in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (1.1.2)
Requirement already satisfied: scikit-image in /software/ml/scikit-image/0.16.2-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (0.16.2)
Requirement already satisfied: typeguard in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (2.13.3)
Requirement already satisfied: pillow in /home/h1/rosa992c/.local/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (7.2.0)
Requirement already satisfied: adabelief-tf in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (0.2.1)
Requirement already satisfied: tqdm in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (4.64.0)
Requirement already satisfied: tensorflow-addons>=0.12.0 in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from tfaip==1.2.6->calamari-ocr) (0.12.1)
Requirement already satisfied: editdistance in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from paiargparse==1.1.2->tfaip==1.2.6->calamari-ocr) (0.6.0)
Requirement already satisfied: marshmallow-enum<2.0.0,>=1.5.1 in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from dataclasses-json->tfaip==1.2.6->calamari-ocr) (1.5.1)
Requirement already satisfied: typing-inspect>=0.4.0 in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from dataclasses-json->tfaip==1.2.6->calamari-ocr) (0.7.1)
Requirement already satisfied: marshmallow<4.0.0,>=3.3.0 in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from dataclasses-json->tfaip==1.2.6->calamari-ocr) (3.15.0)
Requirement already satisfied: appdirs in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from pooch==1.4.0->tfaip==1.2.6->calamari-ocr) (1.4.4)
Requirement already satisfied: packaging in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from pooch==1.4.0->tfaip==1.2.6->calamari-ocr) (19.1)
Requirement already satisfied: requests in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from pooch==1.4.0->tfaip==1.2.6->calamari-ocr) (2.22.0)
Requirement already satisfied: numpy>=1.16.0 in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (1.21.6)
Requirement already satisfied: keras-preprocessing~=1.1.2 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (1.1.2)
Requirement already satisfied: opt-einsum>=3.3.0 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (3.3.0)
Requirement already satisfied: grpcio~=1.32.0 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (1.32.0)
Requirement already satisfied: absl-py~=0.10 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (0.10.0)
Requirement already satisfied: wrapt>=1.12.1 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (1.12.1)
Requirement already satisfied: termcolor>=1.1.0 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (1.1.0)
Requirement already satisfied: tensorflow-estimator<2.5.0,>=2.4.0 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (2.4.0)
Requirement already satisfied: google-pasta>=0.2 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (0.2.0)
Requirement already satisfied: flatbuffers>=1.12.0 in /software/ml/flatbuffers-python/1.12-GCCcore-8.3.0-Python-3.7.4/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (1.12)
Requirement already satisfied: six>=1.12.0 in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (1.12.0)
Requirement already satisfied: typing-extensions>=3.7.4.2 in /software/ml/typing-extensions/3.7.4.3-GCCcore-8.3.0-Python-3.7.4/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (3.7.4.3)
Requirement already satisfied: gast==0.3.3 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (0.3.3)
Requirement already satisfied: protobuf>=3.9.2 in /software/ml/protobuf-python/3.10.0-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (3.10.0)
Requirement already satisfied: wheel>=0.26 in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (0.37.1)
Requirement already satisfied: astunparse~=1.6.3 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (1.6.3)
Requirement already satisfied: tensorboard~=2.4 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorflow>=2.4.0->calamari-ocr) (2.4.1)
Requirement already satisfied: setuptools in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from protobuf>=3.9.2->tensorflow>=2.4.0->calamari-ocr) (62.1.0)
Requirement already satisfied: markdown>=2.6.8 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorboard~=2.4->tensorflow>=2.4.0->calamari-ocr) (3.3.3)
Requirement already satisfied: werkzeug>=0.11.15 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorboard~=2.4->tensorflow>=2.4.0->calamari-ocr) (1.0.1)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorboard~=2.4->tensorflow>=2.4.0->calamari-ocr) (
0.4.2)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorboard~=2.4->tensorflow>=2.4.0->calamari-ocr) (1.8
.0)
Requirement already satisfied: google-auth<2,>=1.6.3 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from tensorboard~=2.4->tensorflow>=2.4.0->calamari-ocr) (1.24.0)
Requirement already satisfied: colorama>=0.4.0 in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from adabelief-tf->tfaip==1.2.6->calamari-ocr) (0.4.4)
Requirement already satisfied: tabulate>=0.7 in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from adabelief-tf->tfaip==1.2.6->calamari-ocr) (0.8.3)
Requirement already satisfied: gitdb<5,>=4.0.1 in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from GitPython->tfaip==1.2.6->calamari-ocr) (4.0.9)
Requirement already satisfied: et-xmlfile in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from openpyxl->tfaip==1.2.6->calamari-ocr) (1.1.0)
Requirement already satisfied: python-dateutil>=2.6.1 in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from pandas->tfaip==1.2.6->calamari-ocr) (2.8.0)
Requirement already satisfied: pytz>=2017.2 in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from pandas->tfaip==1.2.6->calamari-ocr) (2019.2)
Requirement already satisfied: importlib-metadata in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from prettytable->tfaip==1.2.6->calamari-ocr) (0.22)
Requirement already satisfied: wcwidth in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from prettytable->tfaip==1.2.6->calamari-ocr) (0.1.7)
Requirement already satisfied: networkx>=2.0 in /software/ml/scikit-image/0.16.2-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from scikit-image->tfaip==1.2.6->calamari-ocr) (2.4)
Requirement already satisfied: PyWavelets>=0.4.0 in /software/ml/scikit-image/0.16.2-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from scikit-image->tfaip==1.2.6->calamari-ocr) (1.1.1)
Requirement already satisfied: smmap<6,>=3.0.1 in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from gitdb<5,>=4.0.1->GitPython->tfaip==1.2.6->calamari-ocr) (5.0.0)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard~=2.4->tensorflow>=2.4.0->ca
lamari-ocr) (4.2.0)
Requirement already satisfied: rsa<5,>=3.1.4 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard~=2.4->tensorflow>=2.4.0->calamari-ocr) (4.7)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard~=2.4->tensorflow>=2.4.0->calamari-ocr) (0.2.8)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.4->tensorflow>=2.4.0->calamari-ocr) (1.3.0)
Requirement already satisfied: decorator>=4.3.0 in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from networkx>=2.0->scikit-image->tfaip==1.2.6->calamari-ocr) (4.4.0)
Requirement already satisfied: idna<2.9,>=2.5 in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from requests->pooch==1.4.0->tfaip==1.2.6->calamari-ocr) (2.8)
Requirement already satisfied: certifi>=2017.4.17 in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from requests->pooch==1.4.0->tfaip==1.2.6->calamari-ocr) (2019.9.11)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from requests->pooch==1.4.0->tfaip==1.2.6->calamari-ocr) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from requests->pooch==1.4.0->tfaip==1.2.6->calamari-ocr) (1.25.3)
Requirement already satisfied: mypy-extensions>=0.3.0 in /home/h1/rosa992c/my-kernel/powerai-kernel2/lib/python3.7/site-packages (from typing-inspect>=0.4.0->dataclasses-json->tfaip==1.2.6->calamari-ocr) (0.4.3)
Requirement already satisfied: zipp>=0.5 in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from importlib-metadata->prettytable->tfaip==1.2.6->calamari-ocr) (0.6.0)
Requirement already satisfied: attrs in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from packaging->pooch==1.4.0->tfaip==1.2.6->calamari-ocr) (19.1.0)
Requirement already satisfied: pyparsing>=2.0.2 in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from packaging->pooch==1.4.0->tfaip==1.2.6->calamari-ocr) (2.4.2)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard~=2.4->tensorflow>=2.4.0->calamari-ocr) (0.4.7)
Requirement already satisfied: oauthlib>=3.0.0 in /software/ml/TensorFlow/2.4.1-fosscuda-2019b-Python-3.7.4/lib/python3.7/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.4->tensorflow>=2.4.0->calamari-ocr) (3.1.0)
Requirement already satisfied: more-itertools in /software/ml/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/site-packages (from zipp>=0.5->importlib-metadata->prettytable->tfaip==1.2.6->calamari-ocr) (7.2.0)
Installing collected packages: calamari-ocr
  Attempting uninstall: calamari-ocr
    Found existing installation: calamari-ocr 2.2.1
    Uninstalling calamari-ocr-2.2.1:
      Successfully uninstalled calamari-ocr-2.2.1
Successfully installed calamari-ocr-2.2.2

(Note: I cannot use Docker in my cluster, only Singularity – but that would not help with the different CPU arch.)

[bertsky]

Note: the issue re-appeared for me when I tried training models for Calamari 1 again (v1.0.5, everything else as above). This time numpy==1.19.5 did not help. Neither did installing OpenCV 4.5.5 (instead of 4.2.0).

Meanwhile, I found this TF issue, which seems to explain what we see, but offers no solution to date – only tf.compat.v1.disable_eager_execution() as a workaround.

[bertsky]

FYI, I just tried that in a brutish way, by adding that line to calamari_ocr.ocr.backends.tensorflow_backend right at the top at module level.

That made the error…

TypeError: Keras symbolic inputs/outputs do not implement `op`. You may be trying to pass Keras symbolic inputs/outputs to a TF API that does not register dispatching, preventing Keras from automatically converting the API call to a lambda layer in the Functional Model.

…go away, but now I get…

NotImplementedError: Cannot convert a symbolic Tensor (ReadVariableOp_32:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported

Something's wrong in the state of Denmark here.