Training with GPU about calamari

[zhangxiangnick]

Okay I think the problem is with TensorFlow 1.9.0. This is what pip installed by default. After I downgrade it to 1.8.0, the training works.

[ChWick]

Thanks for the info! I'll consider an update to tensorflow 1.9 in the next few days!

[erjinzhi2333]

@zhangxiangnick hi,same problem in tensorflow 1.10,it looks like running in colab? How to downgrade tensorflow to 1.8.0? pip3 install tensorflow==1.8.0 will destroy gpu environment.

[zhangxiangnick]

@erjinzhi2333 in my case, pip install tensorflow-gpu==1.8.

[mbtolou]

After Downgrade to tensorflow-gpu==1.8

InvalidArgumentError (see above for traceback): Cannot colocate nodes 'global_norm_1/L2Loss_4' and 'gradients/cudnn_lstm/CudnnRNN_grad/CudnnRNNBackprop' because no device type supports both of those nodes and the other nodes colocated with them.
Colocation Debug Info:
Colocation group had the following types and devices:
CudnnRNNBackprop: GPU
Identity:
L2Loss: CPU

[ChWick]

@mbaghert Can you please append your call, operating system, Cuda and Cudnn version, this might help to debug you issue. Maybe create a new issue on github for this specific issue, too.

[mbtolou]

Windows 10
CUDA v9.0
Cudnn 7

[mbtolou]

Collecting tensorboard<1.9.0,>=1.8.0 (from tensorflow-gpu==1.8)
Using cached https://files.pythonhosted.org/packages/59/a6/0ae6092b7542cfedba6b2a1c9b8dceaf278238c39484f3ba03b03f07803c/tensorboard-1.8.0-py3-none-any.whl
Requirement already satisfied: numpy>=1.13.3 in c:\programdata\anaconda3\lib\site-packages (from tensorflow-gpu==1.8) (1.14.5)
Requirement already satisfied: grpcio>=1.8.6 in c:\programdata\anaconda3\lib\site-packages (from tensorflow-gpu==1.8) (1.15.0)
Requirement already satisfied: absl-py>=0.1.6 in c:\programdata\anaconda3\lib\site-packages (from tensorflow-gpu==1.8) (0.4.1)
Requirement already satisfied: gast>=0.2.0 in c:\programdata\anaconda3\lib\site-packages (from tensorflow-gpu==1.8) (0.2.0)
Requirement already satisfied: six>=1.10.0 in c:\programdata\anaconda3\lib\site-packages (from tensorflow-gpu==1.8) (1.11.0)
Requirement already satisfied: termcolor>=1.1.0 in c:\programdata\anaconda3\lib\site-packages (from tensorflow-gpu==1.8) (1.1.0)
Requirement already satisfied: astor>=0.6.0 in c:\programdata\anaconda3\lib\site-packages (from tensorflow-gpu==1.8) (0.7.1)
Requirement already satisfied: protobuf>=3.4.0 in c:\programdata\anaconda3\lib\site-packages (from tensorflow-gpu==1.8) (3.6.1)
Requirement already satisfied: html5lib==0.9999999 in c:\programdata\anaconda3\lib\site-packages (from tensorboard<1.9.0,>=1.8.0->tensorflow-gpu==1.8) (0.9999999)
Requirement already satisfied: werkzeug>=0.11.10 in c:\programdata\anaconda3\lib\site-packages (from tensorboard<1.9.0,>=1.8.0->tensorflow-gpu==1.8) (0.14.1)
Requirement already satisfied: bleach==1.5.0 in c:\programdata\anaconda3\lib\site-packages (from tensorboard<1.9.0,>=1.8.0->tensorflow-gpu==1.8) (1.5.0)
Requirement already satisfied: markdown>=2.6.8 in c:\programdata\anaconda3\lib\site-packages (from tensorboard<1.9.0,>=1.8.0->tensorflow-gpu==1.8) (2.6.11)
Requirement already satisfied: setuptools in c:\programdata\anaconda3\lib\site-packages (from protobuf>=3.4.0->tensorflow-gpu==1.8) (39.1.0)

[mbtolou]

problem solved by conda install tensorflow-gpu==1.8