TensorFlow package with support of OpenCL (SYCL) about ck-tensorflow

[Chunosov]

Two packages are implemented in PR #78
lib-tensorflow-sycl-cpu is just like *-src-cpu package but taking sources from codeplay.
lib-tensorflow-sycl-opencl should true OpenCL package but it is not built for the time.

[psyhtest]

I don't expect SYCL to work on NVIDIA GPUs (given their "love" for OpenCL, they probably don't support SPIR), but still I've registered their driver as follows:

$ ck detect soft:lib.opencl
...
  Registering in the CK (/usr/local/cuda-9.2/lib64/libOpenCL.so) ...

  Software entry found: lib.opencl (6eb9a57038688ad3)

  Attempting to detect version automatically (if supported) ...

  WARNING: didn't manage to automatically detect software version!

Enter version of this software (for example, 3.21.6-2 or press Enter if default/unknown): 1.2-cuda-9.2
...
$ ck show env --tags=lib,opencl
Env UID:         Target OS:      Bits: Name:          Version:     Tags:

14c726afc70e4ec3        linux-64    64 OpenCL library 1.2-cuda-9.2 64bits,host-os-linux-64,lib,opencl,target-os-linux-64,v1,v1.2,v1.2.0,v1.2.0.9,v1.2.0.9.2

[psyhtest]

I've successfully built both the CPU and OpenCL SYCL versions with the following options:

$ ck install package:lib-tensorflow-sycl-opencl
...
  -----------------------------------                                                                                                                                                                                                                         
  Resolving software dependencies ...

*** Dependency 1 = bazel (Bazel):

    Resolved. CK environment UID = 567fc69c4a74a4ad (detected version 0.11.1)

*** Dependency 2 = python (Python):

More than one environment found for "Python" with tags="compiler,python" and setup={"target_os_bits": "64", "host_os_uoa": "linux-64", "target_os_uoa": "linux-64"}:

 0) python - v3.5.2 (64bits,compiler,host-os-linux-64,lang-python,python,target-os-linux-64,v3,v3.5,v3.5.2 (30b11dbff8e8cafa))

 1) python - v2.7.12 (64bits,compiler,host-os-linux-64,lang-python,python,target-os-linux-64,v2,v2.7,v2.7.12 (06cbb0742c2a1ed0))

Select one of the options for "Python" with tags="compiler,python" and setup={"target_os_bits": "64", "host_os_uoa": "linux-64", "target_os_uoa": "linux-64"} [ hit return for "0" ]: 

    Resolved. CK environment UID = 30b11dbff8e8cafa (detected version 3.5.2)

*** Dependency 3 = compiler.computecpp (ComputeCPP compiler):

    Resolved. CK environment UID = d2703743e98287fb (detected version 0.9.1)

*** Dependency 4 = lib.opencl (OpenCL library):

More than one environment found for "OpenCL library" with tags="lib,opencl" and setup={"target_os_bits": "64", "host_os_uoa": "linux-64", "target_os_uoa": "linux-64"}:

 0) OpenCL library - v1.2-cuda-9.2 (64bits,host-os-linux-64,lib,opencl,target-os-linux-64,v1,v1.2,v1.2.0,v1.2.0.9,v1.2.0.9.2 (14c726afc70e4ec3))

 1) OpenCL library - v1.2-cuda-9.0 (64bits,host-os-linux-64,lib,opencl,target-os-linux-64,v1,v1.2,v1.2.0,v1.2.0.9,v1.2.0.9.0 (3d76e0df4e544eeb))

Select one of the options for "OpenCL library" with tags="lib,opencl" and setup={"target_os_bits": "64", "host_os_uoa": "linux-64", "target_os_uoa": "linux-64"} [ hit return for "0" ]: 

    Resolved. CK environment UID = 14c726afc70e4ec3 (version 1.2-cuda-9.2)
...
--------------------------------
Run configuration script

You have bazel 0.11.1- (@non-git) installed.
Do you wish to build TensorFlow with half types in SYCL support? [y/N]: n
No half types in SYCL support will be enabled for TensorFlow.

Do you wish to build TensorFlow with double types in SYCL support? [Y/n]: n
No double types in SYCL support will be enabled for TensorFlow.

Please specify which bitcode to target when compiling SYCL code. [Default is spir64]: 
...
Recording CK configuration to /home/anton/CK_TOOLS/lib-tensorflow-sycl-opencl-master-compiler.python-3.5.2-lib.opencl-1.2-cuda-9.2-linux-64/ck-install.json ...

Installation path: /home/anton/CK_TOOLS/lib-tensorflow-sycl-opencl-master-compiler.python-3.5.2-lib.opencl-1.2-cuda-9.2-linux-64

Installation time: 2116.11075807 sec.

[psyhtest]

I've created a Bazel cache directory on velociti, and recommend everyone to do the same so not to run out of disk space:

$ sudo mkdir /bazel-cache
$ sudo mkdir /bazel-cache/anton
$ sudo chown anton /bazel-cache/anton
$ sudo chgrp dvdt /bazel-cache/anton
$ touch /bazel-cache/anton/test

Then, you can install using TensorFlow-SYCL using e.g.

$ ck install package:lib-tensorflow-sycl-opencl --env.CK_BAZEL_CACHE_DIR=/bazel-cache/anton

[psyhtest]

You can test it by running:

$ ck install ck-tensorflow:package:tensorflowmodel-alexnet-py
$ ck install ck-env:package:imagenet-2012-val-min-resized
$ ck run ck-tensorflow:program:image-classification-tf-py

The output should be something like:

    * stdout.log

      Model module:
/home/anton/CK_TOOLS/tensorflowmodel-alexnet-py/alexnet-model.py
      Model weights:
/home/anton/CK_TOOLS/tensorflowmodel-alexnet-py/bvlc_alexnet.npy
      Images dir: ../preprocessed/227-0-87.5
      Image list: image_list.txt
      Image size: 227
      Batch size: 1
      Batch count: 1
      Result dir: predictions
      Normalize: False
      Subtract mean: True
      Use model mean: True
      Weights loaded in 0.115668s
      Net created in 0.966986s

      Batch 1 of 1
      Batch loaded in 0.000556s
      Batch classified in 1.577402s


  (post processing: "python
/home/anton/CK_REPOS/ck-tensorflow/script/cached-benchmark/postprocess.py"


--------------------------------
Process results in predictions
---------------------------------------
ILSVRC2012_val_00000001.JPEG - (65) n01751748 sea snake
0.21 - (58) n01737021 water snake
0.13 - (49) n01697457 African crocodile, Nile crocodile, Croco...
0.12 - (44) n01689811 alligator lizard
0.12 - (26) n01630670 common newt, Triturus vulgaris
0.10 - (32) n01644900 tailed frog, bell toad, ribbed toad, tai...
---------------------------------------

Summary:
-------------------------------
Graph loaded in 1.094906s
All images loaded in 0.000556s
All images classified in 1.577402s
Average classification time: 1.577402s
Accuracy top 1: 0.0 (0 of 1)
Accuracy top 5: 0.0 (0 of 1)
--------------------------------


  (reading fine grain timers from tmp-ck-timer.json ...)

{
  "accuracy_top1": 0.0,
  "accuracy_top5": 0.0,
  "avg_fps": 0.6339538547087311,
  "avg_time_ms": 1577.401876449585,
  "batch_size": 1,
  "batch_time_ms": 1577.401876449585,
  "execution_time": 1.577401876449585,
  "execution_time_sum": 2.674532175064087,
  "frame_predictions": [
    {
      "accuracy_top1": "no",
      "accuracy_top5": "no",
      "class_correct": 65,
      "class_topmost": 58,
      "file_name": "ILSVRC2012_val_00000001.JPEG"
    }
  ],
  "images_load_time_avg_s": 0.0005559921264648438,
  "images_load_time_s": 0.0005559921264648438,
  "net_create_time_s": 0.9669859409332275,
  "prediction_time_avg_s": 1.577401876449585,
  "prediction_time_total_s": 1.577401876449585,
  "setup_time_s": 1.0949060916900635,
  "test_time_s": 1.5796260833740234,
  "weights_load_time_s": 0.11566805839538574
}

Execution time: 1.577 sec.

[psyhtest]

However, running with the TensorFlow-OpenCL version fails with the NVIDIA implementation because it wants... cuBLAS 8.0!

    * stderr.log

      Traceback (most recent call last):
        File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
          from tensorflow.python.pywrap_tensorflow_internal import *
        File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
          _pywrap_tensorflow_internal = swig_import_helper()
        File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
          _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
        File "/usr/lib/python3.5/imp.py", line 242, in load_module
          return load_dynamic(name, filename, file)
        File "/usr/lib/python3.5/imp.py", line 342, in load_dynamic
          return _load(spec)
      ImportError: libcublas.so.8.0: cannot open shared object file: No such file or directory
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "../classify.py", line 13, in <module>
          import tensorflow as tf
        File "/usr/local/lib/python3.5/dist-packages/tensorflow/__init__.py", line 24, in <module>
          from tensorflow.python import *
        File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/__init__.py", line 49, in <module>
          from tensorflow.python import pywrap_tensorflow
        File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 72, in <module>
          raise ImportError(msg)
      ImportError: Traceback (most recent call last):
        File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
          from tensorflow.python.pywrap_tensorflow_internal import *
        File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
          _pywrap_tensorflow_internal = swig_import_helper()
        File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
          _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
        File "/usr/lib/python3.5/imp.py", line 242, in load_module
          return load_dynamic(name, filename, file)
        File "/usr/lib/python3.5/imp.py", line 342, in load_dynamic
          return _load(spec)
      ImportError: libcublas.so.8.0: cannot open shared object file: No such file or directory
      
      
      Failed to load the native TensorFlow runtime.
      
      See https://www.tensorflow.org/install/install_sources#common_installation_problems
      for some common reasons and solutions.  Include the entire stack trace
      above this error message when asking for help.

We should be able to workaround this by providing a dependency on CUDA 8.0, but then it would not work on non-NVIDIA machines. Hmm.

[psyhtest]

I've managed to work around this by:

1. Detecting gcc v4.9 (cuda v8 doesn't work with gcc > v5.4)

$ ck detect soft:compiler.gcc --full_path=/usr/bin/gcc-4.9

2. Detecting cuda v8:

$ ck detect soft:compiler.cuda --full_path=/usr/local/cuda-8.0/bin/nvcc
...
$ ck show env --tags=cuda,v8
Env UID:         Target OS: Bits: Name:                Version: Tags:

5be67e833ceae283   linux-64    64 Nvidia CUDA Compiler 8.0.61   64bits,compiler,cuda,host-os-linux-64,lang-c-cuda,lang-cpp-cuda,target-os-linux-64,v8,v8.0,v8.0.61

3. Running under the cuda v8 virtual env:

$ ck virtual env:5be67e833ceae283
$ ck run ck-tensorflow:program:image-classification-tf-py

(Please somebody remind me what's the handy single-line syntax for that?)

[ens-lg4]

You can probably combine the two lines together, using tags for more portability:

$ ck virtual env --tags=cuda,compiler,v8 --shell_cmd="ck run ck-tensorflow:program:image-classification-tf-py"

[Chunosov]

As we can see from the stacktrace, a pip-installed package is used instead of CK-package - File "/usr/local/lib/python3.5/dist-packages/tensorflow..., and as we have GPU-package installed into Python 3.5 (see tensorflow_gpu-1.4.1.dist-info), so as it requires CUDA.

Conversely, Python 2.7 haven't pip-installed TF package and the program with lib-tensorflow-sycl-cpu-master-compiler.python-2.7.12-linux-64 runs without errors.

I've opened related issue #79: we have to make program load ck-package modules even though pip-packages is installed.

[psyhtest]

Suspended back then as lost interest in supporting SYCL, and now finally closing.