benoitsteiner / trisycl

Some experiments around OpenCL SYCL

triSYCL

triSYCL is an implementation test-bed to experiment with the specification of the OpenCL SYCL C++ layer and to give feedback to the Khronos OpenCL SYCL and OpenCL C++ kernel language committees.

This SYCL implementation is only based on C++1z (2017?) and OpenMP with execution on the CPU right now. So there is almost nothing related to OpenCL or SPIR yet. But since in SYCL there is a host fall-back, this can be seen as an implementation of this fall-back too...

The parallel kernels can be executed in parallel on the CPU with OpenMP in the first range dimension, if compiled with OpenMP support.

For legal reasons, the specification used for this open source project is the published current provisional specification and not the last one currently discussed in the Khronos OpenCL SYCL committee. If you are a Khronos member, you can access to https://gitlab.khronos.org/sycl/triSYCL where you can find more futuristic branches.

This is provided as is, without any warranty, with the same license as LLVM/Clang.

Technical lead: Ronan at keryell point FR

OpenCL SYCL

OpenCL SYCL is a single-source C++14-based DSEL (Domain Specific Embedded Language) aimed at facilitating the programming of heterogeneous accelerators by leveraging the OpenCL language and concepts.

OpenCL SYCL is developed inside the Khronos OpenCL SYCL committee and thus, for more information on SYCL, look at http://www.khronos.org/opencl/sycl

Why you could use SYCL

SYCL has a lot of interesting advantages compared to plain OpenCL or other approaches:

• SYCL is an open standard from Khronos with a working committee (you can contribute!) and we can expect several implementations (commercial or open source) on many platforms soon, ranging from GPU, APU, FPGA, DSP... down to plain CPU;
• it offers a single-source C++ programming model that allows taking advantage of the modern C++14 superpower, unifying both the host and accelerator sides. For example it is possible to write generic accelerated functions on the accelerators in a terse way by using (variadic) templates, meta-programming and lambda expressions. This allows to build templated libraries such as Eigen or TensorFlow in a seamless way;
• SYCL abstracts the concepts behind OpenCL and provides higher-level concepts such as tasks (or command group in OpenCL SYCL jargon) that allow the runtime to take advantage of a more task graph-oriented view of the computations. This allows lazy data transfers between accelerators and host or to use platform capabilities such as OpenCL 2 SVM or HSA for sharing data between host and accelerators;
• the entry cost of the technology is zero since, after all, an existing OpenCL or C++ program is a valid SYCL program;
• the exit cost is low since it is pure C++ without any extension or #pragma. Retargeting the SYCL classes and functions to use other frameworks such as OpenMP 4 or C++AMP is feasible without rewriting a new compiler for example;
• easier debugging
  • since all memory accesses to array parameters in kernels go through accessors, all the memory bound checks can be done in them if needed;
  • since there is a pure host mode, the kernel code can be run also on the host and debugged using the usual tools and use any system (such stdio or iostream...) or data libraries (for nice data visualization);
  • since the kernel code is C++ code even when run on an accelerator, instrumenting the code by using special array classes or overloading some operators allows deep intrusive debugging or code analysis without changing the algorithmic parts of the code;
• SYCL is high-level standard C++14 without any extension, that means that you can use your usual compiler and the host part can use at the same time some cool and common extensions such as OpenMP, OpenHMPP, OpenACC,... or libraries such as MPI or PGAS Coarray++, be linked with other parts written in other languages (Fortran...). Thus SYCL is already Exascale-ready!

• even if SYCL hides the OpenCL world by default, it inherits from all the OpenCL world:

  • same interoperability as the OpenCL underlying platform: Vulkan, OpenGL, DirectX...
  • access to all the underlying basic OpenCL objects behind the SYCL abstraction for interoperability and hard-core optimization;
  • construction of SYCL objects from basic OpenCL objects to add some SYCL parts to an existing OpenCL application;
  • so it provides a continuum from higher-level programming à la C++AMP or OpenMP 4 down to low-level OpenCL, according to the optimization needs, from using simple OpenCL intrinsics or vector operation from the cl::sycl namespace down to providing a real OpenCL kernel to be executed without requiring all the verbose usual OpenCL host API.

  This OpenCL seamless integration plus the gradual optimization features are perhaps the most compelling arguments for SYCL because it allows high-level programming simplicity without giving-up hard-core performance when needed;

• since the SYCL task graph execution model is asynchronous, this can be used by side effect to overcome some underlying OpenCL implementation limitations. For example, some OpenCL stacks may have only in-order execution queues or even synchronous (blocking) ND-range enqueue, or some weird constrained mapping between OpenCL programmer level queue(s) and the hardware queues.

  In this case, a SYCL implementation can deal with this, relying for example on multiple host CPU threads, multiple thread-local-storage (TLS) queues, its own scheduler, etc. atop the limited OpenCL stack to provide computation and communication overlap in a natural pain-free fashion. This relieves the programmer to reorganize her application to work around these limitation, which can be quite a cumbersome work.

for introduction material on the interest of DSEL in this area, look for example at these articles:

• Domain-specific Languages and Code Synthesis Using Haskell, Andy Gill. May 6, 2014 in ACM Queue and Communications of the ACM.
• Design Exploration through Code-generating DSLs, Bo Joel Svensson, Mary Sheeran and Ryan Newton. May 15, 2014 in ACM Queue and Communications of the ACM.

Some presentations and publications related to SYCL

By reverse chronological order:

• Post-modern C++ abstractions for FPGA & heterogeneous computing with OpenCL SYCL & SPIR-V, Ronan Keryell. ANL REFORM 2016: Workshop on FPGAs for scientific simulation and data analytics, Argone National Labs. January 22, 2016.
• From modern FPGA to high-level post-modern C++ abstractions for heterogeneous computing with OpenCL SYCL & SPIR-V, Ronan Keryell. HiPEAC WRC 2016: Workshop on Reconfigurable Computing. Prague, January 19, 2016.
• HiPEAC 2016 tutorial on SYCL: Khronos SYCL for OpenCL. HiPEAC 2016, Prague, January 18, 2016.
• A Tutorial on Khronos SYCL for OpenCL at IWOCL 2015. Stanford, May 12, 2015.
• Modern C++, OpenCL SYCL & OpenCL CL2.hpp, Ronan Keryell (AMD & Khronos OpenCL SYCL Working Group Member). November 18, 2014, Presentation at SuperComputing 2014, OpenCL BoF, New Orleans, USA.
• Implementing the OpenCL SYCL Shared Source C++ Programming Model using Clang/LLVM, Gordon Brown. November 17, 2014, Workshop on the LLVM Compiler Infrastructure in HPC, SuperComputing 2014
• SYCL Specification --- SYCL integrates OpenCL devices with modern C++, Khronos OpenCL Working Group — SYCL subgroup. Editors: Lee Howes and Maria Rovatsou. Version 1.2, Revision 2014-09-16.
• OpenCL 2.0, OpenCL SYCL & OpenMP 4, open standards for heterogeneous parallel programming, Ronan Keryell (AMD & Khronos OpenCL Working Group Member). July 3, 2014, Presentation at the Meetup of the High Performance Computing & Supercomputing Group of Paris.
• OpenCL 2.0, OpenCL SYCL & OpenMP 4, open standards for heterogeneous parallel programming, Ronan Keryell (AMD & Khronos OpenCL Working Group Member). July 2, 2014, Presentation at Forum Ter@tec: Calcul scientifique & Open Source : pratiques industrielles des logiciels libres.
• The Future of Accelerator Programming in C++, Sebastian Schaetz, May 18, 2014. Presentation at C++Now 2014.
• SYCL : Abstraction Layer for Leveraging C++ and OpenCL, Maria Rovatsou (Codeplay & Khronos OpenCL Working Group Member). May 12-13, 2014, IWOCL 2014.
• Building the OpenCL ecosystem - SYCL for OpenCL, Lee Howes (Senior Staff Engineer at Qualcomm & Khronos OpenCL Working Group Member). April 21, 2014, HPC & GPU Supercomputing Group of Silicon Valley.
• SYCL 1.2: Unofficial High-Level Overview, AJ Guillon (Khronos OpenCL Working Group Member). March 19, 2014. Video.
• SYCL for OpenCL, Andrew Richards (CEO Codeplay & Chair SYCL Working group). March 19, 2014, GDC 2014.
• Fusing GPU kernels within a novel single-source C++ API, Ralph Potter, Paul Keir, Jan Lucas, Maurico Alvarez-Mesa, Ben Juurlink and Andrew Richards. January 20, 2014, LPGPU Workshop on Power-Efficient GPU and Many-core Computing (PEGPUM 2014).
• Fusing GPU kernels within a novel single-source C++ API, Ralph Potter, Paul Keir, Jan Lucas, Mauricio Alvarez-Mesa, Ben Juurlink, Andrew Richards. November 18, 2013, Intel Compiler, Architecture and Tools Conference.

There are also many interesting articles in the publication list from Codeplay.

Some implementations

Some other known implementations:

• CodePlay has an implementation based on OpenCL SPIR with Clang/LLVM http://codeplay.com/products
• SYCL-GTX https://github.com/ProGTX/sycl-gtx

Related projects

• Boost.Compute
• VexCL
• ViennaCL
• C++ Parallelism TS https://github.com/cplusplus/parallelism-ts
• C++ Concurrency TS https://github.com/cplusplus/concurrency_ts
• Bolt
• Thrust
• C++AMP
• HCC https://bitbucket.org/multicoreware/hcc/wiki/Home
• Intel SPMD Program Compiler https://ispc.github.io/
• Intel Lab's iHRC https://github.com/IntelLabs/iHRC
• CUDA
• Metal

OpenCL triSYCL code documentation

The documentation of the triSYCL implementation itself can be found in http://amd.github.io/triSYCL/Doxygen/triSYCL/html and http://amd.github.io/triSYCL/Doxygen/triSYCL/triSYCL-implementation-refman.pdf

Installation

Only Clang 3.6+ or GCC 4.9+, Boost.MultiArray (which adds to C++ the nice Fortran array semantics and syntax), Boost.Operators and a few other Boost libraries are needed. If you use the debug mode or OpenMP, this works only with GCC 4.9 since current Clang version does not fully support OpenMP yet or some C++14 with -g.

To install them on latest Linux Debian/unstable (this should work on latest Ubuntu too):

sudo apt-get install clang-3.6 g++4.9 libboost-dev

There is nothing else to do for now to use the include files from triSYCL include directory when compiling a program. Just add a -I.../include option and -std=c++1y when compiling.

Also use -fopenmp if you want to use multicore parallelism on the CPU.

Examples and tests

There are simple examples and tests in the tests directory. Look at tests/README.rst description.

Generating the Doxygen documentation

In the top directory, run

make

that will produce tmp/Doxygen/SYCL with the API documentation and tmp/Doxygen/triSYCL with the documented triSYCL implementation source code.

To publish the documentation on GitHub:

make publish

and finish as explained by the make output.

Possible futures

Some ideas of future developments where you can contribute too: :-)

• finish implementation of basic classes without any OpenCL support;
• move to CMake for better portability (status: Lee Howes has made it on 1 of his private branches. To be merged...);
• improve the test infrastructure (for example move to something more standard with Boost.Test. Status: started);
• use the official OpenCL SYCL test suite to extend/debug/validate this implementation;
• add vector swizzle support by following ideas from https://github.com/gwiazdorrr/CxxSwizzle http://glm.g-truc.net http://jojendersie.de/performance-optimal-vector-swizzling-in-c http://www.reedbeta.com/blog/2013/12/28/on-vector-math-libraries ;
• add first OpenCL support with kernels provided only as strings, thus avoiding the need for a compiler. Could be based on other libraries such as Boost.Compute, VexCL, ViennaCL... (status: started with Boost.Compute);
• make an accelerator version based on OpenMP 4 accelerator target, OpenHMPP or OpenACC;

• make an accelerator version based on wrapper classes for the C++AMP Open Source compiler.

  Extend the current C++AMP OpenCL HSA or SPIR back-end runtime to expose OpenCL objects needed for the SYCL OpenCL interoperability. This is probably the simpler approach to have a running SYCL compiler working quickly.

  The main issue is that since C++AMP support is not yet integrated in the official trunk, it would take a long time to break things down and be reviewed by the Clang/LLVM community;

• extend runtime and Clang/LLVM to generate OpenCL/SPIR from C++ single-source kernels, by using OpenMP outliner. Starting from an open source OpenCL C/C++ compiler sounds great;

• alternatively develop a Clang/LLVM-based version, recycling the outliner which is already present for OpenMP support and modify it to generate SPIR. Then build a specific version of libiomp5 to use the OpenCL C/C++ API to run the offloaded kernels. See https://drive.google.com/file/d/0B-jX56_FbGKRM21sYlNYVnB4eFk/view and the projects https://github.com/clang-omp/libomptarget for https://github.com/clang-omp/llvm_trunk and https://github.com/clang-omp/clang_trunk

  This approach may require more work than the C++AMP version but since it is based on the existing OpenMP infrastructure Intel spent a lot of time to upstream through the official code review process, at the end it would require quite less time for up-streaming, if this is the goal.

  OpenMP4 in Clang/LLVM is getting momentum and making lot of progress backed by Intel, IBM, AMD... so it sounds like the way to go;

• recycle the GCC https://gcc.gnu.org/wiki/Offloading OpenMP/OpenACC library infrastructure to construct an OpenCL interoperability API and adapt the triSYCL classes to leverage OpenMP/OpenACC;
• add OpenCL 2.x support with SYCL 2.x;
• since SYCL is a pretty general programming model for heterogeneous computing, if the OpenCL compatibility layer is not required, some other back-ends could be written besides the current OpenMP one: CUDA, RenderScript, etc.
• SYCL concepts (well, classes) can also be ported to some other languages to provide heterogeneous support: SYJSCL, SYCamlCL, SYJavaCL... It is not clear yet if SYFortranCL is possible with Fortran 2008 or 2015+.