viennacl / viennaclbench-dev

A GUI for displaying benchmark results obtaind for operations in ViennaCL

ViennaCL Benchmark Readme

This is the developer repository of ViennaCL Benchmark GUI.

About ViennaCL Benchmark

The ViennaCL Benchmark GUI is a fresh project aiming to create a cross-platform standalone benchmarking program for the ViennaCL library. It will utilize the capabilities of ViennaCL to perform hardware benchmarking and collection of users' test results.
The project is being developed by Namik Karovic as a part of 2014 Google Summer of Code.
ViennaCL was developed under the aegis of the Institute for Microelectronics at the Vienna University of Technology.
For any technical questions related to ViennaCL, please use our mailing list.

Requirements

The following packages need to be available on your system (similar versions may work): - Qt4 (tested with 4.8.5) or Qt5 (tested with 5.2.0 and 5.3.0)
- Qt Creator with QMake; alternatively, a CMake build is now available
- CMake 2.8.11 (Qt5) or 2.6 (Qt4)
- OpenCL SDK (tested with AMD APP SDK 2.9) Other dependencies are libarchive, zlib, and ViennaCL, which are cloned automatically through git and don't require extra preparation.

System Environment Variables Prerequisites

Qt

When building with CMake, you should have Qt's bin folder in your path. Also, you need to set the environment `CMAKE_PREFIX_PATH` to the respective build configuration of Qt, for example to `C:\Qt5\5.3\msvc2012_opengl`

When using QMake, have QT_QMAKE_EXECUTABLE system environment variable defined to point to qmake.exe location of your local Qt bin folder.

OpenCL

If you wish to enable OpenCL usage in the benchmark, these steps should be followed:
* Make sure you have defined the OPENCLROOT system environment variable. It should point to the root folder (containing the bin/, include/ and lib/ folders) of your OpenCL SDK (e.g. C:\AMDAPPSDK). This variable is used to automatically detect and enable OpenCL.
* Make sure OpenCL.dll can be found in your system environment (AFAIK Windows' system32/ folder may contain a copy of OpenCL.dll; if this is not the case, add the bin/ folder of your OpenCL SDK to the system PATH - e.g. add the folder C:\AMDAPPSDK\bin to your PATH variable)

Build Process

Qt Creator with the qmake build system (Windows):

1. open `ViennaCL_Benchmark.pro` with Qt Creator
2. when prompted to configure project, choose a Qt4 or Qt5 kit (tested with 4.8.5, 5.2.0 and 5.3.0) and click Configure Project
3. you can now run the project (Ctrl+R)

CMake build system - outside Qt Creator (Windows)

Default CMakeLists.txt now supports both Qt4 and Qt5, with OpenCL support. The build will default to Qt5, but if you want to build with Qt4 set USE_QT5 variable to OFF. Before you launch the CMake GUI, please make sure that the environment `CMAKE_PREFIX_PATH` to the respective build configuration of Qt, for example to `C:\Qt5\5.3\msvc2012_opengl`. Also, make sure your Qt-libraries can be found by the system (see above).

Preparing packages (only required once)

1. Navigate into your project folder with the Git Bash and run `git submodule update --init` to clone the external repository viennacl-dev. 2. Navigate to external and clone the release version of libarchive (we've seen build problems with the latest master branch): `git clone https://github.com/libarchive/libarchive`
-If you are on Windows you'll have to use Libarchive v3.1.2 (the latest master build fails on Windows). Navigate to external/libarchive and checkout version 3.1.2: `cd libarchive && git checkout v3.1.2` 3. Navigate back to external and clone zlib: `git clone https://github.com/madler/zlib`
4. Use CMake to build zlib within its own source directory `../external/zlib` *(the build directory is the same as the source directory)*. Use the CMake GUI (if you wish to link zlib statically on Windows, make sure to turn CMAKE_GNUtoMS ON): click on 'Configure' twice and then on 'Generate', open the Visual Studio project file and compile the `zlibstatic` target. If you are using MinGW, call `make`( or `mingw32-make` on Windows ). 5. Use CMake to build libarchive within its own source directory `../external/libarchive` *(the build directory is the same as the source directory)*. Use CMake GUI (if you wish to link libarchive statically on Windows, make sure to turn CMAKE_GNUtoMS ON): set ZLIB_INCLUDE_DIR to `../external/zlib` and `ZLIB_LIBRARY` to the `zlibstatic.lib` (`libzlib.lib` on Windows) in either `../external/zlib` or one of the subfolders (typically Release or Debug)

Building the GUI

2. Open CMakeLists.txt in the main repository folder with the CMake GUI
3. Create a subfolder build/ and select this as your build directory. This ensures that the repository tree does not get cluttered with build files. 4. Click on 'Configure' and select your compiler. This should match the configuration you use with Qt when specifying the environment variable `CMAKE_PREFIX_PATH`. In the above example the compiler should be "Visual Studio 11". Note that by default Qt5 is shipped in 32bit, so you should *not* select 64-bit builds here. 5. Click on "Configure" and wait for all checks to complete (might take a little). 6. If CMake cannot detect the path to OpenCL, specify it manually now. 7. Click on "Configure" again. You should not get any errors now. 8. Click on "Generate". 9. Open the generated project file in build/ with your version of Visual Studio. If you use MinGW, navigate to the build folder in the terminal.
10. Compile the target `ViennaCL_Benchmark_Gui` in Visual Studio, or type 'make' in your MinGW terminal.
11. You may now run the executable `ViennaCL_Benchmark`. Depending on your compiler, this may be located in the subfolder `Debug` or `Release` of your build directory.

CMake build system - outside Qt Creator (Linux)

Make sure you have `libqt5-dev` or `libqt4-dev` installed through your package manager. When using OpenCL you should also have `opencl-headers` and a proprietary graphics driver installed.
1. Change into the viennacl-benchmark-gui folder
2. Run `git submodule update --init` to clone the external repository viennacl-dev. 3. Grab and build zlib: `cd external && git clone https://github.com/madler/zlib && cd zlib && cmake . && make` 4. Grab and build libarchive: `cd .. && git clone https://github.com/libarchive/libarchive && cd libarchive && cmake . -DENABLE_BZip2=Off && make` 5. Create the build folder in the main repository folder and change into it: `cd ../.. && mkdir build && cd build`
6. Run CMake. When using Qt5: `cmake ..` Use `cmake .. -DUSE_QT5=Off` if you only have Qt4 available on your system.
7. Build everyting: `make` 8. Run the GUI: `./ViennaCL_Benchmark`

Important Notice

-When not using OpenCL: Blas3 benchmark is extremely slow. Takes about 5 minutes to complete, depending on the machine. Don't run it if you can't wait;

Additional Info

The main developer Namik is running a devblog dedicated to this project and tries to update it as much as possible. Check it out for detailed reports and updates on the development process.

viennaclbench-dev's Issues

Enhance system info screen.

Certainly won't win a design price in its current shape...

Qt5 Build Process

When making the following modification to the *.pro file:

Add include folders

INCLUDEPATH += "C:\Users\matth_000\AMD APP SDK\2.9\include"
INCLUDEPATH += C:\ViennaCL-1.5.1
INCLUDEPATH += C:\boost_1_54_0

DEPENDPATH += "C:\Users\matth_000\AMD APP SDK\2.9\include"
DEPENDPATH += C:\ViennaCL-1.5.1
DEPENDPATH += C:\boost_1_54_0

The following build errors occur:

Issue:
👎 error: D8021 : invalid numeric argument '/Wno-unused-local-typedefs'
👎 error: D8021 : invalid numeric argument '/Wno-unused-local-typedefs'
👎 error: D8021 : invalid numeric argument '/Wno-unused-local-typedefs'

Logs:
14:19:31: Running steps for project ViennaCL_Benchmark...
14:19:31: Configuration unchanged, skipping qmake step.
14:19:31: Starting: "C:\Qt\Tools\QtCreator\bin\jom.exe" install
C:\Qt\Tools\QtCreator\bin\jom.exe -f Makefile.Release install
C:\Qt\5.3\msvc2013_64\bin\uic.exe ..\viennacl-benchmark-gui-master\src\mainwindow.ui -o ui_mainwindow.h
cl -c -nologo -Zm200 -Zc:wchar_t -FS -Wno-unused-local-typedefs -Wno-unused-parameter -DVIENNACL_WITH_OPENCL -O2 -MD -GR -W3 -w34100 -w34189 -EHsc -DUNICODE -DWIN32 -DWIN64 -DQT_NO_DEBUG -DQT_WIDGETS_LIB -DQT_GUI_LIB -DQT_CORE_LIB -DQT_OPENGL_ES_2 -DQT_OPENGL_ES_2_ANGLE -DNDEBUG -I"..\Users\matth_000\AMD APP SDK\2.9\include" -I"..\ViennaCL-1.5.1" -I"..\boost_1_54_0" -I"C:\viennacl-benchmark-gui-master..........\AMDAPPSDK\2.9\lib\x86_64" -I"..\Qt\5.3\msvc2013_64\include" -I"..\Qt\5.3\msvc2013_64\include\QtWidgets" -I"..\Qt\5.3\msvc2013_64\include\QtGui" -I"..\Qt\5.3\msvc2013_64\include\QtANGLE" -I"..\Qt\5.3\msvc2013_64\include\QtCore" -I"release" -I"." -I"." -I"..\Qt\5.3\msvc2013_64\mkspecs\win32-msvc2013" -Forelease\ @C:\Users\MATTH_~1\AppData\Local\Temp\main.obj.5480.157.jom
cl : Command line error D8021 : invalid numeric argument '/Wno-unused-local-typedefs'
cl -c -nologo -Zm200 -Zc:wchar_t -FS -Wno-unused-local-typedefs -Wno-unused-parameter -DVIENNACL_WITH_OPENCL -O2 -MD -GR -W3 -w34100 -w34189 -EHsc -DUNICODE -DWIN32 -DWIN64 -DQT_NO_DEBUG -DQT_WIDGETS_LIB -DQT_GUI_LIB -DQT_CORE_LIB -DQT_OPENGL_ES_2 -DQT_OPENGL_ES_2_ANGLE -DNDEBUG -I"..\Users\matth_000\AMD APP SDK\2.9\include" -I"..\ViennaCL-1.5.1" -I"..\boost_1_54_0" -I"C:\viennacl-benchmark-gui-master..........\AMDAPPSDK\2.9\lib\x86_64" -I"..\Qt\5.3\msvc2013_64\include" -I"..\Qt\5.3\msvc2013_64\include\QtWidgets" -I"..\Qt\5.3\msvc2013_64\include\QtGui" -I"..\Qt\5.3\msvc2013_64\include\QtANGLE" -I"..\Qt\5.3\msvc2013_64\include\QtCore" -I"release" -I"." -I"." -I"..\Qt\5.3\msvc2013_64\mkspecs\win32-msvc2013" -Forelease\ @C:\Users\MATTH_~1\AppData\Local\Temp\qcustomplot.obj.5480.157.jom
cl -c -nologo -Zm200 -Zc:wchar_t -FS -Wno-unused-local-typedefs -Wno-unused-parameter -DVIENNACL_WITH_OPENCL -O2 -MD -GR -W3 -w34100 -w34189 -EHsc -DUNICODE -DWIN32 -DWIN64 -DQT_NO_DEBUG -DQT_WIDGETS_LIB -DQT_GUI_LIB -DQT_CORE_LIB -DQT_OPENGL_ES_2 -DQT_OPENGL_ES_2_ANGLE -DNDEBUG -I"..\Users\matth_000\AMD APP SDK\2.9\include" -I"..\ViennaCL-1.5.1" -I"..\boost_1_54_0" -I"C:\viennacl-benchmark-gui-master..........\AMDAPPSDK\2.9\lib\x86_64" -I"..\Qt\5.3\msvc2013_64\include" -I"..\Qt\5.3\msvc2013_64\include\QtWidgets" -I"..\Qt\5.3\msvc2013_64\include\QtGui" -I"..\Qt\5.3\msvc2013_64\include\QtANGLE" -I"..\Qt\5.3\msvc2013_64\include\QtCore" -I"release" -I"." -I"." -I"..\Qt\5.3\msvc2013_64\mkspecs\win32-msvc2013" -Forelease\ @C:\Users\MATTH_~1\AppData\Local\Temp\benchmark_vector.obj.5480.188.jom
cl : Command line error D8021 : invalid numeric argument '/Wno-unused-local-typedefs'
jom: C:\build-ViennaCL_Benchmark-Desktop_Qt_5_3_0_MSVC2013_64bit-Release\Makefile.Release [release\qcustomplot.obj] Error 2
cl : Command line error D8021 : invalid numeric argument '/Wno-unused-local-typedefs'
jom: C:\build-ViennaCL_Benchmark-Desktop_Qt_5_3_0_MSVC2013_64bit-Release\Makefile.Release [release\benchmark_vector.obj] Error 2
jom: C:\build-ViennaCL_Benchmark-Desktop_Qt_5_3_0_MSVC2013_64bit-Release\Makefile.Release [release\main.obj] Error 2
jom: C:\build-ViennaCL_Benchmark-Desktop_Qt_5_3_0_MSVC2013_64bit-Release\Makefile [release-install] Error 2
14:19:31: The process "C:\Qt\Tools\QtCreator\bin\jom.exe" exited with code 2.
Error while building/deploying project ViennaCL_Benchmark (kit: Desktop Qt 5.3.0 MSVC2013 64bit)
When executing step 'Make'
14:19:31: Elapsed time: 00:01.

#include "qcustomplot.h" does not compile in VS2013

include "qcustomplot.h" does not compile in VS2013 on line 18 of mainwindow.h. To get it to compile, #include <qcustomplot.h> needs to be changed to

include "qcustomplot.h" to let the compiler know the header is in an include folder. That should be cross a cross platform compatible change and I would recommend it.

Export benchmark results to file

An automated post-processing of benchmark results will benefit from the ability to output benchmark results to simple text files. CSV and possibly JSON are suitable file formats.

Resolve code duplication

Basic and expert view are implemented by sharing only a few percent of code. However, this can also be implemented such that only a few percent of code is not shared. Let's make our maintenance work easy!

MatrixMarket Browser: Group selection does not work.

If I click on a matrix group or on 'details' in 'Browse by matrix group' , nothing happens.

Device and compute backend selection for benchmarks

Currently there is no way to specify the (OpenCL) device and no way to switch between a standard CPU backend (using either single-threaded execution or OpenMP) and OpenCL.

CMake Process

I just downloaded and attempted to the the latest version running using CMake. However, the make-file references a few things which I believe would need to be in my %PATH but I didn't want to do that as I change versions frequently. I tried to hard code them:

set(Boost_USE_STATIC_LIBS OFF)
set(Boost_USE_MULTITHREADED ON)
set(Boost_USE_STATIC_RUNTIME OFF)

find_package(Boost) _commented out in favor of the line below_

include_directories("C:/boost_1_54_0")

find OpenCL

find_package(OpenCL REQUIRED) **commented out in favor of the lines below **

include_directories(${OPENCL_INCLUDE_DIR})
link_directories(${OPENCL_LIBRARIES})
link_directories("C:/Users/matth_000/AMD APP SDK/2.9/lib/x86_64")

After these changes cmake appeared to run fine, although I was unsuccessful in getting the program to build in qt5. Perhaps I am missing some steps in the instructions.

If you could please provide more detailed steps for the cmake build process and provide either directions on how to use the find_package function or leave it as a build parameter it would be helpful.

Eliminate terminal output

Currently there is a lot of noise sent to the terminal. A GUI should not write anything to the terminal, since it is a GUI...

Replace Webkit engine for MatrixMarket

The WebKit engine is fairly heavy-weight and cannot be linked statically. Using a static list of matrices will be faster and simplify deployment a lot.

Fix detailed plots collapsing on y-axis.

Detailed plots that contain a legend (blas3, copy & vector) get their y-axis collapsed when the following things occur:
Step 1: do a benchmark run and have the results displayed on their respective detailed plot
Step 2: do another benchmark run while deselecting one of the benchmarks from the previous run
Step 3: detailed plots of deselected benchmarks now have their y-axis collapsed
The easiest way of reproducing this bug is to first run all benchmarks, then deselect some of the problematic benchmarks (blas3, copy & vector) and observe how their plots get messed up when other benchmarks are executed. (see attached screenshot)
I was unable to pinpoint the source of the problem, but it seems that something goes wrong with the detailed plot layout when it has a legend attached to it. Probably something to do with the way plot legends are initialized over multiple result plotting runs.

System Info: Properly align output

Rather than just dumping the output into a text box, it is better to present this in a table view: The first column contains the labels 'Address Bits', 'Available', 'Compiler Available', etc. and then in the second, third, etc. column the values are presented for the first, second, etc. device. The 'Extension' field may only show the first few elements, just like it is done for overloaded cells in Excel/OpenOffice/LibreOffice.

Align and format numbers in bar chart

Format: Numbers contain at least four digits after the comma. One or two are certainly enough.
Alignment: Performance number for the fastest run exceeds the drawable area. Either make the drawable area larger, or align numbers at the left (i.e. beginning of the bar). Remove the red border around the numbers, consider a semi-transparent background for the label instead.