nvlabs / sassi

Flexible GPGPU instrumentation

SASSI is deprecated!

SASSI is now officially too old to use. For your current NVIDIA GPU instrumentation needs, please use NVBit!

SASSI Instrumentation Tool for NVIDIA GPUs

This project contains the SASSI instrumentation tool. SASSI is not part of the official CUDA toolkit, but instead is a research prototype from the Architecture Research Group at NVIDIA.

SASSI is a selective instrumentation framework for NVIDIA GPUs. SASSI stands for SASS Instrumenter, where SASS is NVIDIA's name for its native ISA. SASSI is a pass in NVIDIA's backend compiler, ptxas, that selectively inserts instrumentation code. The purpose of SASSI is to allow users to measure or modify ptxas-generated SASS by injecting instrumentation code during code generation.

NVIDIA has many excellent development tools. Why the need for another tool? NVIDIA's tools such as cuda-memcheck and nvvp provide excellent, but fixed-function inspection of programs. While they are great at what they are designed for, the user has to choose from a fixed menu of program characteristics to measure. If you want to measure some aspect of program execution outside the purview of those tools you are out of luck. SASSI allows users to flexibly inject their own instrumentation to measure novel aspects of GPGPU execution.

SASSI consists of two main components:

• A closed-source fork of NVIDIA's PTX assembler, ptxas, that is capable of injecting instrumentation code during compilation. SASSI's version of ptxas is distributed on GitHub via "Releases".
• Several realistic samples that demonstrate SASSI's operation.

Newest release notes

• We have added some new features. There is a new instrumentation library that demonstrates how to map a SASS instruction with a given PUPC (SASSI's version of a PC) to the CUDA source. See the "branch" library for its usage. Also see the branch target in example/Makefile for the compiler flags necessary to use the new feature.

• Support for emulating novel SASS instructions for ISA exploration is more stable. We have not yet documented this feature because we are still working out the kinks, but if you are interested in this feature, please contact me.

• Bug fix. The PUPC was invalid for functions with long names. This fix requires installing the latest SASSI binaries.

Prerequisites

SASSI has the following system prerequisites:

1. Platform requirement: SASSI requires an X86 64-bit host; a Fermi-, Kepler-, or Maxwell-based GPU; and at the time of this writing we have generated SASSI for Ubuntu (12, 14, and 15), Debian 7 and 8, and CentOS 6 and 7.
2. Install CUDA 7: At the time of this writing, CUDA 7 can be fetched from here.
3. Make sure you have a 346.41 driver or newer: The CUDA 7 installation script can install a new driver for you that meets this requirement. If you already have a newer driver, that should be fine. You can test your driver version with the nvidia-smi command.
4. The installation script requires Python 2.7 or newer.

Installation

After you have fulfilled your prerequisites, install SASSI by doing the following:

1. Find the release for your platform by clicking on the "release" tab on the GitHub project page, or by navigating here. Find your architecture in the "Downloads" list and download. This download is a very simple binary installer.
2. Run the installer via sh, for example, sh SASSI_x86_64_centos_6.run.

You might need to run the installer as root, depending on where you plan to install SASSI.

Usage

For usage, please follow the instructions in the user guide, which you can find in doc/sassi-user-guide.pdf.

Additionally, ptxas -h lists SASSI's supported options.

Restrictions and caveats

1. 32-bit architectures are not supported.

   This was an early design decision to reduce the large cross product of possible configurations. Please let us know if 32-bit support would be useful though, because it probably wouldn't be too hard to support.

2. Programs currently have to be compiled with -rdc=true, which affects performance.

   SASSI allows users to instrument code by injecting function calls to user-defined functions that are later linked in. In order to perform cross-module function calls in CUDA one must use the "relocatable device code" option, -rdc=true. Future versions of SASSI may remove this restriction.

3. Minimum driver required is 346.41.

   This version of SASSI is designed to work with the CUDA 7 toolchain, which also has that requirement.

sassi's Issues

Does it work for Fermi?

Hello,
In the paper, it says SASSI works for Fermi. But compiling only supports device with capability >= 3.0. Fermi architecture is 2.0+. It will fail during compilation stage when building examples.
How to make SASSI work for Fermi?

Best,
Leiming

Handlers for Error Injection

Hi,

In one of the case studies in the paper, SASSI was used to perform error injection to study application vulnerability. I am wondering if the handlers that were written for such study are available. Or is there a plan to make them available?

Thank you

Error compiling instrumentation handlers

Hi,
I am trying to install and use SASSI, however I am having a problem compiling the instrumentation handlers, Section 6 user guide. I have Ubuntu 12.04, Python 2.7.3 and am compiling for -gencode arch=compute_50,code=sm_50 and have the following environment variable set in env.mk

Any hints as to where I might be going wrong are greatly appreciated, Many Thanks.

env.mk:
export CUDA_HOME=/usr/local/cuda-7.0 export SASSI_HOME=/usr/local/sassi7 export SASSI_SRC=/home/ssc-obdh/Documents/SASSI-master export INST_LIB_DIR=$SASSI_SRC/instlibs/lib export CCBIN=/usr/bin/

`ssc-obdh@ssc-obdh-OptiPlex-790:~/Documents/SASSI-master/instlibs$ make
make -C src install
make[1]: Entering directory '/home/ssc-obdh/Documents/SASSI-master/instlibs/src'
/usr/local/cuda-7.0/bin/nvcc -ccbin /usr/bin/ -std=c++11
--compiler-options -Wall -O3 -gencode arch=compute_50,code=sm_50
-lineinfo -c -rdc=true -o branch.o branch.cu
--maxrregcount=16
--compiler-options -Wall
-I/usr/local/sassi7/include
-I/usr/include/boost/include
-I/usr/local/cuda-7.0/extras/CUPTI/include/
-I/home/ssc-obdh/Documents/SASSI-master/instlibs/utils/
-I/home/ssc-obdh/Documents/SASSI-master/instlibs/include
/home/ssc-obdh/Documents/SASSI-master/instlibs/utils/sassi_srcmap.hpp(272): error: too few arguments for class template "std::set"
...

/usr/include/boost/smart_ptr/shared_ptr.hpp(305): error: function "boost::shared_ptr::shared_ptr(const boost::shared_ptrboost::re_detail::named_subexpressions &) [with T=boost::re_detail::named_subexpressions]" (declared implicitly) cannot be referenced -- it is a deleted function
detected during:
instantiation of "boost::shared_ptr &boost::shared_ptr::operator=(const boost::shared_ptr &) [with T=boost::re_detail::named_subexpressions]"

...
6 errors detected in the compilation of "/tmp/tmpxft_00002f63_00000000-7_branch.cpp1.ii".
make[1]: *** [branch.o] Error 2
make[1]: Leaving directory '/home/ssc-obdh/Documents/SASSI-master/instlibs/src'
make: *** [instlibs] Error `

Issue with Ophist

Hi Mark,

I noticed an issue when using the ophist handler for commits from August 20 and later where the values returned are impossibly large, possibly uninitialized. I am running a GTX680 with CUDA 7.0, driver 346.46 on Ubuntu 14.04. Rolling ophist back to the Aug 13th commit fixed the problem for me.

Is this a bug?

Is there a bug in function getOrInit in sassi_dictionary.hpp? I believe the line *metaPtr = VALID_ENTRY; should come after __threadfence() to guarantee that the threads that are waiting for the entry to become valid, see the correct value - the flag must be set after the new entry has been written.

reg-info,mem-info do not work together

Hi,
I am trying to compile my app with option (-Xptxas --sassi-before-args="reg-info,mem-info"), however it gives me an error "Input file 'mem-info' could not be opened". Is it possible to instrument my app with memory and register info together?
Thanks!

Profiling OpenCL with SASSI?

Hi all,

I have created a SASSI function to profile the CUDA kernel (which works properly), and now I want to apply these functions to profiling the OpenCL kernel. As long as I understand the SASSI user guide, SASSI can instrument for OpenCL kernels. However, I failed to do it. Can anyone tell me how to compile my SASSI functions, and instrument it to the OpenCL kernel?

I tried to compile OpenCL programs with Makefile the SASSI provides(example/Makefile) with "-lOpenCL" flag. It succeeded to build a binary file, but failed to generate profiling information.

Thanks...

compatibility with nvprof/nvvp

All the binary files compiled by SASSI runs well, but it seems that they can not be profiled by nvprof or nvvp because of incompatible CUDA driver version. Is this a common problem? And will it be solved? Thanks.

Installation Script Broken for Ubuntu 14

It seems that the install script does not create the $SASSI/bin folder. This results in the message:

COPYING files from /usr/local/cuda/bin/...
COPYING SASSI's header files...
COPYING SASSI's ptxas...
**********************************************************


ERROR: You may not have permissions to write to $SASSI/bin/ptxas


**********************************************************

It is also possibly due to the fact that my version of ptxas does not have any sassi options (`ptxas -h | grep "sassi" return nothing). There should at least be some warning that this is why there is a failure (I had Cuda 7.5).

Error: driver shutting down, when using "ophist"

I am running SASSI on rodinia benchmarks. When I compiled the program using ophist option and run the executable, it sometimes shows "Error: driver shutting down".

I checked the output, sassi-ophist.txt, is there, but don't know what caused the error msg.

Any suggestions? Thanks in advance.

-Leiming

Problem in running the example

Hi,

I get an error when running the example in SASSI_SRC/example.

here is the bug:

surface86@guo7:make
/g/g90/guo7/guo-libs/sassi7//bin/nvcc -c -o matrixMul.o matrixMul.cu -gencode arch=compute_35,code=sm_35 -Xptxas --sassi-inst-before="memory" -Xptxas --sassi-before-args="mem-info" -g -O3 -dc -I./inc
sh: /g/g90/guo7/guo-libs/sassi7//bin/ptxas: cannot execute binary file
make: *** [matrixMul.o] Error 126

Thanks!

Compatibility with CUDA 7.5RC?

Hi,
just found the project and would like to know if CUDA 7.5 will require new releases of SASSI (and in general every CUDA release) or not?
thanks..

Plans for Pascal and later architecture support?

Hi Mark,

Do you have any plan to extend the SASSI to support Pascal or later architectures (ex. Volta)?

Thanks.
Hyunjun

Installation script broken on line 34 using Python 2.6.6

I think this issue is due to using format in the python-based installation script. As a stopgap solution, I think this issue can be solved by using Python 2.7 or greater.

ptxas -h does not show sassi options

I tried installing sassi . I have a kepler machine with cuda 7.0. The installation script shows success but after I do ptxas -h I find no sassi option. Is the installation script faulty?

PATH, LD_LIBRARY_PATH in Red Hat Linux

Hi,

I have installed sassi7 on my university system (no root access). Directory structure looks like this:

CUDA 7.0:
/usr/local/cuda-7.0/

SASSI7:
/my/home/folder/sassi7/

Now both have nvcc, ptxas in bin/ and related lib files lib/ and lib64/. How do I make sure that sassi7 is installed properly?

Thanks.

Error in running example:CUPTI_ERROR_NOT_INITIALIZED

when I run the example ,the following error occurred. I have no solution to it.
example# ./matrixMul
/SASSI/instlibs/utils/sassi_lazyallocator.hpp:86:Error CUPTI_ERROR_NOT_INITIALIZED for CUPTI API function 'cuptiSubscribe'.

Thanks.