rocm / clarmor

OpenCL tool to detect buffer overflows in GPU kernels

AMD ROCm Software

ROCm is an open-source stack, composed primarily of open-source software, designed for graphics processing unit (GPU) computation. ROCm consists of a collection of drivers, development tools, and APIs that enable GPU programming from low-level kernel to end-user applications.

With ROCm, you can customize your GPU software to meet your specific needs. You can develop, collaborate, test, and deploy your applications in a free, open source, integrated, and secure software ecosystem. ROCm is particularly well-suited to GPU-accelerated high-performance computing (HPC), artificial intelligence (AI), scientific computing, and computer aided design (CAD).

ROCm is powered by AMD’s Heterogeneous-computing Interface for Portability (HIP), an open-source software C++ GPU programming environment and its corresponding runtime. HIP allows ROCm developers to create portable applications on different platforms by deploying code on a range of platforms, from dedicated gaming GPUs to exascale HPC clusters.

ROCm supports programming models, such as OpenMP and OpenCL, and includes all necessary open source software compilers, debuggers, and libraries. ROCm is fully integrated into machine learning (ML) frameworks, such as PyTorch and TensorFlow.

Getting the ROCm Source Code

AMD ROCm is built from open source software. It is, therefore, possible to modify the various components of ROCm by downloading the source code and rebuilding the components. The source code for ROCm components can be cloned from each of the GitHub repositories using git. For easy access to download the correct versions of each of these tools, the ROCm repository contains a repo manifest file called default.xml. You can use this manifest file to download the source code for ROCm software.

Installing the repo tool

The repo tool from Google allows you to manage multiple git repositories simultaneously. Run the following commands to install the repo tool:

mkdir -p ~/bin/
curl https://storage.googleapis.com/git-repo-downloads/repo > ~/bin/repo
chmod a+x ~/bin/repo

Note: The ~/bin/ folder is used as an example. You can specify a different folder to install the repo tool into if you desire.

Installing git-lfs

Some ROCm projects use the Git Large File Storage (LFS) format that may require you to install git-lfs. Refer to Git Large File Storage for more information. For example, to install git-lfs for Ubuntu, use the following command:

sudo apt-get install git-lfs

Downloading the ROCm source code

The following example shows how to use the repo tool to download the ROCm source code. If you choose a directory other than ~/bin/ to install the repo tool, you must use that chosen directory in the code as shown below:

mkdir -p ~/ROCm/
cd ~/ROCm/
~/bin/repo init -u http://github.com/ROCm/ROCm.git -b roc-6.0.x
~/bin/repo sync

Note: Using this sample code will cause the repo tool to download the open source code associated with the specified ROCm release. Ensure that you have ssh-keys configured on your machine for your GitHub ID prior to the download as explained at Connecting to GitHub with SSH.

Building the ROCm source code

Each ROCm component repository contains directions for building that component, such as the rocSPARSE documentation Installation and Building for Linux. Refer to the specific component documentation for instructions on building the repository.

Each release of the ROCm software supports specific hardware and software configurations. Refer to System requirements (Linux) for the current supported hardware and OS.

Build ROCm from source

The Build will use as many processors as it can find to build in parallel. Some of the compiles can consume as much as 10GB of RAM, so make sure you have plenty of Swap Space !

By default the ROCm build will compile for all supported GPU architectures and will take approximately 500 CPU hours. The Build time will reduce significantly if we limit the GPU Architecture/s against which we need to build by using the environment variable GPU_ARCHS as mentioned below.

# --------------------------------------
# Step1: clone source code
# --------------------------------------

mkdir -p ~/WORKSPACE/      # Or any folder name other than WORKSPACE
cd ~/WORKSPACE/
export ROCM_VERSION=6.2.0
~/bin/repo init -u http://github.com/ROCm/ROCm.git -b roc-6.2.x -m tools/rocm-build/rocm-${ROCM_VERSION}.xml
~/bin/repo sync

# --------------------------------------
# Step 2: Prepare build environment
# --------------------------------------

# Option 1: Start a docker container
# Pulling required base docker images:
# Ubuntu20.04 built from ROCm/tools/rocm-build/docker/ubuntu20/Dockerfile
docker pull rocm/rocm-build-ubuntu-20.04:6.2
# Ubuntu22.04 built from ROCm/tools/rocm-build/docker/ubuntu22/Dockerfile
docker pull rocm/rocm-build-ubuntu-22.04:6.2
# Ubuntu24.04 built from ROCm/tools/rocm-build/docker/ubuntu24/Dockerfile
docker pull rocm/rocm-build-ubuntu-24.04:6.2

# Start docker container and mount the source code folder:
docker run -ti \
    -e ROCM_VERSION=${ROCM_VERSION} \
    -e CCACHE_DIR=$HOME/.ccache \
    -e CCACHE_ENABLED=true \
    -e DOCK_WORK_FOLD=/src \
    -w /src \
    -v $PWD:/src \
    -v /etc/passwd:/etc/passwd \
    -v /etc/shadow:/etc/shadow \
    -v ${HOME}/.ccache:${HOME}/.ccache \
    -u $(id -u):$(id -g) \
    <replace_with_required_ubuntu_base_docker_image> bash

# Option 2: Install required packages into the host machine
# For ubuntu20.04 system
cd ROCm/tools/rocm-build/docker/ubuntu20
cp * /tmp && cd /tmp
bash install-prerequisites.sh
# For ubuntu22.04 system
cd ROCm/tools/rocm-build/docker/ubuntu22
cp * /tmp && cd /tmp
bash install-prerequisities.sh
# For ubuntu24.04 system
cd ROCm/tools/rocm-build/docker/ubuntu24
cp * /tmp && cd /tmp
bash install-prerequisites.sh

# --------------------------------------
# Step 3: Run build command line
# --------------------------------------

# Select GPU targets before building:
# When GPU_ARCHS is not set, default GPU targets supported by ROCm6.1 will be used.
# To build against a subset of GFX architectures you can use the below env variable.
# Support MI300 (gfx940, gfx941, gfx942).
export GPU_ARCHS="gfx942"               # Example
export GPU_ARCHS="gfx940;gfx941;gfx942" # Example

# Pick and run build commands in the docker container:
# Build rocm-dev packages
make -f ROCm/tools/rocm-build/ROCm.mk -j ${NPROC:-$(nproc)} rocm-dev
# Build all ROCm packages
make -f ROCm/tools/rocm-build/ROCm.mk -j ${NPROC:-$(nproc)} all
# list all ROCm components to find required components
make -f ROCm/tools/rocm-build/ROCm.mk list_components
# Build a single ROCm packages
make -f ROCm/tools/rocm-build/ROCm.mk T_rocblas

# Find built packages in ubuntu20.04:
out/ubuntu-20.04/20.04/deb/
# Find built packages in ubuntu22.04:
out/ubuntu-22.04/22.04/deb/
# Find built packages in ubuntu24.04:
out/ubuntu-24.04/24.04/deb/

# Find built logs in ubuntu20.04:
out/ubuntu-20.04/20.04/logs/
# Find built logs in ubuntu22.04:
out/ubuntu-22.04/22.04/logs/
# Find built logs in ubuntu24.04:
out/ubuntu-24.04/24.04/logs/
# All logs pertaining to failed components, end with .errrors extension.
out/ubuntu-22.04/22.04/logs/rocblas.errors      # Example
# All logs pertaining to building components, end with .inprogress extension.
out/ubuntu-22.04/22.04/logs/rocblas.inprogress  # Example
# All logs pertaining to passed components, use the component names.
out/ubuntu-22.04/22.04/logs/rocblas             # Example

Note: Overview for ROCm.mk

ROCm documentation

This repository contains the manifest file for ROCm releases, changelogs, and release information.

The default.xml file contains information for all repositories and the associated commit used to build the current ROCm release; default.xml uses the Manifest Format repository.

Source code for our documentation is located in the /docs folder of most ROCm repositories. The develop branch of our repositories contains content for the next ROCm release.

The ROCm documentation homepage is rocm.docs.amd.com.

Building the documentation

For a quick-start build, use the following code. For more options and detail, refer to Building documentation.

cd docs
pip3 install -r sphinx/requirements.txt
python3 -m sphinx -T -E -b html -d _build/doctrees -D language=en . _build/html

Alternatively, CMake build is supported.

cmake -B build
cmake --build build --target=doc

Older ROCm releases

For release information for older ROCm releases, refer to the CHANGELOG.

clarmor's Issues

Add code style guidelines, integrate clang-format

Relies on #4

Once we've finished cleanup and basic sanitizing of clARMOR, we should fix on a standard style guide for code that will be accepted into clARMOR. Ideally, this would be relatively mechanical and could be checked with a tool like clang-format to make sure that any commits meet guidelines.

clSVMFree() is not properly handled with underflow detection

When underflow detection is enabled, the OpenCL interceptor calls the real OpenCL clSVMAlloc() function. The value that is returned to the user is POISON_FILL_LENGTH past the real SVM region's start. We use this padding before the SVM region as the canary region for underflow detection.

For example, if the value returned from the real clSVMAlloc() is 0x501d00000, we may return the address 0x501d01000 to the user.

Later, when the user calls clSVMFree(), they must (as per the OpenCL spec) give us the pointer returned to them -- 0x501d01000 in this example.

In our current wrapper for clSVMFree(), when we are not gathering memory overhead statistics, we take the pointer passed in from the user and directly attempt to call the real clSVMFree() on it. The real OpenCL runtime can fail in this case, because we are not passing into the the pointer that it returned to userland.

When taking memory overhead statistics, we will attempt to call clSVMFree() both on main_svm (which is the real pointer -- e.g. 0x501d00000) and svm_pointer (which is the "bad" pointer -- e.g. 0x501d01000).

I believe the correct solution here is to always set void *main_svm = temp->main_buff; and only attempt a single SVMFree() on that function. However, I need to verify this before committing anything.

Add CI tests to clARMOR repo

We would like to have automated builds (and as much testing as possible) upon commit so that any external contributors can have their code tested before it is accepted for merge. Something like Travis-CI should be integrated into the clARMOR repo.

clARMOR does not build with OpenCL 1.1 headers

Using the Khronos OpenCL 1.1 cl.h header file (found here, for example) will prevent clARMOR from building.

This is a mix of using OpenCL 1.2 data structures, #defines, and functions within clARMOR. This includes:

• clEnqueueMarkerWithWaitList()
• clEnqueueFillBuffer()
• clCreateImage()
• cl_image_desc
• CL_MEM_OBJECT_IMAGE2D_ARRAY
• CL_MEM_OBJECT_IMAGE1D*
• CL_IMAGE_ARRAY_SIZE
• CL_BLOCKING/CL_NON_BLOCKING
• CL_DEVICE_IMAGE_MAX_BUFFER_SIZE
• CL_MEM_HOST_WRITE_ONLY/CL_MEM_HOST_READ_ONLY/CL_MEM_HOST_NO_ACCESS
• CL_DEPTH/CL_DEPTH_STENCIL/CL_UNORM_INT24
• Incorrect preprocessor guards in cl_err.h for some of the OpenCL 1.2 error values.

As such, clARMOR will not work on systems that do not provide OpenCL 1.2+ headers (such as versions of the AMD APP SDK before v2.7, and the CUDA SDK before v7.5).

Start using Coverity Scan

In an effort to increase the quality of clARMOR code, we should integrate Coverity Scan into our repo and release process. Since clARMOR is an open source project, we should meet the requirements to use this for free.

At the very least, we should manually use this tool every so often and try to drive our defect count to 0.

Ideally, we would set this up to integrate with Travis-CI. Completing a scan (and fixing any new issues) should gate any release from the development to master branch. This relies on #2.

Add official version numbers and development branch

Right now, all development and release happens in the main branch, and there are no official "stable" or tested releases. Changes go straight into master and those become the official version. This isn't an ideal way to release a product.

To fix this, we should add a development branch where most commits should go (and where others can submit PRs), and then we should semi-regularly catch master up to these development branch. This is akin to what clSPARSE does in its repo.

In addition, we should add a version number (tag, etc.) to each update to master so that bug reports can come back on particular versions. This also means adding a version check (--version, -v) to the clARMOR executable.

Finish code standardization cleanup

As part of our process of releasing clARMOR from inside of AMD, we began cleaning up some of our code sprint messiness and standardizing style, etc.

We didn't complete all of these in time for our planned release, so some of our current code contains cruft that should be cleaned up. For example: standardizing on line lengths, reducing some of the gargantuan functions into easier-to-maintain chunks, and splitting up some large files into smaller, more readily understood files.

For example, the cl_interceptor directory is still a bit of a mess, as is bufferOverflowDetect.c.

Potential to expand support for HIP Apps?

Hi,

Just curious how challenging it would be to add support for HIP applications. Any thoughts?

Thanks,

Jeff

How to build with ROCm ?

i.e. how to set environment variable for ROCm ?

 ... has failed.Please set one of the environment variables:AMDAPPSDKROOT, ATISTREAMSDKROOT, INTELOCLSDKROOT, CUDA_PATH, or CUDA_INC_PATH/CUDA_LIB_PATH.  Stop.