Comments (6)
For me the error what(): Failed to query occupancy.
occurred when I had called cmake
with too high CUTLASS_NVCC_ARCHS
parameter value before compiling and running the performance test.
from cutlass.
Thank you for reporting this.
Could you verify that the CUDA build and execution environment is properly configured? You might be able to do this by simply executing cutlass_profiler --help
to see if it prints any information about available GPU devices.
If CUDA isn't installed correctly, perhaps you can resolve this by installing the latest CUDA driver.
Beyond that, could you provide the steps you used to build the CUTLASS profiler?
from cutlass.
Closing until further information is provided, feel free to reopen if the issue still exists.
from cutlass.
I also ran into this problem under an GTX 1080 ti. My build log is attached, which is based on https://github.com/NVIDIA/cutlass.
My two GPU's are detected OK, as shown in the --help output below. Plus, I believe the execution environment is installed OK, because I am able to run tensorfow jobs OK on the GPU (e.g., BERT and Albert).
Best,
Tom
$ build/tools/profiler/cutlass_profiler --help
...
--device= CUDA Device ID
[0] - GeForce GTX 1080 Ti - SM 6.1, 28 SMs @ 1645 MHz, L2 cache: 2 MB, Global Memory: 10 GB
[1] - Quadro P400 - SM 6.1, 2 SMs @ 1252.5 MHz, L2 cache: 0 MB, Global Memory: 1 GB
from cutlass.
This error is reported in these four places when calling cudaOccupancyMaxPotentialBlockSize
:
./util/include/cutlass/util/reference/device/tensor_compare.h:133: throw std::runtime_error("Failed to query occupancy.");
./util/include/cutlass/util/reference/device/tensor_compare.h:200: throw std::runtime_error("Failed to query occupancy.");
./util/include/cutlass/util/reference/device/tensor_foreach.h:53: throw std::runtime_error("Failed to query occupancy.");
./util/include/cutlass/util/reference/device/tensor_foreach.h:111: throw std::runtime_error("Failed to query occupancy.");
Would you please 1) identify which one reports the error. 2) Use cudaGetErrorString
(https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__ERROR.html#group__CUDART__ERROR_1g4bc9e35a618dfd0877c29c8ee45148f1) to see what is the cause. 3) write a standalone tiny cuda file to call cudaOccupancyMaxPotentialBlockSize
?
from cutlass.
It turns out that I only built it for the Ampere architecture (i.e., -DCUTLASS_NVCC_ARCHS=80). It works fine after recompiling for the Pascal architecture as shown below.
Best,
Tom
$ build/tools/profiler/cutlass_profiler --device=0 --operation=Gemm --m=1024 --n=102
4 --k=128
=>
Bytes: 5242880 bytes
FLOPs: 270532608 flops
Runtime: 0.0501654 ms
Memory: 97.3342 GiB/s
Math: 5392.81 GFLOP/s
...
build sequence (see attached log):
mkdir build && cd build
export CUDACXX=/usr/local/cuda-11.1/bin/nvcc
cmake .. -DCUTLASS_NVCC_ARCHS=61 -DCUTLASS_ENABLE_TESTS=OFF -DCUTLASS_UNITY_BUILD_ENABLED=ON
make cutlass_profiler -j12
from cutlass.
Related Issues (20)
- [FEA] Hopper group-gemm of mixed type
- [QST] about swizzle_layout HOT 1
- [FEA] BFloat16x2 Atomics HOT 1
- [QST]how to use one threadblock process one matrix multiplication?
- [QST] How to do Batched Gemm with python API
- [BUG] undefined symbol: cuTensorMapEncodeTiled on CUTLASS 3.5.1 HOT 3
- [QST] [CuTe] Why `right_inverse` does not work as expected when shape is not static? HOT 2
- [BUG] undefined reference to `cutlass::library::Handle::~Handle()' collect2: error: ld returned 1 exit status HOT 2
- [QST] How to implement non-interleaving partitioning in CuTe? HOT 5
- ldmatrix instruction HOT 1
- [QST] Why do we only need the result of the last k-loop in `cute::gemm` dispatch-5? HOT 5
- [BUG] ElementC=void kernel reads non-void in `GemmDescription` HOT 1
- undefined reference to `cutlass::library::Handle::~Handle()' collect2: error: ld returned 1 exit status HOT 6
- ldmatrix instruction HOT 1
- [QST] Error: Could not detect active GPU device ID [CUDA driver is a stub library] HOT 3
- [QST] TMA Reduce ADD HOT 3
- [FEA] Support .shared::cta to .shared::cluster TMA copy? HOT 1
- [BUG] example 57 Hopper Grouped GEMM Disposition failed
- [QST] Getting `CUDA error: misaligned address` when calculating GEMM with a small matrix, and multiple other questions on understanding Cutlass. HOT 4
- [QST] Understanding layouts in CuTe MMA atom traits HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cutlass.