gpuopen-tools / gpu-reshape Goto Github PK

GPU Reshape (GRS) is an API agnostic instrumentation framework, with instruction level validation.

License: Other

C++ 68.26% C 5.22% CMake 2.60% HLSL 0.77% Batchfile 0.15% C# 22.81% Python 0.19%

d3d12 directx directx-12 graphics graphics-programming instrumentation validation vulkan vulkan-api gpu-reshape

gpu-reshape's Introduction

GPU Reshape

GPU Reshape offers API agnostic instrumentation of GPU side operations to perform, e.g., validation of potentially undefined behaviour, supporting both Vulkan and D3D12 (DXIL). No application side integration is required.

GPU Reshape is an open collaboration between Miguel Petersen (author), Advanced Micro Devices and Avalanche Studios Group.

Quick Start - Features - Build - API - Motivation - UIX

Current feature scope provides instrumentation on operations which are either undefined behaviour, or typically indicative of user fault such as:

Resource Bounds
Validation of resource read / write coordinates against its bounds.
Export Stability
Numeric stability validation of floating point exports (UAV writes, render targets, vertex exports), e.g. NaN / Inf.
Descriptor Validation
Validation of descriptors, potentially dynamically indexed. This includes undefined, mismatched (compile-time to runtime), out of bounds descriptor indexing, and missing table bindings.
Concurrency Validation
Validation of resource concurrency, i.e. single-producer or multiple-consumer, between queues and events.
Resource Initialization
Validation of resource initialization, ensures any read was preceded by a write.
Infinite Loops
Detection of infinite loops. Experimental.

The future feature scope includes profiling and debugging functionality, such as branch hot spots, live breakpoints, and assertions. For the full planned feature set, see Features.

Instrumentation as a framework

The toolset provides a generalized SSA-based intermediate language from which all instrumentation is done, bi-directionally translated to SPIRV and DXIL. Each feature, such as the validation of out-of-bounds reads, operates solely on the intermediate language and has no visibility on the backend language nor API.

Each feature can alter the program as it sees fit, such as adding basic-blocks, modifying instructions, and even removing instructions. The feature is given a Program, which act as the abstraction for the active backend, from which the user has access to all functions, basic-blocks, instructions, types, etc..., and is able to modify as necessary. After modification, the backend then performs just-in-time recompilation of the modified program back to the backend language.

The toolset additionally provides a set of building blocks needed for instrumentation:

Message Streams help facilitate GPU -> CPU communication, and also serves as the base for inter-process/endpoint communication.
Persistent data such as buffers, textures and push / root constants, visible to all instrumentation features. Certain features require state management.
User programs, entirely user driven compute kernels written through the intermediate language. All persistent data visible.
User side command hooking, certain features may wish to modify state, invoke kernels, before the pending command.
Resource tokens, abstracting away differences in binding models by providing a token from shader resource handles. Each token provides a physical UID, resource type and sub-resource base.
Command scheduling, record and submit custom (abstracted) command buffers / lists.

Features do not need to concern themselves with backend specifics, such as vectorized versus scalarized execution, control-flow requirements, and other implementation details. Given compliance, each feature will translate seamlessly to the backend language.

This toolset aims to serve as a framework for instrumentation, acting as a modular base from which any number of tools, techniques, optimizations, etc..., can be implemented.

Testing Suite

A list of applications GPU Reshape is tested against for stability and validity.

Games

Call of the Wild: The Angler (Avalanche, DX12)
Second Extinction (Avalanche, Vulkan)
Rage 2 (Avalanche / id Software, Vulkan)
Chernobylite (The Farm 51, DX12)
Dying Light 2 (Techland, DX12)
The Lord of the Rings: Gollum (Daedalic Entertainment, DX12)
A Plague Tale: Requiem (Asobo Studio, DX12)
Ratchet & Clank: Rift Apart (Insomniac Games / Nixxes Software, DX12)
Forspoken (Square Enix, DX12)
The Riftbreaker (EXOR Studios, DX12)
Starfield (Bethesda Game Studios, DX12)
Cyberpunk 2077 (CD PROJEKT RED, DX12)

Engines

Unreal Engine 4 (Epic Games, DX12, Vulkan)
Unreal Engine 5 (Epic Games, DX12, Vulkan)

AMD

FidelityFX-SDK (DX12, Vulkan)
FSR2 (DX12, Vulkan)
GLTFSample (DX12, Vulkan)
ParallelSort (DX12, Vulkan)
SPDSample (DX12, Vulkan)
SSSRSample (DX12, Vulkan)
TressFX (DX12, Vulkan)

Khronos

Vulkan-Samples (Vulkan, certain extensions pending support, e.g. descriptor buffers)

Microsoft

MiniEngine ModelViewer (DX12)

Known Issues

Raytracing and mesh shaders are currently pass-through, no instrumentation is done on them.
Initialization feature false positives. Resources can be initialized in a myriad of ways, this will be improved with time.
Loop feature not catching all timeouts. The loop feature is currently experimental, and relies on non-standard guarantees.
Application launches, not attaching, will only connect to the first device, which may not necessarily be the intended device.

Credit

GPU Reshape was initially developed as a prototype tool by Miguel Petersen at Avalanche Studios Group, extending validation tools to shader side operations. It was then requested to continue development externally through an open collaboration on GPUOpen.

Development was supported by:

Lou Kramer (AMD)
Jonas Gustavsson (AMD)
Marek Machlinski (AMD)
Rys Sommefeldt (AMD)
Mark Simpson (AMD)
Daniel Isheden (Avalanche Studios)
Alexander Polya (Avalanche Studios)
Wiliam Hjelm (Avalanche Studios)

gpu-reshape's People

Contributors

Stargazers

Watchers

Forkers

dmitrykolesnikovich brucelevis moarmand kealg wieslawsoltes playeryan hengle hengtek monarchsolutions pinkuburu

gpu-reshape's Issues

Support for separate PDBs / LDBs matched by contents hash

As per the title, GRS is missing support for loading debug information from separate PDB / LDB files.

Install .NET dialog does not work

When starting GPU Reshape.exe without the correct .NET installed, there is a dialog asking to install .NET. Pressing Yes or No both result in nothing happening.
Not sure if it's supposed to open a web page or windows update or similar.

A suggestion would be to list the version of .NET that's needed in the dialog box, then it would be straighforward to find it myself.

Documentation effort

Code is great, when people can understand it. So, let's make an effort to improve documentation.

The following list will be extended over time:

Feel free to add as you see fit.

Establish automatically a connection on app launch

I'm trying to instrument an application, that crashes on launch, to see if the instrumentation info provided by GPU Reshape can give any hints in what could cause the crash. However, since the app crashes on launch I don't have time to click through the menu to establish a connection.
Is there a way to tell GPU Reshape to immediately start instrumenting this application once its launched?

Workspace - Shaders

The workspace has Shaders and Pipelines. Right click on Pipelines gives me the option to add a Filter, while right click on Shaders just shows an empty UI box at the mouse cursor. Is it just down to a not-implemented shader filter yet?

[UIX] Validation resource list control may go out of bounds

The validation resource list can exceed the source code control, it needs to be snapped to the border, i.e. max = codeHeight - resourceHeight.

Question about Instrumentation and validation

When does instrumentation and validation happen: is it a one-time approach, or does it continuously instrument and validate new shaders/pipelines as they come in from the application as soon as I enabled instrumentation in the UI?

Browser WebGPU?

Is support for WebGPU development on your radar?

FSR2 Vulkan public github sample crashes

The public FSR2 Vulkan sample on github (https://github.com/GPUOpen-Effects/FidelityFX-FSR2) crashes when GPUReshape is enabled.
Repro steps:

Enable Discovery via RStudio.
Run FSR2 Vulkan sample

The crash happens during vkCreateGraphicsPipelines of a post processing pixel shader.
Without GPUReshape, the sample runs fine.

[UIX] Dragging dockables is broken

Dragging dockables seems fairly broken at the moment.

[UIX] Connection dialogue row-styling appears like a selection

(First official bug report 🥳, filing on behalf of an external party)

The connection dialogue row-wise styling can be confusing. Below, the first entry appears selected, upon clicking connect nothing will happen. The row-styling is giving a blue shade for every odd row, which is confusing.

[Backend] Message stream compression

Shader export data is typically a repeated stream of a few unique messages, there is a lot of benefit in rolling a custom compression scheme for it. Additionally, this would help with UI performance on large streams.

Crash during CreateSwapChainForHwnd when starting application while GPU Reshape is enabled

When I try to launch a test app, it throws an error during CreateSwapChainForHwnd, when GPU Reshape is enabled in the background.
Repro steps:

Launch Studio.exe in gpu-validation/Publish/PackageDebug
Go to File->Settings: set Discovery Status to Running
Launch test app
-> Crash

The application does not crash when Discovery Status is set to Stopped.

Error output:
Exception thrown at 0x00007FFA0FC7CD29 (KernelBase.dll) in TestApp_DX12d.exe: 0xC0000002: The requested operation is not implemented.
Unhandled exception at 0x00007FFA0FC7CD29 (KernelBase.dll) in TestApp_DX12d.exe: 0xC0000002: The requested operation is not implemented.

when calling CreateSwapChainForHwnd, Call Stack:

KernelBase.dll!00007ffa0fc7cd29()	Unknown
Backends.DX12.Layer {0E9E2355-2167-4414-8891-9021FD875215}.dll!00007ff9d12fdaad()	Unknown
Backends.DX12.Layer {0E9E2355-2167-4414-8891-9021FD875215}.dll!00007ff9d130325a()	Unknown
dxgi.dll!00007ffa0e0d5e90()	Unknown
dxgi.dll!00007ffa0e0ce07f()	Unknown
dxgi.dll!00007ffa0e07afe3()	Unknown
dxgi.dll!00007ffa0e0622c3()	Unknown
dxgi.dll!00007ffa0e0621ea()	Unknown
dxgi.dll!00007ffa0e062105()	Unknown
dxgi.dll!00007ffa0e07ba53()	Unknown
dxgi.dll!00007ffa0e0bf529()	Unknown
dxgi.dll!00007ffa0e0bf27b()	Unknown
Backends.DX12.Layer {0E9E2355-2167-4414-8891-9021FD875215}.dll!00007ff9d116c656()	Unknown
Backends.DX12.Layer {0E9E2355-2167-4414-8891-9021FD875215}.dll!00007ff9d1303ee3()	Unknown

Tested system:
RX 6800 XT
Driver: 22.11.2

Linux Support

Would be really nice to have Linux support for this amazing tool.

In announcement article there were mention that

Linux® support is a planned addition.

I hope it would happen and I create this issue to track this.

Vulkan Layer causes RPS test to fail

Hi,
When the GPUReshape Vulkan Layer is active, the test_built_in_nodes_vk from the RPS sample (https://github.com/GPUOpen-LibrariesAndSDKs/RenderPipelineShaders) fails.
To reproduce the issue:

clone the RPS sample
use the default build batch file to build the vs solution
enable the GPUReshape Vulkan layer
run the ctest from the build folder OR build and run the test test_built_in_nodes_vk manually from the VS solution
The test does not fail when the GPUReshape Vulkan layer is not active.

[Backend] Message stream growth seemingly non-functional

Dynamic message stream growth in the shader export streamer seems to be broken.

DXIL forward reference son non-phi instructions

Certain shaders in FSR2 make use of forward references on DXIL bin ops, which does not handle value forward references yet.

Should be simple enough to support given the type guarantees.

[UIX] Instrumentation warnings

Certain instrumentation features are best enabled at startup, such as the initialization feature which needs to perform state tracking from application start The UI needs to warn the user about this behaviour, as it may not be obvious.

Shader cache

While the instrumentation may change quite often, the initial parsing rarely does. The prototype layer made use a global cache to speed up instrumentation times, we need something similar here.

D3D12 - Only sign DXBC / DXIL when needed

Code signing can optionally be disabled if:

Developer mode is enabled on the machine
D3D12ExperimentalShaderModels has been enabled on the target device

Machine configuration is up to the user, however the detoured devices could enable "experimental shader models" by default, assuming there are no side effects from it.

[UIX] "Launh from" pre-configurations

It would be immensely useful if the Launch From window had a set of configurations ready, so that repeatedly launching programs is much more hazzle-free.

One of such examples is Descriptor instrumentation with synchronous recording and safe-guarding on.

[DX12] Add support for Slim PDBs

External Slim-PDBs are currently not supported. Slim-PDBs only contain the source code and original compilation parameters, not the non-canonical (LLVM) debug module.

Source traceback as it is today creates a mapping from an arbitrary UID to both an instruction offset and source-wise offset. If we were to instead only consider the instruction offset, and then allow the source-wise offset to be resolved later, it could allow for easier Slim PDB support.

Upon a validation error or shader inspection, Reshape would compile a full PDB with the debug module from the Slim PDB contents. As with other tooling, there is a slight risk that the resulting binaries are different, but it seems doable.

UI feedback

Some feedback and questions on the UI:

when I establish a connection to an application, it shows up in my workspace list. the application remains listed there even after it's terminated. If I open and close to many application to which I establish a connection, this gets cluttered very quickly.
Log: this listed applications in my workspace continue to produce Log output even after they have been terminated. e.g., "connection has been re-established". It feels like it's wrong, or I'm not sure what it's suppose to mean, because the application is not even running anymore. It makes the Log also a bit unreadable because you have to filter out these messages to be able to catch the ones for the current application that is actually running.
In the workspace list with all applications connected, there is a little icon next to the executable name. it's first white, then red. I think it's red after the application terminated, but I'm not sure. My question is: what does this icon mean?
In the bottom of the UI, it shows a note when compilation happens. this note stays there even if the application terminated due to some crash that happened during instrumentation. Since I'm not sure what it's compiling, I am confused if it's actually still doing something, or if it stopped compilation because it was application specific.
I also noticed that while the app is running and connected with GPU Reshape, the Log is constantly outputting that it Lost the Connection and Regained it. I'm not sure what it means at all - is there something wrong and if yes, what and how to fix it?

[UIX] Embedded documentation

Online documentation is great until you're not online, provide some kind of embedded documentation in the UI.

[Vulkan] debug build crashes the application

When I have a debug build of GPUReshape running with discoverability enabled, my Vulkan application crashes with following callstack:

Crash detected, current frames:
[00007FFC9DA4BDD0] DebugBreak
C:\Windows\System32\KERNELBASE.dll
[00007FFBBA43D92B] Hook_vkGetDeviceProcAddr
D:\Tools\GPU Reshape\23-6-28\artifacts-debug\Bin\MSVC\Debug\GRS.Backends.Vulkan.Layer.dll
[00007FFBBA43D92B] Hook_vkGetDeviceProcAddr
D:\Tools\GPU Reshape\23-6-28\artifacts-debug\Bin\MSVC\Debug\GRS.Backends.Vulkan.Layer.dll
[00007FFBBA43D92B] Hook_vkGetDeviceProcAddr
[00007FFBBA43D92B] Hook_vkGetDeviceProcAddr
[00007FFBBA43D92B] Hook_vkGetDeviceProcAddr
[00007FFC0D63EAF0] vkDestroyDescriptorPool
[00007FFC0D63EAF0] vkDestroyDescriptorPool
[00007FFC0D63EAF0] vkDestroyDescriptorPool
[00007FFC0D63EAF0] vkDestroyDescriptorPool
[00007FFC0D63EAF0] vkDestroyDescriptorPool
[00007FFC0D63EAF0] vkDestroyDescriptorPool
[00007FFC0D63EAF0] vkDestroyDescriptorPool
[00007FFC0D63EAF0] vkDestroyDescriptorPool
[00007FFC0D63EAF0] vkDestroyDescriptorPool
[00007FFC9F792690] BaseThreadInitThunk
[00007FFCA034A9D0] RtlUserThreadStart

Waiting for debugger to attach...

This does not happen with the release build of GPUReshape. Also note, that at this point I have not added the Vulkan application to my workspace nor instrumented anything.
Are there any validation layers enabled in debug mode that are not enabled in release mode?

Validation Feature: Detect potential endless loops

I was wondering if GPU Reshape can detect potential endless loops in shaders?

Implement null descriptors

Currently null descriptors, being a concept in both DX12 & Vulkan in which any load / write is zeroed or discarded, are not supported. Easiest way about this is likely to introduce another special PUID, checking it in a performant manner.

Prefix every process with GPU Reshape / GRS or anything recognizable so people know where it comes from

Counter for messages

I think it does overflow at some point? I'm also not sure what's the point of a counter that increases insanely fast and wonder if we can come up with something better / more useful 🤔

[UIX] Settings menu styling pass

The settings menu desperately needs a styling pass, especially the application list.

Improved workspace counters

Message counters as of themselves are not that interesting, what, however, is interesting is:

Messages per second
Messages per "frame" (would have to answer what a frame is)
Total percentage, percentage over a second, etc...

This task is to change the display to reflect this, additionally, to make it configurable in some sense.

No rendering at all when the app is launched with Reshape

Hi,

When I run my engine with Reshape, there is no rendering at all inside the window. It is completely black.
Also, my app freezes when trying to close the window.

Everything works fine if it is ran directly without Reshape.

I tried forcing a standard 8-bit swapchain format instead of the 10-bit one but that did not change anything.

You can find it at https://github.com/Cyphall/Cyph3D.
Note that the Debug configuration requires Validation Layers to be available. Since Reshape does not expose them, only the Release configuration is usable with it.

Block graph visualization seemingly broken on entry points

Workspace icon "trash bin" description

The little workspace icon of a trash bin has the same description as the other icon on its left: "Collapse all items".
Also it deletes the workspace but not really, because it's still doing work?
and when I re-add it all my counters are reset to 0 I think

[UIX] Application names with underscores are incorrectly sanitized

From an external reporter, if the application / process name has an underscore in it, it seemingly disappears from the workspace name. Should be a simple fix.

GRS vulkan layer causes a stackoverflow

Hi,
I just tried the GPUReshape app on my program and it caused my program to simply crash when launched from Reshape.
At first i could not find a cause but later i enabled the global hook. This hook installs a global Vulkan layer that is always enabled. When i then launched my app outside of GPUReshape i got a proper asset catch within my IDE so i could see the problem. Here i am assuming, that the GRS vulkan layer is enabled for any app launched from GPUReshape.

When launched outside of Reshape with the global hook enabled, the app throws an error within the GRS vulkan layer complaining about a stack overflow.
I think this could maybe be caused by a large descriptor set (containing 100000 buffers and 100000 image descriptors) i am using in my "bindless" vulkan backend. I had multiple tools stumble over my large descriptor sets already so i hope this would be a quick fix.

My app:
https://github.com/Sunset-Flock/Timberdoodle
I hope its not too inconvenient to build. it required vcpkg and the vulkan sdk installed.

Crashes in GRS.Backends.DX12.Layer

Hellow

Running GPU Reshape on my application, causes a crash in some GRS.Backends.DX12.Layer which happens right after D3D12CreateDevice(). This is the application which was tested on https://github.com/Nickelium/ComputePlayground.

I tried to build from source to see whats happening but the build process fails with following:

Nickelium

Can't expand message box

I seem unable to expand the message box, which prevents me to read the full error message, e.g.,
Descriptor mismatch detected, shader expected Sample but rec ... ?

Hovering with the mouse over the message also does not show the full message as tooltip.

Compiler error when building from source

When I try to build GPUReshape from source, I get a compile error.
Repro steps:

get a fresh clone from the repository
run the VisualStudio2019.bat file
open the gpu-validation.sln in cmake-build-vs2019
build the solution

Error list:

Error C2039 'next_CreateCommandQueue1': is not a member of 'ID3D12Device8DetourVTable' Layer C:...\GPUReshape\gpu-validation\Source\Backends\DX12\Layer\Source\CommandList.cpp 124

Error LNK1104 cannot open file '........\Lib\MSVC\Debug\Backends.DX12.Layer.lib' Tests (Source\Backends\DX12\Tests) C:...\GPUReshape\gpu-validation\cmake-build-vs2019\Source\Backends\DX12\LINK 1
...

Loading HLSL shader source code is really slow

Native DXBC leads to a crash during instrumentation

When I instrument the pipelines, my DX12 application crashes. I get the following GPUReshape callstack:

KernelBase.dll!00007ffd6577bdd2() Unknown
GRS.Backends.DX12.Layer {5E2507A2-DA03-446F-9F65-7E133DF1276E}.dll!Detail::Break(const char * message) Line 52 C++
GRS.Backends.DX12.Layer {5E2507A2-DA03-446F-9F65-7E133DF1276E}.dll!DXBCPhysicalBlockFeatureInfo::Compile() Line 53 C++
GRS.Backends.DX12.Layer {5E2507A2-DA03-446F-9F65-7E133DF1276E}.dll!DXBCPhysicalBlockTable::Compile(const DXCompileJob & job) Line 110 C++
GRS.Backends.DX12.Layer {5E2507A2-DA03-446F-9F65-7E133DF1276E}.dll!DXBCModule::Compile(const DXCompileJob & job, DXStream & out) Line 118 C++
GRS.Backends.DX12.Layer {5E2507A2-DA03-446F-9F65-7E133DF1276E}.dll!ShaderCompiler::CompileShader(const ShaderJob & job) Line 203 C++
GRS.Backends.DX12.Layer {5E2507A2-DA03-446F-9F65-7E133DF1276E}.dll!ShaderCompiler::Worker(void * data) Line 103 C++
GRS.Backends.DX12.Layer {5E2507A2-DA03-446F-9F65-7E133DF1276E}.dll!Detail::DelegateCreator<void (__cdecl ShaderCompiler::*)(void *)>::MakeFrameProxy::__l2::<lambda_1>::operator()(void * frame, void * <args_0>) Line 79 C++
GRS.Backends.DX12.Layer {5E2507A2-DA03-446F-9F65-7E133DF1276E}.dll!_Closure_wrapper_095bea0a_1::<lambda_invoker_cdecl>(void * __p1, void * __p2) Line 76 C++
GRS.Backends.DX12.Layer {5E2507A2-DA03-446F-9F65-7E133DF1276E}.dll!Delegate<void __cdecl(void *)>::Invoke(void * <args_0>) Line 46 C++
GRS.Backends.DX12.Layer {5E2507A2-DA03-446F-9F65-7E133DF1276E}.dll!DispatcherWorker::ThreadEntry() Line 66 C++

It hits the following assert: ASSERT(table.dxilModule, "PSV not supported for native DXBC");

I presume that GPU Reshape does not support this game based on what I see from the assert. Release mode GPU Reshape simply crashes the game with an access violation error.
Rather than crashing the game, I would prefer to get an error/warning message in my GPU Reshape UI and let the game continue to run.

Validation warning for uninitialized placed resource

I am not sure if GPU Reshape is the right tool for that, but a developer mentioned that a validation warning about using a resource that has not been cleared nor discarded would be great and wondered if this existed already. This is only relevant for placed resources, as committed resources are initialized to 0 anyway.

DX12 sampler heap PRMT handling

Omitted path, time to implement it.

Automated test-suite

While extending unit tests / TDD is preferable, nothing quite beats mature applications. Automated testing of application suites has immense benefits, and reduces manual labor on change validation.

DirectX12 Agility SDK

Hi,

I gave GPU Reshape a quick try with my app on AMD 6700s hardware.

It didn't work, whether I tried it by launching the app or attaching to the app.

I wondered, is this because my app is using the DirectX12 Agility SDK rather than standard DX12?

Thanks

[UIX] Traceback support for IL

There's quite a lot of value in being able to perform traceback to the IL, and be able to see the resources, bounds being accessed, etc., even without the source code.

Vulkan - having Khronos validation layer fails instance creation

VK_LAYER_KHRONOS_validation

Disabling Vulkan layer issue

If I enable Discovery via RStudio.exe, an implicit GPUReshape Vulkan layer gets created and enabled (HKEY_CURRENT_USER->SOFTWARE->Khronos->Vulkan->ImplicitLayers). If I stop the Discovery, the layer gets correctly removed.
However, if I don't stop the Discovery but just close RStudio, the implicit layer is still active and GPUReshape hooks into Vulkan applications.
If I then start RStudio again and check in the Settings the Discovery status, it reports as Running - which seems correct, as the layer is indeed still running. However, if I then disable the Discovery, the layer remains active.
First of all, at least when the Discovery gets disabled upon restart or RStudio.exe, the layer should be disabled as well.
Second, I think that the layer should also get disabled when I simply close RStudio.exe, as it's easy to forget to explicitly disable the Discovery before closing the application. If we want to let it remain active, we need some sort of Icon (maybe in the taskbar?) indicating that it's still running in the background.
I don't know how the DX12 hooking behaves in this scenario, but we also should ensure that the Vulkan hooking and the DX12 hooking behave the same way.

IL document occasionally fails syntax highlighting

[Backend] Validate networking across machines

It should just work, but has never been properly tested.