timmhess / unrealimagecapture

A small tutorial repository on capturing images with semantic annotation from UnrealEngine to disk.

Image Capturing With UnrealEngine 4 or 5 For Deep Learning

To review the code please find it in the Plugin directory! (The CaptureToDisk is currently not up to date, I will try to find time to fix this in the near future!)

A Small Introduction

UnrealEngine is known to be a powerful tool to create virtual worlds as it is a highly valued AAA production game engine. Generating temporally consistent data with automatic pixel-wise annotations from complex scenes, such as traffic scenarios, is a capability worth leveraging, especially for training and validation of machine learning, or more explicitly deep learning, applications, and has been explored in a variety of projects already. Already, there are plugins available that allow rendering images from UE4 to disk at runtime, such as prominently UnrealCV and AirSim. This repository aims to be a tutorial that demonstrates such an 'image capturing' mechanism in detail fo you to understand its inner workings, and in turn enable you to reuse it in a custom fashions that suit the needs of your project.

When I was setting up scenes for my research the formerly mentioned plugins were just not yet supporting the latest engine versions I wanted/needed to use, and also I was missing a place where the knowledge of how to render images to disk myself was explains for non-advanced graphics-programmers. Of course, there are lots of sources for code available online and also there are community blog-entries scattered across multiple platforms explaining parts of the problem as well as possible solutions, even though they typically are targeting different issues.

In this repository I want to condense my findings on how to implement a component to capture images to disk from an arbitrary UE4 (which fortunately apply likewise to UE5) scene from scratch lowering the bar for UE novices. This will include:

• Rendering images at high FPS without blocking the UE rendering thread
• Rendering segmentation (or other graphics buffers) at the same time

Disclaimer: I do not own any of the code. Merely, IO condensed the sources already available online for easier use and provide an overview to the general functionality of this particular approach!

Kudos to the UE4 AnswerHub community!

Plugin Support

The general idea of this repository is to communicate a possible setup for custom image capturing in code. This shall provide a baseline for further development to adapt the code to ones individual needs.

I understand that Unreal's Blueprint interface is powerful and some people have their reasons not to dive into C++ development with UE.

This is why now there is also a Plugin version of the code available. However, it will still need you to have a C++ project rather than Bluerpint-onle.

To incorporate the Plugin in to your project: Create a Plugins directory in your project and copy the \UnrealImageCapture\Plugins\CameraCaptureToDisk directory. Load the plugin in your project, if not automatically done by the editor, and place the CameraCaptureManager_BP, which is to be found in the plugin's contents, in the scene and fill in its required slots as depicted below. This will require you to place a SceneCapture2D in your scene. A PostProcessMaterial for segmentation is also located in the plugin's contents.

Currently one should use JPEG for Color and PNG for Pixel-Segmentation

An image-capturing-command can be triggered from Blueprint as exemplary depicted for the Level-Blueprint below:

How to Save Images to Disk In UE4 (without blocking the rendering thread)

I will go through the code step-by-step so that hopefully it will be easier to implement each step as you are following along. The full source code is placed in this repository.

Prerequisite

You will need a UE4 (or UE5) C++ project.

Also, you might have to add a few packages to your 'YourProjectName'.Build.cs file. These are part of UnrealEngine, however, sometimes they are not added automatically resulting in unpleasant linker errors. Find the 'YourProjectName'.Build.cs file in the Source/'YourProjectName/ directory, and add or extend it to include all modules listed in the following line:

PublicDependencyModuleNames.AddRange(new string[] {"Core", "CoreUObject", "Engine", "InputCore", "ImageWrapper", "RenderCore", "Renderer", "RHI" });

Setup A ColorCapture Component

I am using a SceneCaptureComponent2D as the basis for capturing images. Placing one of these into your scene will give you an ASceneCaptureComponent which is its Actor instance. It basically behaves like any other camera component, but its viewport is not restricted by your computer's monitor or main camera viewport. This provides us the possibility to render images of arbitrary resolution independent from the actual screen resolution.

Add a CaptureManager class of type Actor to your project.

In the CaptureManager.h we add the following:
CaptureManager.h

#pragma once
class ASceneCapture2D; //forward declaration

#include ...

and to our public variables:

// Color Capture  Components
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category="Capture")
ASceneCapture2D* ColorCaptureComponents;

This enables you to assign a CaptureComponent2d to your CaptureManager code.

Compile and place a CaptureManager in your scene.

As it does not have any primitive to render you will only see it in the editor's outline. In the details panel of the placed CaptureManager you can now see the ColorCaptureComponent assigned to None. From the drop down menu select the CaptureComponent2D you already placed in the scene.

Back to code: We will now prepare our yet "naked" CaptureComponent2D class for capturing images, creating and assigning a RenderTarget, which is basically a Texture to store our image data to, and setting the camera properties. Note: You could also do this in the Editor but if you deal with, i.e. multiple capture components, you may find it handy not to worry about creating and assigning all the components by hand!

Create a setup function to put all your setup code for the CaptureComponents in the CaptureManger:

CaptureManager.h

protected:

    void SetupColorCaptureComponent(ASceneCapture2D* captureComponent);

CaptureManager.cpp

#include ...

#include "Runtime/Engine/Classes/Components/SceneCaptureComponent2D.h"
#include "Runtime/Engine/Classes/Engine/TextureRenderTarget2D.h"
#include "Engine.h"
#include <Runtime/Engine/Classes/Kismet/GameplayStatics.h>
#include <Runtime/Engine/Public/ShowFlags.h>

#include "RHICommandList.h"

#include "IImageWrapper.h"
#include "IImageWrapperModule.h"
#include "ImageUtils.h"


void ACaptureManager::SetupColorCaptureComponents(ASceneCapture2D* captureComponent){
    // Create RenderTargets
    UTextureRenderTarget2D* renderTarget2D = NewObject<UTextureRenderTarget2D>();

    // Set FrameWidth and FrameHeight
    renderTarget2D->TargetGamma = 1.2f;// for Vulkan //GEngine->GetDisplayGamma(); // for DX11/12

    // Setup the RenderTarget capture format
    renderTarget2D->InitAutoFormat(256, 256); // some random format, got crashing otherwise
    int32 frameWidth = 640;
    int32 frameHeight = 480;
    renderTarget2D->InitCustomFormat(frameWidth, frameHeight, PF_B8G8R8A8, true); // PF_B8G8R8A8 disables HDR which will boost storing to disk due to less image information
    renderTarget2D->RenderTargetFormat = ETextureRenderTargetFormat::RTF_RGBA8;
    renderTarget2D->bGPUSharedFlag = true; // demand buffer on GPU

    // Assign RenderTarget
    captureComponent->GetCaptureComponent2D()->TextureTarget = renderTarget2D;

    // Set Camera Properties
    captureComponent->GetCaptureComponent2D()->CaptureSource = ESceneCaptureSource::SCS_FinalColorLDR;
    captureComponent->GetCaptureComponent2D()->ShowFlags.SetTemporalAA(true);
    // lookup more showflags in the UE4 documentation.. 
}

Call the code during BeginPlay of the CaptureManager

CaptureManager.cpp

// Called when the game starts or when spawned
void ACaptureManager::BeginPlay()
{
	Super::BeginPlay();

    // Setup CaptureComponents
    SetupColorCaptureComponent(ColorCaptureComponents);
}

Now that because we have a RenderTarget applied to our CaptureComponent we can read its data to disk.

Organize RenderRequests

We do this by basically re-implementing UE4's code for taking screenshots. Importantly, with the addition of not flushing our rendering pipeline. This prevents rendering hiccups that drop the framerate to 3 - 5 FPS.

This addition will come with the price of needing to handle 'waiting times' before an image is done and copied from GPU. This is important to prevent reading old or uninitialized buffers (remember that RenderThread and GameThread are asynchronous). We do this by keeping a queue of RenderRequest that we can probe for being completed.

We add the following struct to our CaptureManager class:

CaptureManager.h

#include ...

[...]

struct FRenderRequest{
    FIntPoint ImageSize;
    FRHIGPUTextureReadback Readback;
    FRenderCommandFence RenderFence;

    FRenderRequest(
        const FIntPoint& ImageSize,
        const FRHIGPUTextureReadback& Readback) :
            ImageSize(ImageSize),
            Readback(Readback) {}
};

[...]
UCLASS(Blueprintable)
class ...
[...]

The FRHIGPUTextureReadback will hold the rendered results, e.g. color or depth values. The RenderFence is a neat feature of UE, letting you put a 'fence' into the render pipeline that can be checked to notify when it has passed the full pipeline. This gives a way to determine whether our render command has passed as well.

We need to add a TQueue as a data structure to keep track of our render requests:

CaptureManger.h

protected:
    // RenderRequest Queue
    TQueue<TSharedPtr<FRenderRequest>> RenderRequestQueue;

Implement the image capturing function:

This function will place a render request on the UE rendering pipeline asking the data captured from our CaptureComponent to be copied in our Image buffer so that we can further process it in GameThread.

CaptureManger.h

public:
    UFUNCTION(BlueprintCallable, Category = "ImageCapture")
    void CaptureNonBlocking();

CaptureManger.cpp

void ACaptureManager::CaptureNonBlocking(){
    if(!IsValid(CaptureComponent)){
        UE_LOG(LogTemp, Error, TEXT("CaptureColorNonBlocking: CaptureComponent was not valid!"));
        return;
    }

    CaptureComponent->GetCaptureComponent2D()->TextureTarget->TargetGamma = GEngine->GetDisplayGamma();

    // Get RenderConterxt
    FTextureRenderTargetResource* renderTargetResource = CaptureComponent->GetCaptureComponent2D()->TextureTarget->GameThread_GetRenderTargetResource();

    // Init new RenderRequest
    TSharedPtr<FRenderRequestStruct> renderRequest = MakeShared<FRenderRequestStruct>(renderTargetResource->GetSizeXY(), FRHIGPUTextureReadback(TEXT("CameraCaptureManagerReadback")));

    // Send command to GPU
    ENQUEUE_RENDER_COMMAND(SceneDrawCompletion)(
    [renderTargetResource](FRHICommandListImmediate& RHICmdList) {
        FTexture2DRHIRef Target = renderTargetResource->GetRenderTargetTexture();
        RenderRequest->Readback.EnqueueCopy(RHICmdList, Target);
    });

    // Notifiy new task in RenderQueue
    RenderRequestQueue.Enqueue(renderRequest);

    // Set RenderCommandFence
    renderRequest->RenderFence.BeginFence();
}

With this, the image data is already stored in our queue, and we now need to store it to disk. Note: UFUNCTION(BlueprintCallable, Category = "ImageCapture") exposes this function to blueprint, so that you can easily test it

Save Image Data to Disk

To do so, in each tick of the CaptureManager we look up the first element of the RenderQueue, if it's RenderFence is completed and also whether the data is ready to read. Only if both check-out true, we proceed with saving the image to disk.

The last thing we need is a procedure to write the data to disk, preferably without blocking our GameThread.

We implement an asynchronous procedure storing the data to disk.

CaptureManager.h

UCLASS()
class ... {
[...]
};

class AsyncSaveImageToDiskTask : public FNonAbandonableTask{
    public:
        AsyncSaveImageToDiskTask(TArray<uint8> Image, FString ImageName);
        ~AsyncSaveImageToDiskTask();

    // Required by UE4!
    FORCEINLINE TStatId GetStatId() const{
        RETURN_QUICK_DECLARE_CYCLE_STAT(AsyncSaveImageToDiskTask, STATGROUP_ThreadPoolAsyncTasks);
    }

protected:
    TArray<uint8> ImageCopy;
    FString FileName = "";

public:
    void DoWork();
};

CaptureManager.cpp

#include ...

// Static ImageWrapperModule to prevent reloading -> this thing does not like to be reloaded..
static IImageWrapperModule &ImageWrapperModule = FModuleManager::LoadModuleChecked<IImageWrapperModule>(FName("ImageWrapper"));

[...]

AsyncSaveImageToDiskTask::AsyncSaveImageToDiskTask(TArray<uint8> Image, FString ImageName){
    ImageCopy = Image;
    FileName = ImageName;
}

AsyncSaveImageToDiskTask::~AsyncSaveImageToDiskTask(){
    //UE_LOG(LogTemp, Warning, TEXT("AsyncTaskDone"));
}

void AsyncSaveImageToDiskTask::DoWork(){
    FFileHelper::SaveArrayToFile(ImageCopy, *FileName);
    UE_LOG(LogTemp, Log, TEXT("Stored Image: %s"), *FileName);
}

And a call from the CaptureManager to start the async saving process:

CaptureManager.h

protected:
    // Creates an async task that will save the captured image to disk
    void RunAsyncImageSaveTask(TArray<uint8> Image, FString ImageName);

CaptureManager.cpp

void ACaptureManager::RunAsyncImageSaveTask(TArray64<uint8> Image, FString ImageName){
    (new FAutoDeleteAsyncTask<AsyncSaveImageToDiskTask>(Image, ImageName))->StartBackgroundTask();
}

Override the Tick function of the CaptureManager:

CaptureManager.h

public:	
	// Called every frame
	virtual void Tick(float DeltaTime) override;

CaptureManager.cpp

// Called every frame
void ACameraCaptureManager::Tick(float DeltaTime)
{
	Super::Tick(DeltaTime);

	 // Read pixels once RenderFence is completed
    if(!RenderRequestQueue.IsEmpty()){
        // Peek the next RenderRequest from queue
        TSharedPtr<FRenderRequestStruct> nextRenderRequest = *RenderRequestQueue.Peek();

        //int32 frameWidht = 640;
        //int32 frameHeight = 480;

        if(nextRenderRequest){ //nullptr check
            if(nextRenderRequest->RenderFence.IsFenceComplete() && nextRenderRequest->Readback.IsReady()) { // Check if rendering is done, indicated by RenderFence & Readback

                // Load the image wrapper module 
                IImageWrapperModule& ImageWrapperModule = FModuleManager::LoadModuleChecked<IImageWrapperModule>(FName("ImageWrapper"));

                // Get Data from Readback
                int64 RawSize = nextRenderRequest->ImageSize.X * nextRenderRequest->ImageSize.Y * sizeof(FColor);
                void* RawData = nextRenderRequest->Readback.Lock(RawSize);


                // Decide storing of data, either jpeg or png
                FString fileName = "";
                if(UsePNG){
                    //Generate image name
                    fileName = FPaths::ProjectSavedDir() + SubDirectoryName + "/img" + "_" + ToStringWithLeadingZeros(ImgCounter, NumDigits);
                    fileName += ".png"; // Add file ending

                    // Prepare data to be written to disk
                    static TSharedPtr<IImageWrapper> imageWrapper = ImageWrapperModule.CreateImageWrapper(EImageFormat::PNG); //EImageFormat::PNG //EImageFormat::JPEG
                    imageWrapper->SetRaw(RawData, RawSize, FrameWidth, FrameHeight, ERGBFormat::BGRA, 8);
                    const TArray<uint8>& ImgData = imageWrapper->GetCompressed(5);
                    RunAsyncImageSaveTask(ImgData, fileName);
                } else{
                    // Generate image name
                    fileName = FPaths::ProjectSavedDir() + SubDirectoryName + "/img" + "_" + ToStringWithLeadingZeros(ImgCounter, NumDigits);
                    fileName += ".jpeg"; // Add file ending

                    // Prepare data to be written to disk
                    static TSharedPtr<IImageWrapper> imageWrapper = ImageWrapperModule.CreateImageWrapper(EImageFormat::JPEG); //EImageFormat::PNG //EImageFormat::JPEG
                    imageWrapper->SetRaw(RawData, RawSize, FrameWidth, FrameHeight, ERGBFormat::BGRA, 8);
                    const TArray<uint8>& ImgData = imageWrapper->GetCompressed(0);
                    RunAsyncImageSaveTask(ImgData, fileName);
                }

                if(VerboseLogging && !fileName.IsEmpty()){
                    UE_LOG(LogTemp, Warning, TEXT("%f"), *fileName);
                }

                ImgCounter += 1;

                // Delete the first element from RenderQueue
                RenderRequestQueue.Pop();

                UE_LOG(LogTemp, Log, TEXT("Done..."));
            }
        }
    }

}

For testing purposes we can call the CaptureColorBlocking() from the LevelBlueprint by attaching it to a button pressed event.

[Image of the level blueprint]

The captured images will now be saved into your project's Saved directory.

Capturing Segmentation

To get labels for our images we will add a second CaptureComponent equipped with a PostProcessMaterial that visualizes CustomDepth. The CustomDepthStencil is settable for each actor in the scene, effectively letting us label and visualize categories of, as well as individual, actors.

Enable Custom Depth Stencils

Find the ProjectSettings in your editor and search for stencil which will bring up Custom Depth-Stencil Pass. Switch this option from Enabled to Enabled with Stencil.

Setting Up The PostProcess Material

Add a new Material to your project content. (I will call it PP_Segmentation)

Click on the material's output node and switch MaterialDomain from Surface to PostProcess.

Right-click to open the node search and type SceneTexture, select the node from Texture-Category.

In the details of this node, select CustomStencil as SceneTextureId.

Add a Division node and connect the SceneTexture's Color output to the division node. Set the division to be by 255. Note: This is needed because the image buffer seems to be float valued, leading to values > 1 having no meaning, as image information ranges from 0.0 to 1.0.

Apply and save the material.

Setting up Custom-Depth Stencils

You can set the custom depth in editor or from code. For simplicity I chose the editor. Place an arbitrary object(MeshActor) into the scene, and search for custom depth in its details panel. Under Rendering enable Render CustomDepth Pass, and set CustomDepth Stencil Value to whatever you like. For illustration purposes set it to 200.

Organize the Segmentation CaptureComponent

To be able to render color and segmentation at the same time, we need a second SceneCapture2D component in our scene. To not worry about placement and setup later on we will spawn this component by code, aligning it to our ColorCapture, and add our post process material.

To add the post process material we first need access to it by code. We could do a search for it through our project content, but since this would be done by the name of the material I found it to be a rather unsafe method. I prefer to add a reference to the material to the CaptureManager

CaptureManager.h

public:
    // PostProcessMaterial used for segmentation
    UPROPERTY(EditAnywhere, Category="Segmentation Setup")
    UMaterial* PostProcessMaterial = nullptr;

You can now reference the post process material in the details panel of the CaptureManager in the editor just like before the SceneCapture2D.

Spawn the SegmentationCapture Component

Add code to spawn the new SceneCapture2D component and get settings from the ColorCapture:

CaptureManager.h

protected:
    ASceneCapture2D* SegmentationCapture = nullptr;

    void SpawnSegmentationCaptureComponent(ASceneCapture2D* ColorCapture);

CaptureManager.cpp

void ACaptureManager::SpawnSegmentationCaptureComponent(ASceneCapture2D* ColorCapture){
    // Spawning a new SceneCaptureComponent
    ASceneCapture2D* newSegmentationCapture = (ASceneCapture2D*) GetWorld()->SpawnActor<ASceneCapture2D>(ASceneCapture2D::StaticClass());
    if(!newSegmentationCapture){ // nullptr check
        UE_LOG(LogTemp, Error, TEXT("Failed to spawn SegmentationComponent"));
        return;
    }
    // Register new CaptureComponent to game
    newSegmentationCapture->GetCaptureComponent2D()->RegisterComponent();
    // Attach SegmentationCaptureComponent to match ColorCaptureComponent
    newSegmentationCapture->AttachToActor(ColorCapture, FAttachmentTransformRules::SnapToTargetNotIncludingScale);

    // Get values from "parent" ColorCaptureComponent
    newSegmentationCapture->GetCaptureComponent2D()->FOVAngle = ColorCapture->GetCaptureComponent2D()->FOVAngle;

    // Set pointer to new segmentation capture component
    SegmentationCapture = newSegmentationCapture;

    UE_LOG(LogTemp, Warning, TEXT("Done..."));
}

Setup the SegmentationComponent

CaptureManager.h

protected:
    void SetupSegmentationCaptureComponent(ASceneCapture2D* SegmentationCapture);

CaptureManager.cpp

void ACaptureManager::SetupSegmentationCaptureComponent(ASceneCapture2D* ColorCapture){
    // Spawn SegmentationCaptureComponents
    SpawnSegmentationCaptureComponent(ColorCapture);

    // Setup SegmentationCaptureComponent
    SetupColorCaptureComponent(SegmentationCapture);

    // Assign PostProcess Material
    if(PostProcessMaterial){ // check nullptr
        SegmentationCapture->GetCaptureComponent2D()->AddOrUpdateBlendable(PostProcessMaterial);
    } else {
        UE_LOG(LogTemp, Error, TEXT("PostProcessMaterial was nullptr!"));
    }
}

Add the function call to BeginPlay

CaptureManager.cpp

void ACaptureManager::BeginPlay()
{
	Super::BeginPlay();

    SetupColorCaptureComponent(ColorCaptureComponents);
    SetupSegmentationCaptureComponent(ColorCaptureComponents);
}

To save the image information from SegmentationCapture we can simply use the CaptureColorNonBlocking() method. Be sure to set isSegmentation = true to get PNG compressed data.

Known Issues

The IImageWrapperModule's wrapping of the data is still done in GameThread rather than in an async call, which can actually consume more runtime than the saving to disk. Simply pushing the WrapperModule into the async procedure does suffice since 1) it is a shared pointer, 2) the ImageWrapperModule.CreateImageWrapper(...) needs to be called from GameThread. I am grateful for any ideas on that..

It is possible that an image is saved every game tick at high fps. If saving to disk is actually slower than the delta time of the game tick another call to the shared IImageWrapper is made while its buffer is read for saving to disk. This results in a game crash. This should be fixable by adding semaphores, I just did not have the time to test this yet.

unrealimagecapture's Issues

Save raw scene depth values to disk

Hey there,

I was wondering if you could point me in right direction of using your code to access the raw scene depth (as given by the "Scene Depth" node in the material editor) and save it to disk.

I managed to get a 0-255 grayscaled depth image using a post processing material, but I could not manage yet to get a hold of the scene depth using Blueprint or C++, since the raw scene has a much higher precision than my approach.

One would have to access the gbuffer somehow, but I could not find any good resource on that topic yet.

Any hints are much appreciated.

Depth data

After all your help, I was able to capture the wanted images.

One thing I don't understand is how to get the depth values.

When I'm opening the depth (exr files) they look very similar to the segmentation.

How the depth is calculated.

I'm attaching what I tried:

How to Access Image Data in C++ (not save to disk)

Hi,

Apologies for the newby question. I have a 2DSceneCapture component that I have very simply made to follow an actor. I want to access the data from the scene capture in C++ so I can do some postprocessing with the image. I don't want to save to disk however.

Is this code under the Setup A ColorCapture Component section in your read me assigning the raw image data from the scene capture to the variable renderTarget2D? So my question is, is this render target basically holding the image data?:

// Assign RenderTarget
    captureComponent->GetCaptureComponent2D()->TextureTarget = renderTarget2D;

Thanks in advance

Segmentation

Hi again,

This time I used the plugin in CityPark.
I'm able to save color and depth but the segmentation comes out black.
I repeated the same steps as in this project. I would like to get different segmentation for trees/ sidewalk for example.
Thank you.

weird camera exposure & color

Hi, It's me again :)
I have one problem. When rendering pictures by code, I got weird camera exposure and color issues. The detail can be seen in the following, the first is the preview screen when hit the play button in the ue4 editor, and the second is rendered by code.

I'm running the rendering code as you are, including the config of the UTextureRenderTarget2D. I think the problem comes from the config of UTextureRenderTarget2D and I don't know how to modify the config. Am I right or do you know the reason? Thanks a lot~

Running the image taking

Hi there,
I've just pulled your project and opened CaptureToDisk Unreal project.
But I got stuck generating the images. Can you help me, what exactly do I have to do to start saving images?
Many thanks in advance

Cannot capture images

Hello @TimmHess ! Thank you so much for your outstanding work! I am currently working on generating synthesized RGB data with corresponding depth for my deep learning training. I am very new to the unreal engine and this area but I can follow your steps in readme and try to manage to capture materials. However, with my blueprint setting, I cannot see any log info and images cannot be saved in my directory. What am I doing wrong? Or how can I achieve this by capturing RGB and depth in pure c++?(I am ok with c++)

Attached are my blueprint settings.

Thanks in advance! Hope you will spare your time to save my silly question...

The image captured has lower quality than the image rendered in Game Play.

I use a camera actor, and attached a SceneCaptured2D component, and use CamereCaptureToDisk plugin. I can save images while game playing, but the quality of saved image is much lower than the Game Play shows. How to adjust the rendering settings of this plugin?

Crash when using depth

Dear TimmHess

Thanks for your code! I am trying to use your plugin to obtain scene depth but crashes every time when I tried to capture depth image.

I am using UE5.03 and created a blank C++ project. I added the UE5 plugin to the project, add the CameraCaptureManager_BP to the scene, and added a SceneCapture2D as capture component, used PP_depth as the material

The level blueprint is shown as follow.

When I tried to capture depth, the UE crashed at line
imageWrapper->SetRaw(nextRenderRequest->Image.GetData(), nextRenderRequest->Image.GetAllocatedSize(), FrameWidth, FrameHeight, ERGBFormat::BGRA, 16);

I got the error. Assertion failed: (int64)ChannelCount * Width * Height * RawBitDepth == RawData.Num() * 8
I am check the image size using nextRenderRequest->Image.GetAllocatedSize(), it returns 2457600 = 640x480x8. should that be 640x480x8x4 since there are RGBA 4 channels?

I really hope you can shed light on my question and help me here, I've been struggling for last 2 days.

Your help is really appreciated!

Plugin version?

Hi,
I'd like to know if it would be difficult to create a plugin from your code?
I'm using blueprints (I'm not a coder), and I think it would be great to have a plugin that could let the users choose the outputs they need (ground truth, semantic segmentation, depth, normals...).
I can't use UnrealCV anymore (no more update), so your code could be a great alternative!
Best,
n.

Sample Project and plugins seems to crash when attempting render in UE 5.0.3 and 5.1.0

As the title claims, I cannot get a image capture to compleate without crashing unreal. I've tried compilin the sample project in both UE 5.0.3 and 5.1.0, and also tried installing it as a plugin in a fresh project. Same result.

I'm triggering the capture in a utility widget like so.

Error message always throws at line 185 in "C:\Users\*****\Documents\Unreal Projects\ImageCapture\Plugins\CameraCaptureToDisk\Source\CameraCaptureToDisk\Private\CameraCaptureManager.cpp". Attempting to run a float-non-blocking results in similar crash at line 250.

Error:

LoginId:9b7e3dc9443559f5ff884dad673f1823
EpicAccountId:1ea2120a2eaa42989234477b1afde595

Unhandled Exception: EXCEPTION_ACCESS_VIOLATION writing address 0x00000000000004b0

UnrealEditor_CameraCaptureToDisk!ACameraCaptureManager::CaptureNonBlocking() [C:\Users\******\Desktop\CaptureToDisk\Plugins\CameraCaptureToDisk\Source\CameraCaptureToDisk\Private\CameraCaptureManager.cpp:185]
UnrealEditor_CoreUObject
UnrealEditor_CoreUObject
UnrealEditor_CoreUObject
UnrealEditor_CoreUObject
UnrealEditor_CoreUObject
UnrealEditor_CoreUObject
UnrealEditor_CoreUObject
UnrealEditor_CoreUObject
UnrealEditor_CoreUObject
UnrealEditor_CoreUObject
UnrealEditor_UMG
UnrealEditor_UMG
UnrealEditor_UMG
UnrealEditor_Slate
UnrealEditor_Slate
UnrealEditor_Slate
UnrealEditor_Slate
UnrealEditor_Slate
UnrealEditor_Slate
UnrealEditor_Slate
UnrealEditor_ApplicationCore
UnrealEditor_ApplicationCore
UnrealEditor_ApplicationCore
UnrealEditor_ApplicationCore
user32
user32
UnrealEditor_ApplicationCore
UnrealEditor
UnrealEditor
UnrealEditor
UnrealEditor
UnrealEditor
UnrealEditor
kernel32
ntdll

Access violation

Heya, i'm having some issues when trying to use your plugin (which i greatly appreciate that even exists! shame the others got abandoned) which is triggering an access violation when i try to implement it into my scene.

I've got the CameraCaptureManager_BP in my scene with the same settings (it was basically a copy and paste) from your demo scene.
whilst typing this i noticed there wasnt a "/" in the sub directory name, i tested it with a "/" and still same result.

with the same node network in my level BP

And i've also got the ActorSceneCaptureComp2d as a child of my pawn, just like your demo scene.

But within your demo scene I've also tried to move the network to run the "Capture Float Non Blocking" node inside my actor instead of the level BP and disconnected the network in the level BP. This also seems to trigger a memory access violation in the line.

I don't suppose you've yet to run into this? I'm on UE 5.1.1
My end goal would be to get the image in C++ and not to save to disk. It was great to see that others are looking to try and do this as well as we may be having the same goal. Hoping to get the raw 32bit image later down the line though.

Thanks

How to improve the quality of captured images

Hi! I tried to extract visual features from these captured images, but just found that the image quality is not good enough for supporting my algorithm to run through it.

So I really want to know if there is any way to improve the quality of stored images (I don't care whether huge volumn of raw data will slow down my game or not) and forget about the performance, the clearer the images I get the better!

For now, I've tried several ways but none of them have achieved my goal.

• increase the value of renderTarget2D->SizeX in SetupColorCaptureComponent() does not provide any help
• change renderTarget2D->InitCustomFormat(640, 480, PF_B8G8R8A8, true); to
  renderTarget2D->InitCustomFormat(640, 480, PF_R32G32B32A32_UINT, true); and
  renderTarget2D->RenderTargetFormat = ETextureRenderTargetFormat::RTF_RGBA32f.
  Of course the same operation will be done in the Tick() function, but this end up to a bad result that AsyncSaveImageToDiskTask cannot cope with TArray<uint32>
• in SetupColorCaptureComponent(), add captureComponent->GetCaptureComponent2D()->ShowFlags.ScreenPercentage = 200 and
  captureComponent->GetCaptureComponent2D()->ShowFlags.SetScreenPercentage(true)
  this was done since I found that if you increase the "Screen Percentage" value from the menu in viewport, the quality of showing pictures in viewport will become much better and I don't know why. But adding this in C++ codes does not get any improvement either.

Linking error of ACaptureManager.cpp

Hi there! I am new to UE4 and I just found your job of solving the image saving stuck problem.
I tried all my best to practice your solution but VS just brought up so much wrong info during the compiling process.

• (1) cannot convert to "const TArray<uint8, FDefaultAllocator> &" from "const TArray<uint8, FDefaultAllocator64>"

• (2) LNK2019 �޷��������ⲿ���� "__declspec(dllimport) public: void __cdecl FRHICommandListImmediate::ReadSurfaceData(class FRHITexture *,struct FIntRect,class TArray<struct FColor,class TSizedDefaultAllocator<32> > &,class FReadSurfaceDataFlags)" (_imp?ReadSurfaceData@FRHICommandListImmediate@@QEAAXPEAVFRHITexture@@UFIntRect@@aeav?$TArray@UFColor@@v?$TSizedDefaultAllocator@$0CA@@@@@VFReadSurfaceDataFlags@@@z)���÷����ں��� "public: void __cdecl ACaptureManager::CaptureColorNonBlocking(class ASceneCapture2D *,bool)" (?CaptureColorNonBlocking@ACaptureManager@@QEAAXPEAVASceneCapture2D@@_N@Z) �б����� IMUTrial C:\Users\l00595286\Documents\Unreal Projects\IMUTrial\Intermediate\ProjectFiles\CaptureManager.cpp.obj 1

Problem (1) happened when "const TArray& ImgData = imageWrapper->GetCompressed(5);" in void ACaptureManager::Tick() function from CaptureManager.cpp. And I solved this by adding a const_cast<> to force the converting

Problem (2) just cannot be solved and in fact, I do not really kown why there is so much wierd symbols instead of showing English.

I also tried to copy your code into my game project(I have adapted it to suit my running environment), but the same error happened and I just don't know what I can do about that.

Depth & Float Capture?

Does float capture & depth capture work right now? I can't seem to get this working, at least in UE5.0.3.

The color RGBA capture works fine though.

Read access on imageWrapper->SetRaw...

Maybe I configured something wrong?

I think the code creates it own render target and such, but I used:
RenderTarget setup with RGBA16f
SceneCaptureComponent2D setup with HDR in RGB, SceneDepth in A.
CameraCaptureManager setup with UseFloat on and the capture component set. UsePNG off.

Error inserting CameraCaptureManager_BP blueprint into the scene

Hey there!

First of all let me congratulate you for this plugin. I haven't been able to get it running yet but it looks really well written and should be a great starting point for trying to understand what's going on in the guts of Unreal Engine.

I'm using UE 5.2.0 and I had some trouble adding CameraCaptureManager_BP.uasset to the scene. The editor was always giving errors such as:

VerifyImport: Failed to find script package for import object 'Package /Script/CameraCaptureToDisk'
CreateExport: Failed to load Parent for BlueprintGeneratedClass /CameraCaptureToDisk/CameraCaptureManager_BP.CameraCaptureManager_BP_C
Unable to load Default__CameraCaptureManager_BP_C with outer Package /CameraCaptureToDisk/CameraCaptureManager_BP because its class (CameraCaptureManager_BP_C) does not exist

I tried both using the project already included in your repo and creating new projects (with all kinds of configurations I could think of), without success.

At first I thought what was missing was the reference to the the C++ class, so I tried moving the CameraCaptureManager.cpp/.h pair pretty much everywhere, again without success. I finally came across a post about redirects in Config/DefaultEngine.ini and that seems to have finally solved my problem.

Basically, after creating a new project with the first person template (called TestImageCapture1), I copied CameraCaptureManager.cpp/.h to Source/TestImageCapture1, CameraCaptureManager_BP.uasset to Content/CameraCaptureToDisk/Blueprints and I added these two lines to Config/DefaultEngine.ini under the [/Script/Engine.Engine] section:

+ActiveGameNameRedirects=(OldGameName="/Script/CameraCaptureToDisk.CameraCaptureManager",NewGameName="/Script/TestImageCapture1.CameraCaptureManager")
+ActiveClassRedirects=(OldClassName="/Script/CameraCaptureToDisk.CameraCaptureManager",NewClassName="/Script/TestImageCapture1.CameraCaptureManager")

Now the blueprint has a parent again and I can add it to the scene.

Save image from target render every frame.

Hello Everyone,
I am trying to capture and save image to disc every frame from a camera sensor in unreal engine 5.2. At every tick i try to capture and save image to disc. i tried to use code from this repo. But when i try to save every frame, scene freezes and same image is rendered at every frame. but if i capture image at alternate frame it works. Here are my code. I would really appreciate any help.
// PinHoleCineCamera.cpp
#include "PinHoleCineCamera.h"
#include "Engine.h"
#include "Engine/SceneCapture2D.h"
#include "Runtime/Engine/Classes/Components/SceneCaptureComponent2D.h"
#include "CineCameraComponent.h"
#include "Kismet/GameplayStatics.h"
#include "ShowFlags.h"
#include "RHICommandList.h"
#include "ImageWrapper/Public/IImageWrapper.h"
#include "ImageWrapper/Public/IImageWrapperModule.h"
#include "ImageUtils.h"
#include "Modules/ModuleManager.h"
#include "Misc/FileHelper.h"
#include "Engine/TextureRenderTarget2D.h"
#include "Engine/Texture2D.h"
#include "Misc/FileHelper.h"
#include "IImageWrapper.h"
#include "HAL/PlatformProcess.h"
#include "RenderingThread.h"
#include "ImageUtils.h"
#include "Engine/World.h"
#include "Engine/GameViewportClient.h"
#include "IImageWrapperModule.h"

APinHoleCineCamera::APinHoleCineCamera(const FObjectInitializer& ObjectInitializer) : ACineCameraActor(ObjectInitializer)
{
//RootComponent = CreateDefaultSubobject(TEXT("RootComponent"));
UCineCameraComponent* CineCameraComponent1 = GetCineCameraComponent();
SceneCaptureComponent = CreateDefaultSubobject(TEXT("SceneCaptureComponent"));
SceneCaptureComponent->AttachToComponent(CineCameraComponent1, FAttachmentTransformRules::KeepRelativeTransform);
// Set the SceneCaptureComponent properties
SceneCaptureComponent->ProjectionType = ECameraProjectionMode::Perspective;
//SceneCaptureComponent->OrthoWidth = 0.0f; // Set OrthoWidth to 0 for perspective projection
SceneCaptureComponent->FOVAngle = 93.665;
// Set Camera Properties
SceneCaptureComponent->CaptureSource = ESceneCaptureSource::SCS_FinalColorHDR;
SceneCaptureComponent->ShowFlags.SetTemporalAA(true);
}
void APinHoleCineCamera::AttachPinHoleCameraToVehicle(AActor* Vehicle, PINHOLECINECAMERACfg SensorCfg_, float FixedDeltaTime_) {

SensorCfg = SensorCfg_;
FixedDeltaTime = FixedDeltaTime_;
if (Vehicle) {
    SetActorRelativeLocation(SensorCfg.TranslationSensorToVehicle);
    SetActorRelativeRotation(SensorCfg.RotationSensorToVehicle);
    if (AttachToActor(Vehicle, FAttachmentTransformRules::KeepRelativeTransform)) {
        UE_LOG(LogTemp, Log, TEXT("PinHole: Sensor attached to EGO vehicle."));
    }
    else {
        UE_LOG(LogTemp, Error, TEXT("PinHole: Error during sensor attachment to EGO vehicle."));
    }
}
else {
    UE_LOG(LogTemp, Error, TEXT("PinHole: No parent vehicle found."));
}
// Make sure that output folder exists
if (!FPaths::DirectoryExists(SensorCfg.PathSave)) {
    FPlatformFileManager::Get().GetPlatformFile().CreateDirectoryTree(*SensorCfg.PathSave);
}  
UE_LOG(LogTemp, Log, TEXT("[PinHoleCineCamera] Pin Hole camera settings applied"));

}

void APinHoleCineCamera::GetPinHoleCineCameraData() {
const uint32_t Timestamp = std::round(FixedDeltaTime * Frame * 1000);
Frame = Frame + 1U;
// Read pixels once RenderFence is completed
if (!RenderRequestQueue.IsEmpty()) {
// Peek the next RenderRequest from queue
FRenderRequestStruct* nextRenderRequest = nullptr;
RenderRequestQueue.Peek(nextRenderRequest);
if (nextRenderRequest) { //nullptr check
if (nextRenderRequest->RenderFence.IsFenceComplete()) { // Check if rendering is done, indicated by RenderFence
// Load the image wrapper module
IImageWrapperModule& ImageWrapperModule = FModuleManager::LoadModuleChecked(FName("ImageWrapper"));
FString FrameNumberString = FString::Printf(TEXT("%06d.png"), Timestamp);
FString FilePath = FPaths::Combine(SensorCfg.PathSave, FrameNumberString);
static TSharedPtr imageWrapper = ImageWrapperModule.CreateImageWrapper(EImageFormat::PNG); //EImageFormat::PNG //EImageFormat::JPEG
imageWrapper->SetRaw(nextRenderRequest->Image.GetData(), nextRenderRequest->Image.GetAllocatedSize(), FrameWidth, FrameHeight, ERGBFormat::BGRA, 8);
const TArray64& ImgData = imageWrapper->GetCompressed(5);
//const TArray& ImgData = static_cast<TArray<uint8, FDefaultAllocator>> (imageWrapper->GetCompressed(5));
RunAsyncImageSaveTask(ImgData, FilePath);
}
}
}
}

void APinHoleCineCamera::CaptureNonBlocking() {
if (!IsValid(SceneCaptureComponent)) {
UE_LOG(LogTemp, Error, TEXT("CaptureColorNonBlocking: CaptureComponent was not valid!"));
return;
}
SceneCaptureComponent->TextureTarget->TargetGamma = GEngine->GetDisplayGamma();
FTextureRenderTargetResource* renderTargetResource = SceneCaptureComponent->TextureTarget->GameThread_GetRenderTargetResource();

UE_LOG(LogTemp, Warning, TEXT("Got display gamma"));
struct FReadSurfaceContext { FRenderTarget* SrcRenderTarget;
                             TArray<FColor>* OutData;
                             FIntRect Rect;
                             FReadSurfaceDataFlags Flags;};

UE_LOG(LogTemp, Warning, TEXT("Inited ReadSurfaceContext"));
// Init new RenderRequest
FRenderRequestStruct* renderRequest = new FRenderRequestStruct();
UE_LOG(LogTemp, Warning, TEXT("inited renderrequest"));

// Setup GPU command
FReadSurfaceContext readSurfaceContext = {
    renderTargetResource,
    &(renderRequest->Image),
    FIntRect(0,0,renderTargetResource->GetSizeXY().X, renderTargetResource->GetSizeXY().Y),
    FReadSurfaceDataFlags(RCM_UNorm, CubeFace_MAX)
};
UE_LOG(LogTemp, Warning, TEXT("GPU Command complete"));
// Above 4.22 use this
ENQUEUE_RENDER_COMMAND(SceneDrawCompletion)(
    [readSurfaceContext](FRHICommandListImmediate& RHICmdList) {
        RHICmdList.ReadSurfaceData(
            readSurfaceContext.SrcRenderTarget->GetRenderTargetTexture(),
            readSurfaceContext.Rect,
            *readSurfaceContext.OutData,
            readSurfaceContext.Flags
        );
    });

// Notifiy new task in RenderQueue
RenderRequestQueue.Enqueue(renderRequest);
// Set RenderCommandFence
renderRequest->RenderFence.BeginFence();
renderRequest->RenderFence.Wait();
if (renderRequest->RenderFence.IsFenceComplete()) {
    UE_LOG(LogTemp, Warning, TEXT("fencing complete"));
    GetPinHoleCineCameraData();
    // Delete the first element from RenderQueue
    RenderRequestQueue.Pop();
    delete renderRequest;
}
else {
    UE_LOG(LogTemp, Error, TEXT("fencing not complete"));
    FPlatformProcess::Sleep(0.01f);
    CaptureNonBlocking();
}

}
void APinHoleCineCamera::RunAsyncImageSaveTask(TArray64 Image, FString ImageName) {
(new FAutoDeleteAsyncTask(Image, ImageName))->StartBackgroundTask();
}

AsyncSaveImageToDiskTask::AsyncSaveImageToDiskTask(TArray64 Image, FString ImageName) {
ImageCopy = Image;
FileName = ImageName;
}

AsyncSaveImageToDiskTask::~AsyncSaveImageToDiskTask() {
UE_LOG(LogTemp, Warning, TEXT("AsyncTaskDone"));
}

void AsyncSaveImageToDiskTask::DoWork() {
FFileHelper::SaveArrayToFile(ImageCopy, *FileName);
//UE_LOG(LogTemp, Error, TEXT("Stored Image: %s"), *FileName);
}
// PinHoleCineCamera.h
#pragma once

#include "CoreMinimal.h"
#include "GameFramework/Actor.h"
#include "CineCameraActor.h"
#include "Containers/Queue.h"
#include "Engine/SceneCapture2D.h"
#include "Components/SceneCaptureComponent2D.h"
#include "PinHoleCineCamera.generated.h"

struct PINHOLECINECAMERACfg {
// Rotation from sensor coordinate system to vehicle coordinate system.
FRotator RotationSensorToVehicle;
// Translation from sensor coordinate system to vehicle coordinate system.
FVector TranslationSensorToVehicle;
// Path where to save the csv logger file.
FString PathSave;
// Sensor frequency
float Frequency;
};

USTRUCT()
struct FRenderRequestStruct {
GENERATED_BODY()
TArray Image;
FRenderCommandFence RenderFence;
FRenderRequestStruct() {}
};

UCLASS()
class AVL_UE5_API APinHoleCineCamera : public ACineCameraActor{
GENERATED_BODY()
public:
APinHoleCineCamera(const FObjectInitializer& ObjectInitializer);
protected:
//virtual void BeginPlay() override;
virtual void BeginPlay() override {
Super::BeginPlay();
};

public:
//virtual void Tick(float DeltaTime) override;
void AttachPinHoleCameraToVehicle(AActor* Vehicle, PINHOLECINECAMERACfg SensorCfg, float FixedDeltaTime);
void GetPinHoleCineCameraData();
UFUNCTION(BlueprintCallable, Category = "ImageCapture")
void CaptureNonBlocking();
//bool IsPreviousFenceComplete() const;
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Capture")
int FrameWidth = 4112;
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Capture")
int FrameHeight = 2176;
protected:
int ImgCounter = 0;
void RunAsyncImageSaveTask(TArray64 Image, FString ImageName);
// RenderRequest Queue
TQueue<FRenderRequestStruct*> RenderRequestQueue;
private:
UPROPERTY(EditAnywhere, Category = "Capture")
USceneCaptureComponent2D* SceneCaptureComponent;
PINHOLECINECAMERACfg SensorCfg;
FString TestSequenceNumber;
float FixedDeltaTime;
int32_t Frame = 0;
};

class AsyncSaveImageToDiskTask : public FNonAbandonableTask {
public:
AsyncSaveImageToDiskTask(TArray64 Image, FString ImageName);
~AsyncSaveImageToDiskTask();
FORCEINLINE TStatId GetStatId() const {
RETURN_QUICK_DECLARE_CYCLE_STAT(AsyncSaveImageToDiskTask, STATGROUP_ThreadPoolAsyncTasks);
}
protected:
TArray64 ImageCopy;
FString FileName = "";
public:
void DoWork();
};

Getting raw images to be sent via TCP

Hey! I just bumped into this useful plugin, since I am trying to get frames from two (or more) Scene Capture components to be sent via TCP. I managed to get it working with a simple C++ class with a single camera, but I am new to both Unreal and C++, so I can't go further. Did you ever try to implement this kind of pipeline?

Rendering PNG not working

Hi there,
I've tried rendering PNGs in the sample project and it doesn't generate anything. It first the image taking flow but then the folder remains empty...
Do you have any idea where to start looking to fix it?

cant run project after package

after add "RHI", "Renderer", "RenderCore" modules,it's can run as editor, but after package ,it cant run ,even dont create any log.

Crash

Hi,
I downloaded your project, compiled it and tried to run it.
But when I press C the game crush with no apparent error.
Do you know why?

The RenderRequest is not defined

I'm using this code in vs2022, the compile returns not defined error. Am I missing something? Any hint will be appreciated.

ENQUEUE_RENDER_COMMAND(SceneDrawCompletion)(
[renderTargetResource](FRHICommandListImmediate& RHICmdList) {
FTexture2DRHIRef Target = renderTargetResource->GetRenderTargetTexture();
RenderRequest->Readback.EnqueueCopy(RHICmdList, Target);
});

Packaging project

Hi there,
I've tried to package the project but apparently, it doesn't work. This is the error I'm having:
UATHelper: Packaging (Windows (64-bit)): CaptureManager.cpp.obj : error LNK2005: "public: __cdecl AsyncSaveImageToDiskTask::~AsyncSaveImageToDiskTask(void)" (??1AsyncSaveImageToDiskTask@@QEAA@XZ) already defined in Module.CameraCaptureToDisk.cpp.obj UATHelper: Packaging (Windows (64-bit)): CaptureManager.cpp.obj : error LNK2005: "public: void __cdecl AsyncSaveImageToDiskTask::DoWork(void)" (?DoWork@AsyncSaveImageToDiskTask@@QEAAXXZ) already defined in Module.CameraCaptureToDisk.cpp.obj UATHelper: Packaging (Windows (64-bit)): Creating library C:\Users\Administrator\Desktop\UnrealImageCapture-master (1)\UnrealImageCapture-master\CaptureToDisk\Binaries\Win64\CaptureToDisk.lib and object C:\Users\Administrator\Desktop\UnrealImageCapture-master (1)\UnrealImageCapture-master\CaptureToDisk\Binaries\Win64\CaptureToDisk.exp UATHelper: Packaging (Windows (64-bit)): C:\Users\Administrator\Desktop\UnrealImageCapture-master (1)\UnrealImageCapture-master\CaptureToDisk\Binaries\Win64\CaptureToDisk.exe : fatal error LNK1169: one or more multiply defined symbols found UATHelper: Packaging (Windows (64-bit)): Took 18.544797s to run UnrealBuildTool.exe, ExitCode=6 UATHelper: Packaging (Windows (64-bit)): UnrealBuildTool failed. See log for more details. (C:\Users\Administrator\AppData\Roaming\Unreal Engine\AutomationTool\Logs\C+Program+Files+Epic+Games+UE_4.26\UBT-CaptureToDisk-Win64-Development.txt) UATHelper: Packaging (Windows (64-bit)): AutomationTool exiting with ExitCode=6 (6) UATHelper: Packaging (Windows (64-bit)): BUILD FAILED PackagingResults: Error: Unknown Error

Do you have any idea how I can fix this?

Many thanks in advance

Cannot save .png or .jpeg image

Hi,
I set Use Float to false to export .png file, but when I press the control button, the simulation engine crashes.

Upgrade to UE4.26

There's an TArray change to lines 80 & 90 of CameraCaptureManager.cpp. In UE4.26, IImageWrapper->GetCompressed returns a TArray64, however RunAsyncImageSaveTask digests a TArray. Need to convert the TArray64 into a TArray to get it working. I've already made the change on my end and tested it in 4.26.

UnrealImageCapture in UE-5.3

These are the steps I followed to get a minimal version of this pipeline working.

I rebuilt the Plugin for UE-5.3, but could not get the Plugin to work properly (There were errors in the CameraCaptureManager_BP and I could not place it in the Editors).

Similarly to #35, I ended up porting the source files into a project (MyProject) and building everything in that project. Here are the notes I took along the way:

• Moved from UnrealImageCapture/Plugin_UE5, the files CameraCaptureManager.cpp, CameraCaptureManager.h to MyProject/Source/MyProject/.
• Solve the linking issues by modifying MyProject.Build.cs

public class MyProject : ModuleRules
{
	public MyProject(ReadOnlyTargetRules Target) : base(Target)
	{
		PCHUsage = PCHUsageMode.UseExplicitOrSharedPCHs;
	
		PublicDependencyModuleNames.AddRange(new string[] { 
			"Core",
			"InputCore", 
			"ImageWrapper", 
			"RenderCore", 
			"Renderer", 
			"RHI",
			"CoreUObject",
			"Engine",
			"Slate",
			"SlateCore",
		});

}

• You should be able to compile the C++ code now using VSCode's build task.
• Check that things are working by looking for CameraCaptureManager in the Editor's content manager under C++ Classes.
• Drag and drop CameraCaptureManager into the viewport.
• Place a SceneCapture2D actor (from the Place Actor Panel)
• Set the CaptureComponent of CameraCaptureManager to point to the SceneCapture2D camera.
• Open the Level's Blueprint and connect a Capture Non Blocking function to a EnhancedInputComponent that triggers when the desired key-binding is pressed.
• Play the map, and press the selected capture key-binding.

Corrupted segmentation masks in UE5.2 first person map

Hey there,

pardon me for keep opening issues on probably trivial problems to solve, but this one is really defying my debugging abilities.

The problem is: if I use your CameraCaptureToDisk example (without plugin, I can't find a way to compile it), I can capture RGB images and semantic segmentation masks without any problem.

Conversely, if I create a new project (always without plugin) using the first person template and map, and I try to repeat all the steps:

1. copy CameraCaptureManager.h/.cpp from the plugin to Prjname/Source/Prjname,
2. add the dependencies to Prjname.Build.cs to avoid linking errors,
3. copy CameraCaptureManager_BP.uasset to Content/CameraCaptureToDisk/Blueprints and PP_Segmentation.uasset to Content,
4. build+cook+run
5. open the project and add the two CameraCaptureManager objects and the two SceneCapture2D objects (children of a CameraActor object for convenience),
6. have the two CameraCaptureManagers use the respective SceneCapture2D objects and, only for that concerning the segmentation masks, tick the PNG flag and add PP_Segmentation as postprocess material,
7. edit the level blueprint and make the capture pipeline activate by pressing a key on the keyboard,

I can still capture RGB images but I capture corrupt semantic segmentation masks.

I've tried every possible fix/workaround that came to my mind, the first of which was obviously disabling the alpha channel fix that I proposed in the pull request (it seems to be irrelevant in this sense, it works fine on both CameraCaptureToDisk and my project). What am I doing wrong? Sounds like an alignment issue. The image seems to be there but it's badly aligned, do I see correctly?

Passing and saving a UTextureRenderTarget2D object.

Hey i thought it would be better to make a new issue for people to better find this than my question being embedded into an unrelated issue topic.

I wanted to ask though @TimmHess. Would it require a lot of rewriting and editing to make it save a UTextureRenderTarget2D object instead of saving what is currently shown via the capture2d camera? Long story short i'm wanting to send my Composure renders over the network but my first goal is to at least get it to save to a jpeg without issue first.
Composure has a lovely virtual override function called "RelayOutput_Implementation" where i can get the UTextureRenderTarget2D object. I tried to simply slot in the object into your code hoping it would work but i'm still newish to C++ and UE and have been learning it through trial by fire at this point but when it comes to sending requests to the GPU, that's when i get a bit out of my depth.

added input for a UTextureRenderTarget2D and tried to use that for the renderTargetResource variable

But on further inspection i think i was going about this all wrong as the UTextureRenderTarget2D already has an image and i'm not needing to request an image from the GPU as that's all already handled by Composure. Below is an error i get with my current code, as well as the line where it breaks. But again i assume this is because i'm not requesting the image from the gpu like your current setup does.

I can see that when you use GetData on line 118 you have nextRenderRequest->Image.GetData() which returns a TArray which i assume is the data type that's needed for this to work. Can you see a way for this to work if i already have a UTextureRenderTarget2D object? The ReadPixels function is so ungodly slow and assume the Image.GetData() is a much faster process in comparison.

The Sample project keep crashing in UE5

Hi,
I downloaded your project, compiled it and tried to run it with UE5.
But when I hit play, and press C the whole UE5 crush with the following errors:

Unhandled Exception: EXCEPTION_ACCESS_VIOLATION reading address 0x0000009800000052

ucrtbase
ucrtbase
ucrtbase
UnrealEditor_Core
UnrealEditor_ImageWrapper
UnrealEditor_ImageWrapper
UnrealEditor_CameraCaptureToDisk!ACameraCaptureManager::Tick() [D:\code\UnrealImageCapture\CaptureToDisk\Plugins\CameraCaptureToDisk\Source\CameraCaptureToDisk\Private\CameraCaptureManager.cpp:72]
UnrealEditor_Engine
UnrealEditor_Engine
UnrealEditor_Engine
UnrealEditor_Engine
UnrealEditor_Core
UnrealEditor_Core
UnrealEditor_Core
UnrealEditor_Engine
UnrealEditor_Engine
UnrealEditor_Engine
UnrealEditor_Engine
UnrealEditor_UnrealEd
UnrealEditor_UnrealEd
UnrealEditor
UnrealEditor
UnrealEditor
UnrealEditor
UnrealEditor
UnrealEditor
kernel32
ntdll

Do you know why?

compile error in new project

Appreciated this tutorial a lot, this helped me very much and really saved my time. Thanks! :) By the way, I'm an AI researcher fresh new to ue4.

I had a few problems when I reproduce the code in a new project step by step, I can barely compile the code successfully. I've tried this code in both ue4.25 with vs2019 and ue4.22 with vs2017, neither of them can successfully reproduce this repo.
I'll list some of the issues I met.

1. The struct FRenderRequest added at the very begin, I had to modify the code from FRenderCommandFence RenderFence to FRenderCommandFence* RenderFence, and then it can be compiled.
   But when I use FRenderCommandFence* RenderFence, it will cause other errors, for instance in line 184 code renderRequest->RenderFence->BeginFence(); will output error:

2>     Creating library D:\Users\56485\Documents\Unreal Projects\VelocityDemo\Intermediate\Build\Win64\UE4Editor\Development\VelocityDemo\UE4Editor-VelocityDemo-0056.suppressed.lib and object D:\Users\56485\Documents\Unreal Projects\VelocityDemo\Intermediate\Build\Win64\UE4Editor\Development\VelocityDemo\UE4Editor-VelocityDemo-0056.suppressed.exp
2>CaptureManager.cpp.obj : error LNK2019: unresolved external symbol "__declspec(dllimport) public: void __cdecl FRenderCommandFence::BeginFence(bool)" (__imp_?BeginFence@FRenderCommandFence@@QEAAX_N@Z) referenced in function "public: void __cdecl ACaptureManager::CaptureColorNonBlocking(class ASceneCapture2D *,bool)" (?CaptureColorNonBlocking@ACaptureManager@@QEAAXPEAVASceneCapture2D@@_N@Z)
2>D:\Users\56485\Documents\Unreal Projects\VelocityDemo\Binaries\Win64\UE4Editor-VelocityDemo-0056.dll : fatal error LNK1120: 1 unresolved externals

2. code using ENQUEUE_RENDER_COMMAND and RenderRequestQueue:

    ENQUEUE_RENDER_COMMAND(SceneDrawCompletion)(
    [readSurfaceContext](FRHICommandListImmediate& RHICmdList){
        RHICmdList.ReadSurfaceData(
            readSurfaceContext.SrcRenderTarget->GetRenderTargetTexture(),
            readSurfaceContext.Rect,
            *readSurfaceContext.OutData,
            readSurfaceContext.Flags
        );
    });

    // Notify new task in RenderQueue
    RenderRequestQueue.Enqueue(renderRequest);

    // Set RenderCommandFence
    renderRequest->RenderFence.BeginFence();

It just never worked and the output errors are related to LNK2019 and LNK1120 :

2>     Creating library D:\Users\56485\Documents\Unreal Projects\VelocityDemo\Intermediate\Build\Win64\UE4Editor\Development\VelocityDemo\UE4Editor-VelocityDemo-0036.suppressed.lib and object D:\Users\56485\Documents\Unreal Projects\VelocityDemo\Intermediate\Build\Win64\UE4Editor\Development\VelocityDemo\UE4Editor-VelocityDemo-0036.suppressed.exp
2>CaptureManager.cpp.obj : error LNK2019: unresolved external symbol "__declspec(dllimport) public: void __cdecl FRHICommandListImmediate::ReadSurfaceData(class FRHITexture *,struct FIntRect,class TArray<struct FColor,class TSizedDefaultAllocator<32> > &,class FReadSurfaceDataFlags)" (__imp_?ReadSurfaceData@FRHICommandListImmediate@@QEAAXPEAVFRHITexture@@UFIntRect@@AEAV?$TArray@UFColor@@V?$TSizedDefaultAllocator@$0CA@@@@@VFReadSurfaceDataFlags@@@Z) referenced in function "public: void __cdecl ACaptureManager::CaptureColorNonBlocking(class ASceneCapture2D *,bool)" (?CaptureColorNonBlocking@ACaptureManager@@QEAAXPEAVASceneCapture2D@@_N@Z)
2>CaptureManager.cpp.obj : error LNK2019: unresolved external symbol "__declspec(dllimport) public: void __cdecl FRenderCommandFence::BeginFence(bool)" (__imp_?BeginFence@FRenderCommandFence@@QEAAX_N@Z) referenced in function "public: void __cdecl ACaptureManager::CaptureColorNonBlocking(class ASceneCapture2D *,bool)" (?CaptureColorNonBlocking@ACaptureManager@@QEAAXPEAVASceneCapture2D@@_N@Z)
2>CaptureManager.cpp.obj : error LNK2019: unresolved external symbol "__declspec(dllimport) public: static enum ENamedThreads::Type __cdecl FRenderCommand::GetDesiredThread(void)" (__imp_?GetDesiredThread@FRenderCommand@@SA?AW4Type@ENamedThreads@@XZ) referenced in function "private: class TRefCountPtr<class FGraphEvent> __cdecl TGraphTask<class TEnqueueUniqueRenderCommandType<struct `public: void __cdecl ACaptureManager::CaptureColorNonBlocking(class ASceneCapture2D *,bool)'::`2'::SceneDrawCompletionName,class <lambda_6990b626c546b10a41a4abd053c55940> > >::Setup(class TArray<class TRefCountPtr<class FGraphEvent>,class TInlineAllocator<4,class TSizedDefaultAllocator<32> > > const *,enum ENamedThreads::Type)" (?Setup@?$TGraphTask@V?$TEnqueueUniqueRenderCommandType@USceneDrawCompletionName@?1??CaptureColorNonBlocking@ACaptureManager@@QEAAXPEAVASceneCapture2D@@_N@Z@V<lambda_6990b626c546b10a41a4abd053c55940>@@@@@@AEAA?AV?$TRefCountPtr@VFGraphEvent@@@@PEBV?$TArray@V?$TRefCountPtr@VFGraphEvent@@@@V?$TInlineAllocator@$03V?$TSizedDefaultAllocator@$0CA@@@@@@@W4Type@ENamedThreads@@@Z)
2>CaptureManager.cpp.obj : error LNK2019: unresolved external symbol "__declspec(dllimport) public: static enum ESubsequentsMode::Type __cdecl FRenderCommand::GetSubsequentsMode(void)" (__imp_?GetSubsequentsMode@FRenderCommand@@SA?AW4Type@ESubsequentsMode@@XZ) referenced in function "public: static class TGraphTask<class TEnqueueUniqueRenderCommandType<struct `public: void __cdecl ACaptureManager::CaptureColorNonBlocking(class ASceneCapture2D *,bool)'::`2'::SceneDrawCompletionName,class <lambda_6990b626c546b10a41a4abd053c55940> > >::FConstructor __cdecl TGraphTask<class TEnqueueUniqueRenderCommandType<struct `public: void __cdecl ACaptureManager::CaptureColorNonBlocking(class ASceneCapture2D *,bool)'::`2'::SceneDrawCompletionName,class <lambda_6990b626c546b10a41a4abd053c55940> > >::CreateTask(class TArray<class TRefCountPtr<class FGraphEvent>,class TInlineAllocator<4,class TSizedDefaultAllocator<32> > > const *,enum ENamedThreads::Type)" (?CreateTask@?$TGraphTask@V?$TEnqueueUniqueRenderCommandType@USceneDrawCompletionName@?1??CaptureColorNonBlocking@ACaptureManager@@QEAAXPEAVASceneCapture2D@@_N@Z@V<lambda_6990b626c546b10a41a4abd053c55940>@@@@@@SA?AVFConstructor@1@PEBV?$TArray@V?$TRefCountPtr@VFGraphEvent@@@@V?$TInlineAllocator@$03V?$TSizedDefaultAllocator@$0CA@@@@@@@W4Type@ENamedThreads@@@Z)
2>CaptureManager.cpp.obj : error LNK2019: unresolved external symbol "__declspec(dllimport) class FRHICommandListImmediate & __cdecl GetImmediateCommandList_ForRenderCommand(void)" (__imp_?GetImmediateCommandList_ForRenderCommand@@YAAEAVFRHICommandListImmediate@@XZ) referenced in function "public: void __cdecl ACaptureManager::CaptureColorNonBlocking(class ASceneCapture2D *,bool)" (?CaptureColorNonBlocking@ACaptureManager@@QEAAXPEAVASceneCapture2D@@_N@Z)
2>CaptureManager.cpp.obj : error LNK2019: unresolved external symbol "__declspec(dllimport) bool GIsThreadedRendering" (__imp_?GIsThreadedRendering@@3_NA) referenced in function "public: void __cdecl ACaptureManager::CaptureColorNonBlocking(class ASceneCapture2D *,bool)" (?CaptureColorNonBlocking@ACaptureManager@@QEAAXPEAVASceneCapture2D@@_N@Z)
2>CaptureManager.cpp.obj : error LNK2019: unresolved external symbol "__declspec(dllimport) class TAtomic<bool> GMainThreadBlockedOnRenderThread" (__imp_?GMainThreadBlockedOnRenderThread@@3V?$TAtomic@_N@@A) referenced in function "public: void __cdecl ACaptureManager::CaptureColorNonBlocking(class ASceneCapture2D *,bool)" (?CaptureColorNonBlocking@ACaptureManager@@QEAAXPEAVASceneCapture2D@@_N@Z)
2>CaptureManager.cpp.obj : error LNK2019: unresolved external symbol "__declspec(dllimport) struct Trace::FChannel RenderCommandsChannel" (__imp_?RenderCommandsChannel@@3UFChannel@Trace@@A) referenced in function "public: void __cdecl TEnqueueUniqueRenderCommandType<struct `public: void __cdecl ACaptureManager::CaptureColorNonBlocking(class ASceneCapture2D *,bool)'::`2'::SceneDrawCompletionName,class <lambda_6990b626c546b10a41a4abd053c55940> >::DoTask(enum ENamedThreads::Type,class TRefCountPtr<class FGraphEvent> const &)" (?DoTask@?$TEnqueueUniqueRenderCommandType@USceneDrawCompletionName@?1??CaptureColorNonBlocking@ACaptureManager@@QEAAXPEAVASceneCapture2D@@_N@Z@V<lambda_6990b626c546b10a41a4abd053c55940>@@@@QEAAXW4Type@ENamedThreads@@AEBV?$TRefCountPtr@VFGraphEvent@@@@@Z)
2>D:\Users\56485\Documents\Unreal Projects\VelocityDemo\Binaries\Win64\UE4Editor-VelocityDemo-0036.dll : fatal error LNK1120: 8 unresolved externals

I can hardly find a proper way to righten this.

---

And when I compile the project that clones from this repo, no errors would appear. This really confused me. Looking forward to a reply :)

weird camera exposure & color

Hi @TimmHess, thanks for this repo. I am having similar issues. Would you be able to help me with it? I tried what you suggested above (the GEngine->GetDisplayGamma() part) but they change very little. Right now, I have:

Editor:

Saved Image:

I am using jpegs to save by the way - I changed the ERGB format on CameraCaptureManager.cpp Tick() to RGBA. Also changed the pixel format on SetupCameraComponent to PF_R8G8B8A8 to match it.

Thanks a lot for your help! (Also commented on #8, not sure whether you want a new issue or just the old one.)

Hair disapper when RenderOffScreen on linux platform

I run this plugin normally on windows.
But on linux server, when render offscreen, the metahuman's groom hair disappered in some frame, so the hair blinks in the final video.
Have you tried this code RenderOffScreen on linux server？Weird problems.
Any hint will be appreciated~