Scene Understanding about microsoft-openxr-unreal

[noon974]

Hi there, looks like 4.27.1 has just released !
What can we expect from Microsoft Open XR Scene Understanding exactly ?

[dgough]

I'm currently working on adding Scene Understanding support to the Unreal plugin using the XR_MSFT_scene_understanding OpenXR extension.

[shukenmg]

I'm currently working on adding Scene Understanding support to the Unreal plugin using the XR_MSFT_scene_understanding OpenXR extension.

Very cool, thanks for letting me know.

[shukenmg]

Any ETA on this?
Thnaks.

[sw5813]

Hi @shukenmg, we're waiting on an engine fix we need to come out in 4.27.1- after that, SU will be ready to go.

[shukenmg]

Ok, looking forward to it.

[fieldsJacksonG]

#28

Documentation is upcoming, but you can take a look at the SpatialMappingActor for how SU works.
Use SceneUnderstanding in ToggleARCapture to use, and GetObjectClassification on the ARTrackedGeometry to identify what type of mesh it is.

There are also a few ini entries you can set to control the behavior of scene understanding:

1. ShouldDoSceneUnderstandingMeshDetection will add meshes that overlap planes. Otherwise SU will only map planes. (This will be added as a new checkbox in the ARSessionConfig in a future engine release)
2. SceneUnderstandingVolumeHeight is an optional entry to control the height of the scan volume. Otherwise the scan will be a sphere sized by the SpatialMeshingVolumeSize in the HoloLens project settings.

[shukenmg]

Nice, this works well, thanks for all the work!
:)

[shukenmg]

Does this mean that full scene understanding can now be supported via remoting?
Thanks!

[shukenmg]

Great! Thanks so much @fieldsJacksonG !

[fieldsJacksonG]

Version 1.1.12 is going into the marketplace now with this remoting version.

[shukenmg]

@fieldsJacksonG Hi!

When I enable generate collision like this

UE5 crashes with this:

Assertion failed: CurrentCookHelper == nullptr [File:D:\build\++UE5\Sync\Engine\Source\Runtime\Engine\Private\PhysicsEngine\BodySetup.cpp] [Line: 619]

UnrealEditor_Engine!UBodySetup::CreatePhysicsMeshesAsync() [D:\build\++UE5\Sync\Engine\Source\Runtime\Engine\Private\PhysicsEngine\BodySetup.cpp:619]
UnrealEditor_MRMesh!UMRMeshComponent::SendBrickData_Internal() [D:\build\++UE5\Sync\Engine\Source\Runtime\MRMesh\Private\MRMeshComponent.cpp:768]
UnrealEditor_MRMesh!UMRMeshComponent::UpdateMesh() [D:\build\++UE5\Sync\Engine\Source\Runtime\MRMesh\Private\MRMeshComponent.cpp:1101]
UnrealEditor_AugmentedReality!UARPlaneComponent::UpdateVisualization_Implementation() [D:\build\++UE5\Sync\Engine\Source\Runtime\AugmentedReality\Private\ARComponent.cpp:418]
UnrealEditor_CoreUObject!UFunction::Invoke() [D:\build\++UE5\Sync\Engine\Source\Runtime\CoreUObject\Private\UObject\Class.cpp:5912]
UnrealEditor_CoreUObject!UObject::ProcessEvent() [D:\build\++UE5\Sync\Engine\Source\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:2003]
UnrealEditor_AugmentedReality!UARPlaneComponent::OnRep_Payload() [D:\build\++UE5\Sync\Engine\Source\Runtime\AugmentedReality\Private\ARComponent.cpp:175]
UnrealEditor_AugmentedReality!UARPlaneComponent::execServerUpdatePayload() [D:\build\++UE5\Sync\Engine\Intermediate\Build\Win64\UnrealEditor\Inc\AugmentedReality\ARComponent.gen.cpp:1606]
UnrealEditor_CoreUObject!UFunction::Invoke() [D:\build\++UE5\Sync\Engine\Source\Runtime\CoreUObject\Private\UObject\Class.cpp:5912]
UnrealEditor_CoreUObject!UObject::ProcessEvent() [D:\build\++UE5\Sync\Engine\Source\Runtime\CoreUObject\Private\UObject\ScriptCore.cpp:2003]
UnrealEditor_AugmentedReality!UARPlaneComponent::ServerUpdatePayload() [D:\build\++UE5\Sync\Engine\Intermediate\Build\Win64\UnrealEditor\Inc\AugmentedReality\ARComponent.gen.cpp:1651]
UnrealEditor_AugmentedReality!UARPlaneComponent::Update() [D:\build\++UE5\Sync\Engine\Source\Runtime\AugmentedReality\Private\ARComponent.cpp:175]
UnrealEditor_OpenXRAR!FOpenXRARSystem::OnObjectUpdated_GameThread() [D:\build\++UE5\Sync\Engine\Plugins\Runtime\OpenXR\Source\OpenXRAR\Private\OpenXRAR.cpp:744]
UnrealEditor_OpenXRAR!TBaseSPMethodDelegateInstance<0,FOpenXRARSystem,1,void __cdecl(void),FDefaultDelegateUserPolicy,TSharedPtr<FOpenXRARTrackedGeometryData,1> >::ExecuteIfSafe() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Public\Delegates\DelegateInstancesImpl.h:307]
UnrealEditor_OpenXRAR!TGraphTask<FSimpleDelegateGraphTask>::ExecuteTask() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Public\Async\TaskGraphInterfaces.h:975]
UnrealEditor_Core!FNamedTaskThread::ProcessTasksNamedThread() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\Async\TaskGraph.cpp:753]
UnrealEditor_Core!FNamedTaskThread::ProcessTasksUntilQuit() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\Async\TaskGraph.cpp:642]
UnrealEditor_Core!FTaskGraphCompatibilityImplementation::WaitUntilTasksComplete() [D:\build\++UE5\Sync\Engine\Source\Runtime\Core\Private\Async\TaskGraph.cpp:2169]
UnrealEditor_Engine!FTickTaskSequencer::ReleaseTickGroup() [D:\build\++UE5\Sync\Engine\Source\Runtime\Engine\Private\TickTaskManager.cpp:565]
UnrealEditor_Engine!FTickTaskManager::RunTickGroup() [D:\build\++UE5\Sync\Engine\Source\Runtime\Engine\Private\TickTaskManager.cpp:1592]
UnrealEditor_Engine!UWorld::RunTickGroup() [D:\build\++UE5\Sync\Engine\Source\Runtime\Engine\Private\LevelTick.cpp:790]
UnrealEditor_Engine!UWorld::Tick() [D:\build\++UE5\Sync\Engine\Source\Runtime\Engine\Private\LevelTick.cpp:1531]
UnrealEditor_UnrealEd!UEditorEngine::Tick() [D:\build\++UE5\Sync\Engine\Source\Editor\UnrealEd\Private\EditorEngine.cpp:1777]
UnrealEditor_UnrealEd!UUnrealEdEngine::Tick() [D:\build\++UE5\Sync\Engine\Source\Editor\UnrealEd\Private\UnrealEdEngine.cpp:474]
UnrealEditor!FEngineLoop::Tick() [D:\build\++UE5\Sync\Engine\Source\Runtime\Launch\Private\LaunchEngineLoop.cpp:5215]
UnrealEditor!GuardedMain() [D:\build\++UE5\Sync\Engine\Source\Runtime\Launch\Private\Launch.cpp:183]
UnrealEditor!GuardedMainWrapper() [D:\build\++UE5\Sync\Engine\Source\Runtime\Launch\Private\Windows\LaunchWindows.cpp:147]
UnrealEditor!LaunchWindowsStartup() [D:\build\++UE5\Sync\Engine\Source\Runtime\Launch\Private\Windows\LaunchWindows.cpp:283]

How can I fix this? Thanks!

[shukenmg]

@fieldsJacksonG
I found a issue with SU, it doesn't return mesh data, only planes.
In the docs it mentions that it can return a watertight mesh and also object meshes.

[shukenmg]

@fieldsJacksonG
It would also be cool if it were possible to actually tag/classify the Unknown objects as BACKGROUND/INFERRED/UNKNOWN
Maybe this is waiting on an engine change?

Thanks!