microsoft / hololens2forcv

Sample code and documentation for using the Microsoft HoloLens 2 for Computer Vision research.

page_type	name	description	languages	products
sample	HoloLens2ForCV samples	HoloLens 2 Research Mode samples showcasing raw streams on device, including depth camera, gray-scale cameras, and IMU.	cpp	windows-mixed-reality hololens

HoloLens2ForCV samples

HoloLens 2 Research Mode enables access to the raw streams on device (depth camera, gray-scale cameras, IMU).

By releasing the Research Mode API together with a set of tools and sample apps, we want to make it easier to use HoloLens 2 as a powerful Computer Vision and Robotics research device.

The project was launched at ECCV 2020. We plan to extend it over time and welcome contributions from the research community.

Contents

This repository contains documentation and samples for HoloLens 2 Research Mode.

Take a look at the API documentation to get familiar with the new HoloLens 2 Research Mode API.

Take a look at the slides of the ECCV tutorial to obtain detailed information about Research Mode and its main features, read about how to set up your device, and learn how to use the apps in this repository.

Take a look at this technical report for an introduction to Research Mode.

The repository contains four sample apps:

• The CalibrationVisualization app shows a visualization of depth and gray-scale cameras coordinate frames live on device.
• The CameraWithCVAndCalibration app shows how to process Research Mode streams live on device: it uses OpenCV to detect arUco markers in the two frontal gray-scale cameras and triangulate the detections.
• The SensorVisualization app shows how to visualize Research Mode streams live on device.
• The StreamRecorder app shows how to capture simultaneously Research Mode streams (depth and gray-scale cameras) plus head, hand and eye tracking, save the streams to disk on device. It also contains a set of python scripts for offline postprocessing.

The StreamRecorder app uses Cannon, a collection of wrappers and utility code for building native mixed reality apps using C++, Direct3D and Windows Perception APIs. It can be used as-is outside Research Mode for fast and easy native development.

Setup

The earliest build that fully supports research mode is 19041.1364. Please join the Windows Insider Program to get preview builds. After that, in the device portal, enable research mode, different than recording mode. See https://github.com/microsoft/HoloLens2ForCV/blob/main/Docs/ECCV2020-Tutorial/ECCV2020-ResearchMode-Api.pdf (slides 6, 7 and 8) or Setup section in https://github.com/microsoft/HoloLens2ForCV/blob/main/Docs/ResearchMode-ApiDoc.pdf. Finally only arm64 applications are supported for now.

Citing our work

If you find HoloLens 2 Research Mode useful for your research, please cite our work:

@article{hl2_rm,
         title     = {{HoloLens 2 Research Mode as a Tool for Computer Vision Research}},
         author    = {Dorin Ungureanu and Federica Bogo and Silvano Galliani and Pooja Sama and Xin Duan and Casey Meekhof and Jan St\{"u}hmer and Thomas J. Cashman and Bugra Tekin and Johannes L. Sch\{"o}nberger and Bugra Tekin and Pawel Olszta and Marc Pollefeys},
         journal   = {arXiv:2008.11239},
         year      = {2020}
}

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

hololens2forcv's Issues

Trying to rung CameraWithCVAndCalibration app

I am trying to run CameraWithCVAndCalibration sample app. But I am having trouble building the app.

Error LNK1107 invalid or corrupt file: cannot read at 0x86 CameraWithCVAndCalibration C:\Users\xxx\Desktop\HoloLens2ForCV-main\Samples\CameraWithCVAndCalibration\OpenCvInstallArm64-412d\x64\vc15\staticlib\opencv_world412d.lib 1

Could someone please elaborate on how to install opencv for this project? Thanks

CameraWithCVAndCalibration Project release issues

Samples \ CameraWithCVAndCalibration \ OpenCvInstallArm64-412 - d contains arm64 headers and Opencv library distribution.These require Git LFS. What does that mean。
I didn't follow the above method because I didn't know what to do, so I posted the following error，Can someone help me out

The accuracy of the depth camera

Hi!
As for my project, I want to acquiring point cloud from depth camera for object 6DOF estimation. I find there is not much docs about the accuracy of depth camera. I use the code from
https://github.com/petergu684/HoloLens2-ResearchMode-Unity
then, visualize the point cloud in HoloLens.
As for plane like the desk, the point cloud could align the plane well. when it comes to some stuff with complex surface, the misalignment is about 2cm under the surface.
The paper below indicates the misalignment could be related to the Multi Path Effect of ToF sensor.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7070293/
Is there any way could solve this？

Any reply will be appreciated.

StreamRecorder release failure

When I use VS compiler release StreamRecorder project VS2019 hint I: MSB3073 StreamRecorder \ StreamRecorderApp \ packages \ Microsoft Windows. CppWinRT. 2.0.200729.8 \ build \ native ....\bin" cppwinRT @"ARM64\Release\cppwinrt_ref.rsp" "retired with code 1.
. In Microsoft Windows. CppWinRT. The targets file of 544 rows

This has caused my project to fail to launch. Can anyone help me?thank you

Recording mode in Hololens 2

I have been trying to activate the research mode and run the samples, However the windows developer portal only displays Recording mode and the applications fail on deployments with "Access Denied" . Can you help what the issue is?

How to get aligned depth?

Hello, I would like to consult you some question about how to get aligned depth. Now I want to capture RGB-D data through HoloLens 2, the sensor stream used is PV+Depth Long Throw or VLC LF+Depth Long Throw, since I don't know the one-to-one mapping relationship between the pixels in depth image and the pixels in RGB or Gray image, I can only use the sensor poses to find some sparse correspondences through backprojection and reprojection. So I want ask how can I directly get the dense aligned depth image? Or will you plan to open the mapping table of pixels between the depth image and rgb(or gray) images?

Access research mode from a c# script

Hello, we have the SensorVisualization sample working and would like to use the IR and depth camera in our C# (unity) application. The Media Foundation contains code that seems to created for this purpose, with functions like: “MediaFrameSourceKind.Depth” there are also people mention that its possible: (https://stackoverflow.com/questions/59948317/mediafoundation-gives-four-cameras-into-one-single-frame)

Somehow we can only see the locatable camera, not the depth/IR camera using this method. Do you know if it is possible to acces the other camera's of the hololens2 in this way? Any help would be appreciated!

Bug in save_pclouds.py (L162-L166)

There are wrong indices to generate the paths in rgb.txt/depth.txt.

rgb_parts = Path(rgb_proj_path).parts[2:]
rgb_tmp = Path(rgb_parts[1]) / Path(rgb_parts[2])
depth_parts = Path(depth_proj_path).parts[2:]
depth_tmp = Path(depth_parts[1]) / Path(depth_parts[2])

This should be changed to:

# Create rgb and depth paths
rgb_parts = Path(rgb_proj_path).parts[2:]
rgb_tmp = Path(rgb_parts[-2]) / Path(rgb_parts[-1])
depth_parts = Path(depth_proj_path).parts[2:]
depth_tmp = Path(depth_parts[-2]) / Path(depth_parts[-1])

Then, the paths are correct. Otherwise, tsdf-integration.py does not find the files and reconstruction fails.

How to use the differents scripts

Hello,

I'm novice and i'm trying to use the differents scripts.

I manage to use recorder_console to download et process files but I don't understand how to use the other scripts.

Does anyone have a procedure for using others?

thanks in advance

Problem building and deploying the different solution files

Hi,
I have some errors building and deploying the different solution files. One error is instructing to reinstall Windows, but I first want to verify whether you have other suggestions? Thank you in advance for your advice.
Best regards,
kv

CameraWithCVAndCalibration.sln

Following error occurred during building:

This error refers to the following error code: LNK1107
https://docs.microsoft.com/en-us/cpp/error-messages/tool-errors/linker-tools-error-lnk1107?f1url=%3FappId%3DDev16IDEF1%26l%3DEN-US%26k%3Dk(LNK1107)%26rd%3Dtrue&view=vs-2019
I thought it might be related to the use of OpenCV.

SensorVisualization.sln
Building succeeded

During Deploy the following error message occurred:

‘Unable to activate Windows Store app ‘…SensorVisualization’.
A Microsoft Windows component is not correctly registered. If the problem persists, try repairing your Windows installation, or reinstalling Windows.

StreamRecorder.sln
Building succeeded
During Deploy the same error as for SensorVisualization.sln occurred.

StreamRecorder cannot find Cannon files

Hello,
i was able to deploy the StreamRecorderApp two days ago without any problems but now when i try to deploy it again none of the Cannon files can be found in the solution tree.

has anybody experience this before? how can i solve this issue?

Question about IMU Timestamp

Hello, I would like to consult you some question about IMU timestamp. Refer to the acquisition method given in the API document, I have successfully accessed the IMU data，but I found a question about timestamp. For IMU sensor，there are two kind of timestamps, which are in HostTicks (hundred of nanoseconds) and SensorTicks (nanaseconds). According to the collected data, the SensorTicks in nanoseconds grows at equal intervals, even between different sensor frames, but HostTicks in hundreds of nanaseconds of all samples in each sensor frame are the same, I want to ask if this is normal? Why it came out like this? And how can I get the accurate timestamp in HostTicks of each sample？

Where is ResearchModeAPI.dll?

When I use LoadLibraryA("ResearchModeAPI") in UE4, and add UE4 dll search path "C:/Windows/System32/" manually, or LoadLibrary("C:/Windows/System32/ResearchModeAPI.dll"), it's failed!

So, what's the location of "ResearchModeAPI.dll", and how can I load this dll in UE4?

Help me please!

Colored point cloud offset / mismatch

Hi!
I made some recordings of PV and DepthAHAT streams with the StreamRecorder and processed them with process_all.py to obtain colored point clouds.
In those point clouds, there is a quite large mismatch (I would guess around 2 cm) between depth and rgb information, as can be seen from this example:

With the HoloLens1 using my own code, the matching is much better:

Is this behavior expected, or is something wrong with my recordings? Is there anything I can do (except for adding a fixed offset)
to obtain a better matching between DepthAHAT and PV?

Thanks!

Possibility to reduce framerate?

Hello,

I investigate on the pre-heating-time of the HoloLens2, so I need to run the StreamRecorderApp for about 90 minutes, only activating the Long Throw Depth Sensor. Unfortunately I noticed that the App runs very instable when using it for 10 or 15 minutes. After hitting the "Stop"-Button, the App takes several minutes to reload and it does not always give reliable results. Sometimes the app just crashes down without saving the recordings.

I suspect that this happens because of the high framerate (5fps in release-mode), so the app collects a big amount of data in that period of time.. Is there any possibility to reduce the framerate to make longer investigations possible (eg. 1 fps or still fewer)?

Interpreting magnetometer data

Hello!

I'm reading the magnetometer values from the Hololens 2, but I can't figure out what they are supposed to mean or if my readings are erroneous.
The documentation doesn't state any unit of measurement or meaning for these values.

When I tested it, the values where [464, -198, 483] and after turning 180 degrees they were [456, -223, 463].
What do these values mean an how are they to be processed.

Thank you!

Best regards,
Max

Is there any plan to support machine learning on holoLens2

Hello,

Do you have any existing tries or plan to support machine learning library using its GPU in HoloLens2 ?

Recent days, I have tried to run onnx model in HoloLens2. It was successful to build and run on Its CPU (ARM architecture) using onnxruntime(or WinML) library . But to use its GPU, these library require additional GPU provider such as CUDA, DirectML, OpenCL.

As you know, The cpu of Hololens2 is 'Snapdragon 850 (based on ARM)' and GPU is 'Qualcomm Adreano 630'.

And according to follow documentation, The GPU supports to use OpenCL 2.0, and DX12 and so on. (see : https://www.qualcomm.com/products/snapdragon-850-mobile-compute-platform)

So I have been looking for a way to use these API in ML libraries. In onnxruntime and WinML , it is possible to use 'DirectML' provider, which use DX12. But unfortunately, it was not support to ARM but only x64, x86. (So it seems possible to run only hololens 1)

So I want to know is there any idea or plan to run any ML library on HoloLen2 using Its GPU.

I always thanks your contributions.

Exception thrown using StreamRecorder

I am trying to run the StreamRecorder and i get an exception when it is asking for consent (Exception thrown at 0x00E1B63C in StreamRecorder.exe: 0xC0000005: Access violation reading location 0x00000000). This happens at line 92 in SensorScenario.cpp. OS build of hololens is 10.0.19041.1373. Do you know why I am getting this error and how to resolve it?

Package the project into DLL libraries

How to put the project into Unity after packaging the DLL library。
Who can tell me the complete process of packaging a project into a DLL library and how to call this DLL inside Unity?
I used the call method shown in the figure below but got an error in Unity.Who can help me?

Trouble getting AHAT Depth Frame in SensorVisualization

In SensorVisualizationScenario.cpp, removing the line: #define DEPTH_USE_LONG_THROW should make the sample work for DEPTH_AHAT.
When I do that, the line: hr = pSensorFrame->QueryInterface(IID_PPV_ARGS(&pDepthFrame)); in SlateCameraRenderer.cpp returns depthFrame as NULL. Am I missing something?

In SensorVisualizationScenario::~SensorVisualizationScenario(), m_pAHATSensor is never Released. I believe this is a typo as m_pLTSensor is repeated.

Thanks.

Exception "hresult_error" when debugging

I have successfully built the sample solution. But got "hresult_error" when debugging it on Hololens.

Exception thrown at 0x00007FFDB5CC39EC (KernelBase.dll) in CalibrationVisualization.exe: WinRT originate error - 0x8000FFFF : 'Catastrophic failure'.
Exception thrown at 0x00007FF6833EF284 in CalibrationVisualization.exe: 0xC0000005: Access violation reading location 0x0000000000000000.

The first exception occurred at "winrt\base.h"
throw hresult_error(result, take_ownership_from_abi);

The second exception occurred at "Windows.UI.Core.h"
template <typename D> WINRT_IMPL_AUTO(void) consume_Windows_UI_Core_ICoreWindow<D>::KeyDown(winrt::event_token const& cookie) const noexcept { WINRT_VERIFY_(0, WINRT_IMPL_SHIM(Windows::UI::Core::ICoreWindow)->remove_KeyDown(impl::bind_in(cookie))); }

I don't know anything about winrt and have no idea how to adress this problem.
Would anyone help me with this?

Detection works but the virtual content (coordinate system and ray) offset with some transformation

Awesome!
When I run the CameraWithCVAndCalibration sample, I can see the two coordinate systems and two rays showing the direction of marker from the coordinate systems of the two VLC cameras. Is it an issue or a feature that the rays do not align with the actual physical device? It looks like the origin of the whole virtual scene is offseted with a transformation.

how to get 4 tracking cameras intrinsics and distort imgs buffer

when i use StreamRecorder to record sensors data, i found that the saved imgs of 4 tracking cameras are undistorted.
How to get raw data of 4 tracking cameras and these intrinsics?

Problem when try to build a DLL for the API

Hello!

I want to get the access to the sensor data in Unity. Now what I am doing is trying to build a Universal Windows DLL file. And I got this error:

I am new in this field and I guess I might have made some basic mistakes about this project.

1. Will it work if I use DLL(Universal Windows) to export the ResearchMode APIs Theoretically?
2. If 1) is right, what should I do to creatre a project environment where the APIs can work and clear the error above?
3. If my start point is totally wrong. Is there another possible way to do this?

Best,
whirwind

Question about coordinate transforms

Hi,

I am trying to get a point from the depth camera and view it in my Unity application. I use a background app to get the point and view it in my Unity app. I am doing the following steps but my 3d point is slightly off:

1. I get the (u,v) point from the depth image.
2. I use pCameraSensor->MapImagePointToCameraUnitPlane(uv, xy); to get a xy on the unit plane, followed by multiplying it with the depth of the (u,v) point to get the 3D coordinates.
3. I get my final 3D point in the left eye coordinates by pre-multiplying the 3d coordinates with the extrinsic matrix that I get from pCameraSensor->GetCameraExtrinsicsMatrix(&cameraPose).

I have to translate the point by ~6cms to the right to align it with the center. But even after doing that, the final 3d point is off from the expected position by a height(y) of 0.1 meters when I try to feed it to my Unity app?
Am I doing something wrong with my transforms? OR does Unity apps have a different main camera pose from the rig pose?

Also, If I remove the translation from the extrinsic matrix the resultant point is only off by a 2-3 cm from the real world point.

Thanks!

StreamRecorder Project release failure

When I use VS compiler release StreamRecorder project VS2019 hint I: MSB3073 StreamRecorder \ StreamRecorderApp \ packages \ Microsoft Windows. CppWinRT. 2.0.200729.8 \ build \ native ....\bin" cppwinRT @"ARM64\Release\cppwinrt_ref.rsp" "retired with code 1.
. In Microsoft Windows. CppWinRT. The targets file of 544 rows

This has caused my project to fail to launch. Can anyone help me?thank you

IMU sensors not supported?

Hello!

I'm trying to read the IMU (Acceleration, Gyro, Magnetometer) data on a Hololens two.
For this I extended the StreamRecorder sample to also handle the IMU_ACCEL, IMU_GYRO, and IMU_MAG sensor types.

Getting the sensor via m_pSensorDevice->GetSensor(IMU_MAG, &m_pMAGSensor); returns S_OK and a valid sensor reference;

But when OpenStream() is called on that sensor, then the returned result is HRESULT_FROM_WIN32(ERROR_GEN_FAILURE) : A device attached to the system is not functioning..

Is this an hardware issue?
Is there some setting or initialization I'm missing?
I could not find anything in the ResearchMode-ApiDoc.pdf regarding differences in the initial set-up.

Thanks in advance!

By the way, in ResearchMode-ApiDoc.pdf, page 9 there seems to be a typo:

Each sensor object defines sensor specific interfaces that can by QIed for.

Unable to run Sensor Visualization sample

I tried to run Sensor Visualization sample but failed with the Exception: WinRT originate error - 0x80004002 : 'No such interface supported'.

Windows SDK version: 10.0.19041.0
Hololens OS build: 10.0.19041.1113

Stride, BitsPerPixel, BytesPerPixel of ResearchModeSensorResolution always 0

It looks like the fields (Stride, BitsPerPixel, BytesPerPixel) of ResearchModeSensorResolution are always 0. Width and Height are correct.

Notation mistake of coordinate frame in api document

I was trying to understand the extrinsic between different cameras based on the Figure 1 in api document (page 7, https://github.com/microsoft/HoloLens2ForCV/blob/main/Docs/ResearchMode-ApiDoc.pdf) and noticed a mistake in y-axis of AHAT/Long throw camera. I assume the y-axis should be into the plane (cross) to make it right-handed.

StreamRecorderApp is not working

Hello everybody,

I just wanted to experiment a little bit with the new functionalities but sadly none of them (except CalibrationVisualization) is working currently.
It would be nice if at least the StreamRecorderApp would work.
The Error is:

What can I do to solve this issue?
Everything is so far set up.
System build (HL): 10.0.19041.1375
Windows Insider Preview is set on.
Research Mode is enabled.

Thank you very much in advance.

errors while building in VS2019

Hi,

I just want to test research mode on my HL2 but in VS2019 I got many errors such as:

Severity Code Description Project File Line Suppression State
Error C2664 'void BasicHologram::XYZAxisModel::SetColors(DirectX::XMFLOAT3 &,DirectX::XMFLOAT3 &,DirectX::XMFLOAT3 &)': cannot convert argument 1 from 'DirectX::XMFLOAT3' to 'DirectX::XMFLOAT3 &' CalibrationVisualization C:\Users\robin.popelka\Desktop\HoloLens2ForCV-main\Samples\CalibrationVisualization\CalibrationVisualization\CalibrationVisualizationScenario.cpp 151

I tried to build the project within windows SDK 10.0.17* Version and the newest one 10.0.19* but no success.

Can you help me with this issue?

StreamRecorder Error - "Unable to activate Windows Store app"

Hello and thanks very much for this repository.

I am able to run the CalibrationVisualization sample, however I am running into errors when I try to run the StreamRecorder sample (and similar errors for the other samples).

When I press "Start with debugging" Visual Studio 2019 builds and deploys without error. During runtime, sometimes the following message is displayed in an error window:

Unable to activate Windows Store app
'31fa9d1a-a222-4878-a6fc-77aff92195b5_ph1m9x8skttmg!App'. The
activation request failed with error 'Invalid value for registry'.

See help for advice on troubleshooting the issue

Othertimes, it reaches a breakpoint in thread:

Here is the full debug output from StreamRecorder

'SensorVisualization.exe' (Win32): Loaded 'U:\USERS\DefaultAccount\AppData\Local\DevelopmentFiles\C33DB2CF-841E-466C-8CC0-FB956BC75B07VS.Debug_ARM64.Nick\SensorVisualization.exe'. Symbols loaded.
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\ntdll.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\KernelBase.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\combase.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\ucrtbase.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\rpcrt4.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\kernel32legacy.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\oleaut32.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\msvcp_win.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\d2d1.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\msvcrt.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\d3d11.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\win32u.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\dxgi.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\DWrite.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\forwarders\kernel32.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'P:\WindowsApps\Microsoft.VCLibs.140.00.Debug_14.0.27810.0_arm64__8wekyb3d8bbwe\msvcp140d_app.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'P:\WindowsApps\Microsoft.VCLibs.140.00.Debug_14.0.27810.0_arm64__8wekyb3d8bbwe\vcruntime140d_app.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'U:\USERS\DefaultAccount\AppData\Local\DevelopmentFiles\C33DB2CF-841E-466C-8CC0-FB956BC75B07VS.Debug_ARM64.Nick\ucrtbased.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\gdi32.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\gdi32min.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\MinUser.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\msvcp110_win.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\SHCore.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\bcryptprimitives.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\procthreadexthost.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\kernel.appcore.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\twinapi.appcore.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\sechost.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\WinTypes.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\dcomp.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\CoreUIComponents.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\ntmarta.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\advapi32legacy.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\CoreMessaging.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\bcrypt.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\ws2_32.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\Windows.StateRepositoryCore.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\Windows.UI.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\TextInputFramework.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\WindowManagementAPI.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\InputHost.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\propsys.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\OneCoreUAPCommonProxyStub.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\d2d1debug3.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\WindowsCodecs.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\Windows.Gaming.Input.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\cfgmgr32.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\rometadata.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\WinMetadata\Windows.Graphics.winmd'. Module was built without symbols.
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\MrmCoreR.dll'. 
onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(1) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(2) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(3) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(4) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(5) tid(1b14) 8685C003 'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\Windows.Shell.ServiceHostBuilder.dll'. 
onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(6) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(7) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(8) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(9) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(10) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(11) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(12) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(13) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(14) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(15) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(16) tid(1b14) 8685C003 'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\execmodelproxy.dll'. 
onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(17) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(18) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(19) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(20) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(21) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(22) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(23) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(24) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(25) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(26) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(27) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(28) tid(1b14) 8685C003 'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\rmclient.dll'. 
onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(29) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(30) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(31) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(32) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(33) tid(1b14) 8685C003 onecoreuap\xbox\devices\api\winrt\pnpapiwrapper.cpp(385)\Windows.Gaming.Input.dll!00007FFC01585444: (caller: 00007FFC0158975C) ReturnHr(34) tid(1b14) 8685C003 Exception thrown at 0x00007FFC0B5439EC in SensorVisualization.exe: Microsoft C++ exception: Cn::XH at memory location 0x0000004DBB0FD6B0.
onecoreuap\windows\moderncore\inputv2\inputhost\components\cursor\client\cursorclient.cpp(44)\InputHost.dll!00007FFBFC93D6E4: (caller: 00007FFBFC93CDEC) ReturnHr(1) tid(1804) 87B20809 onecoreuap\windows\moderncore\inputv2\inputhost\components\cursor\client\cursorclient.cpp(76)\InputHost.dll!00007FFBFC93CE0C: (caller: 00007FFC0AE7D37C) ReturnHr(2) tid(1804) 87B20809 'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\UiaManager.dll'. 
Exception thrown at 0x00007FFC0B5439EC in SensorVisualization.exe: Microsoft C++ exception: Cn::XH at memory location 0x0000004DBB0FD670.
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\Windows.UI.Core.TextInput.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\edgegdi.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\Windows.Perception.Stub.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\Windows.Mirage.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\D3D12.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\d3d10warp.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\ResourcePolicyClient.dll'. 
'SensorVisualization.exe' (Win32): Unloaded 'C:\Windows\system32\ResourcePolicyClient.dll'
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\DXCore.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'B:\Windows\System32\DriverStore\FileRepository\qcdx_ffu850.inf_arm64_40919ccee4c23226\qcdx12arm64um850.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\forwarders\advapi32.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\dxilconv.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'B:\Windows\System32\DriverStore\FileRepository\qcdx_ffu850.inf_arm64_40919ccee4c23226\qcdxarm64compiler850.DLL'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\cryptbase.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\D3DSCache.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\userenv.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\profext.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\profapi.dll'. 
'SensorVisualization.exe' (Win32): Unloaded 'B:\Windows\System32\DriverStore\FileRepository\qcdx_ffu850.inf_arm64_40919ccee4c23226\qcdxarm64compiler850.DLL'
'SensorVisualization.exe' (Win32): Unloaded 'C:\Windows\system32\dxilconv.dll'
'SensorVisualization.exe' (Win32): Unloaded 'C:\Windows\system32\forwarders\advapi32.dll'
'SensorVisualization.exe' (Win32): Unloaded 'B:\Windows\System32\DriverStore\FileRepository\qcdx_ffu850.inf_arm64_40919ccee4c23226\qcdx12arm64um850.dll'
'SensorVisualization.exe' (Win32): Loaded 'B:\Windows\System32\DriverStore\FileRepository\qcdx_ffu850.inf_arm64_40919ccee4c23226\qcdx12arm64um850.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\forwarders\advapi32.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\dxilconv.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'B:\Windows\System32\DriverStore\FileRepository\qcdx_ffu850.inf_arm64_40919ccee4c23226\qcdxarm64compiler850.DLL'. 
'SensorVisualization.exe' (Win32): Unloaded 'B:\Windows\System32\DriverStore\FileRepository\qcdx_ffu850.inf_arm64_40919ccee4c23226\qcdxarm64compiler850.DLL'
'SensorVisualization.exe' (Win32): Unloaded 'C:\Windows\system32\dxilconv.dll'
'SensorVisualization.exe' (Win32): Unloaded 'C:\Windows\system32\forwarders\advapi32.dll'
'SensorVisualization.exe' (Win32): Unloaded 'B:\Windows\System32\DriverStore\FileRepository\qcdx_ffu850.inf_arm64_40919ccee4c23226\qcdx12arm64um850.dll'
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\DXGIDebug.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\d3d11_3SDKLayers.dll'. 
'SensorVisualization.exe' (Win32): Unloaded 'C:\Windows\system32\d3d11_3SDKLayers.dll'
'SensorVisualization.exe' (Win32): Loaded 'B:\Windows\System32\DriverStore\FileRepository\qcdx_ffu850.inf_arm64_40919ccee4c23226\qcdx11arm64um850.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\forwarders\advapi32.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\d3d11_3SDKLayers.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'B:\Windows\System32\DriverStore\FileRepository\qcdx_ffu850.inf_arm64_40919ccee4c23226\qcdxarm64compiler850.DLL'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\ResearchModeAPI.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\CalibrationAPI.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\mfplat.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\mfreadwrite.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\mf.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\AnalogHupApi.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\VinylSoc.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\mfcore.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\powrprof.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\crypt32.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\ksuser.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\RTWorkQ.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\umpdc.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\Windows.Web.dll'. 
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\iertutil.dll'. 
Exception thrown at 0x00007FFC0B5439EC (KernelBase.dll) in SensorVisualization.exe: WinRT originate error - 0x80004002 : 'No such interface supported'.
Exception thrown at 0x00007FFC0B5439EC in SensorVisualization.exe: Microsoft C++ exception: winrt::hresult_no_interface at memory location 0x0000004DBB0FD650.
Exception thrown at 0x00007FFC0B5439EC in SensorVisualization.exe: Microsoft C++ exception: [rethrow] at memory location 0x0000000000000000.
onecoreuap\shell\coreapplication\application\lib\coreapplicationview.cpp(371)\twinapi.appcore.dll!00007FFC0640E598: (caller: 00007FFC0BB41064) ReturnHr(1) tid(1804) 80004002 No such interface supported
    CallContext:[\AppRunOrActivateView] 
onecoreuap\shell\coreapplication\application\lib\coreapplicationviewagilecontainer.cpp(474)\twinapi.appcore.dll!00007FFC063FF4C0: (caller: 00007FFC06386500) ReturnHr(2) tid(1924) 80004002 No such interface supported
    CallContext:[\AppRunOrActivateView] 
onecoreuap\shell\coreapplication\application\lib\coreapplicationviewagilecontainer.cpp(347)\twinapi.appcore.dll!00007FFC063FC964: (caller: 00007FFC063870B4) ReturnHr(3) tid(1924) 80004002 No such interface supported
    CallContext:[\AppRunOrActivateView] 
Exception thrown at 0x00007FFC0B5439EC in SensorVisualization.exe: Microsoft C++ exception: Cn::XH at memory location 0x0000004DBB2FF200.
Exception thrown at 0x00007FFC0B5439EC (KernelBase.dll) in SensorVisualization.exe: WinRT originate error - 0x8000000B : 'The operation attempted to access data outside the valid range'.
'SensorVisualization.exe' (Win32): Loaded 'C:\Windows\system32\threadpoolwinrt.dll'. 

My development machine is Windows 10 Version 1909 (Build 18363.1016). The Hololens 2 is build 10.0.19041.1113

Any Ideas for how I should proceed?

Start/stop button doesn't function

Hello, I am unable to interact with the start/stop button. I can see the buttons when I turn on the app, but can't "press" it. Could you help? Thanks.

MapCameraSpaceToImagePoint returning (0, 0) for any input

Hi there! I tried to call MapCameraSpaceToImagePoint to get the inverse mapping of MapImagePointToCameraUnitPlane, but it looks like whatever xy (in unit plane) is given, the resulting uv is (0, 0).

Access Camera Intrinsics

For many OpenCV functions I need to input the Intrinsics of the cameras. However they are not directly accessible as written in the HoloLens2 API paper. Could you please provide some Information on how to access the camera Intrinsics of the HoloLens2 cameras?

Strange depth value with AHAT

Capturing AHAT depth frame using SensorRecording sample app -> converting into PLY -> visualize using Open3D, provides this results below (closer pixel is red). Notice the envelope-like area around the hand. It supposes to be a wall with some distance.

When visualized with the SensorVisualization sample app, it is also confirmed that the background wall in this region is detected to have similar depth as the hand. Is it possible to filter this area out?

IMU Framerate

In which frame rate can the IMU sensor data be retrieved?
As i understand, via polling (with GetNextBuffer()) you can get the first sensor value of an IMU frame (GetCalibratedAccelaration()) or all of them batched (GetCalibratedAccelarationSamples()). however, the IMU frames themselve are limited to a certain rate, which makes this access method not particularly suitable for realtime data processing. I figured out 12fps for the accel and around 24fps for the gyro.
Is there also an event based approach? So that you get values ​​as soon as new ones are available from the sensor.

SensorVisualization Project operation problem

When I run the program, it will only show the application window. What is the situation? I hope someone can answer for me.
I checked the logs in the background and found no.

Can't download StreamRecorder data

Thanks for posting the repo!

I'm able to run the StreamRecorder and I can see Depth Long Throw.tar (among other files) in the Device Portal, but when I try to save them to my PC, the browser times out.

Perhaps similarly, when I run recorder_console.py, I get the following error: urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)>

I did install the rootcertificate from the HoloLens2 onto my PC, if that's relevant.

Opencv UWP ARM64

This issue is peripherally related to the Hololens2ForCV repository:

The included OpenCV modules in the CameraWithCVAndCalibration sample do not include the dnn module, which is apparently not compatible with UWP at the moment (and eg. the vcpkg version also does not include dnn for UWP Arm64). It would be great if Microsoft can put effort into making OpenCV completely compatible with UWP and ARM64 applications, as this would greatly benefit development of CV applications for the Hololens 2.

Question about framerates of different cameras

Hi,

I did some experiments with the DEPTH_AHAT framerate. For all my tests, I ran SensorVisualization sample with some frame processing using OpenCV. The application was in the Release configuration and only the DEPTH_AHAT sensor running.
With all the extra processing the frame rate dropped and I got a framerate of 16 to 45 fps, with an average of 18 fps over 10000 frames.
I noticed that when the Hololens is lying still (on a table or a stand) OR the image in front of the Hololens doesn't move, the framerate drops to a constant framerate of 5 fps. Any changes in the frame causes the framerate to immediately jump up to the aforementioned average of 18fps (ranging from 16fps to 45 fps). This suggests some sort of adaptive framerate.

Am I understanding this correctly, does some sort of filtering take place to change the framerate?
If so, does this also happen for other sensors (I have only tested DEPTH_AHAT)?
Is there any way to stop this from happening?

Thanks.

Compiling error

Hello, I would like to know how to run the demos.
I just downloaded this repo and opened the .sln file. Then, compiled with Debug mode for ARM64 over USB.
I got these errors.

identifier "await" is undefined SensorVisualization C:\HoloLens\HoloLens2ForCV-main\Samples\SensorVisualization\SensorVisualization\Content\ModelRenderer.cpp 271

'GyroUpdateThread': illegal qualified name in member declaration (compiling source file Content\GyroRenderer.cpp) SensorVisualization C:\HoloLens\HoloLens2ForCV-main\Samples\SensorVisualization\SensorVisualization\Content\GyroRenderer.h 55

'MagUpdateThread': illegal qualified name in member declaration (compiling source file Content\MagRenderer.cpp) SensorVisualization C:\HoloLens\HoloLens2ForCV-main\Samples\SensorVisualization\SensorVisualization\Content\MagRenderer.h 55

'AccelUpdateThread': illegal qualified name in member declaration (compiling source file Content\AccelRenderer.cpp) SensorVisualization C:\HoloLens\HoloLens2ForCV-main\Samples\SensorVisualization\SensorVisualization\Content\AccelRenderer.h 55

'AccelUpdateThread': illegal qualified name in member declaration (compiling source file AppView.cpp) SensorVisualization C:\HoloLens\HoloLens2ForCV-main\Samples\SensorVisualization\SensorVisualization\Content\AccelRenderer.h 55

I googled 'await', but nothing found. The other errors are removed by removing '&' from 'SetColor(DirectX::XMFLOAT3 &color)'
Would you tell me how to solve this problem?

Occur Crush in InitializeSensors()

Hello,

I'm newbee on hololense developer, and thanks your contributions.

I have troubled to run sample projects.

The sample 'CalibrationVisualization' is OK. But the others are not worked.

When I tried debugging, the error occurred on line below.

winrt::check_hresult(m_pSensorDevice->QueryInterface(IID_PPV_ARGS(&m_pSensorDeviceConsent)));

In here, 'check_hresult' throws the exception

Of course, I already allowed access to camera stream in [hololense2 setting/privacy] and [device portal/research mode].

Please let me know the solution.
Thanks!

DEPTH_AHAT frame rate

Hi,

The API doc states that the DEPTH_AHAT sensor runs at 45 fps. We are trying to figure out the actual frame rate that we receive.
We found that the time difference between each frame is 0.2 to 0.1 seconds.
I just took the difference between the SensorTicks and divided them by SensorTicksPerSocond to get that.
The difference between System Time for the frames is consistent with the SensorTicks.
I tested this in CameraUpdateThread in the SensorVisualization Sample.
Did I miss something or is that the best framerate for DEPTH_AHAT?

Thanks.

Frame rates from StreamRecorder application

Hello,
after looking more closely at the timestamps of frames recorded by the StreamRecorder application, I noticed that the frame rates from research mode sensors are highly fluctuating. While the PV camera very reliably delivers a frame every 33 ms or so, the time delta between research mode frames (looking at AHAT, LF and RF at the moment) are frequently much bigger (around 100 ms or even up to 500 or 1000 ms) and are constantly changing. This makes it very difficult to work with the data if one requires synchronized frames.
My question is, is this due to the processing / architecture of the StreamRecorder (and could therefore be improved), or is this just how the frames arrive from the sensors?
Thanks

Processing of recordings fails (save_pclouds.py)

I record with those settings:

std::vector<ResearchModeSensorType> AppMain::kEnabledRMStreamTypes = {
	ResearchModeSensorType::DEPTH_LONG_THROW,
	ResearchModeSensorType::LEFT_FRONT,
	ResearchModeSensorType::LEFT_LEFT,
	ResearchModeSensorType::RIGHT_FRONT,
	ResearchModeSensorType::RIGHT_RIGHT
};
/* Supported not-ResearchMode streams:
{
	PV,  // RGB
	EYE  // Eye gaze tracking
}*/
std::vector<StreamTypes> AppMain::kEnabledStreamTypes = { StreamTypes::PV, StreamTypes::EYE };

When processing a recording with the provided python scripts I am getting the following error:

(recorder console) process 0
Extracting 2020-09-26-164659\Depth Long Throw.tar
Extracting 2020-09-26-164659\PV.tar
Extracting 2020-09-26-164659\VLC LF.tar
Extracting 2020-09-26-164659\VLC LL.tar
Extracting 2020-09-26-164659\VLC RF.tar
Extracting 2020-09-26-164659\VLC RR.tar
Processing images
.................................................................................................................
Saving point clouds
.Transform not found for timestamp 1.3245605218805149e+17
.Traceback (most recent call last):
  File "StreamRecorderConverter/recorder_console.py", line 278, in <module>
    main()
  File "StreamRecorderConverter/recorder_console.py", line 274, in main
    rs.cmdloop()
  File "C:\Users\[REDACTED]\AppData\Local\Programs\Python\Python37\lib\cmd.py", line 138, in cmdloop
    stop = self.onecmd(line)
  File "C:\Users\[REDACTED]\AppData\Local\Programs\Python\Python37\lib\cmd.py", line 217, in onecmd
    return func(arg)
  File "StreamRecorderConverter/recorder_console.py", line 89, in do_process
    recording_name)
  File "C:\[REDACTED]\HoloLens2ForCV\Samples\StreamRecorder\StreamRecorderConverter\process_all.py", line 38, in process_all
    save_pclouds(w_path, sensor_name)
  File "C:\[REDACTED]\HoloLens2ForCV\Samples\StreamRecorder\StreamRecorderConverter\save_pclouds.py", line 327, in save_pclouds
    disable_project_pinhole
  File "C:[REDACTED]\HoloLens2ForCV\Samples\StreamRecorder\StreamRecorderConverter\save_pclouds.py", line 164, in save_single_pcloud
    rgb_tmp = Path(rgb_parts[1]) / Path(rgb_parts[2])
IndexError: tuple index out of range

The script seems to work if I change the tuple indices from 1, 2 to 0, 1 for rgb_tmp and depth_tmp on lines 164 and 166 respectively:

rgb_tmp = Path(rgb_parts[0]) / Path(rgb_parts[1])
depth_parts = Path(depth_proj_path).parts[2:]
depth_tmp = Path(depth_parts[0]) / Path(depth_parts[1])

In DEPTH_AHAT mode, the StreamRecorder program cannot work

When setting the following configuration：
std::vector AppMain::kEnabledRMStreamTypes = { ResearchModeSensorType::DEPTH_AHAT };
The following logic does not work correctly：

 _while (!pCameraReader->m_fExit && pCameraReader->m_pRMSensor)
    {
        HRESULT hr = S_OK;
        IResearchModeSensorFrame* pSensorFrame = nullptr;

        hr = pCameraReader->m_pRMSensor->GetNextBuffer(&pSensorFrame);

        if (SUCCEEDED(hr))
        {
            std::lock_guard<std::mutex> guard(pCameraReader->m_sensorFrameMutex);
            if (pCameraReader->m_pSensorFrame)
            {
                pCameraReader->m_pSensorFrame->Release();
            }
            pCameraReader->m_pSensorFrame = pSensorFrame;
        }
    }_

The GetNextBuffer function crashed, resulting in the inability to perform the next step

Live stream to Desktop for online processing?

Hi,

Is it possible to live stream sensor streams to my desktop for online processing? I'm trying out some registration algorithms but I'm not sure whether the HPU can handle them? Thanks! (And thank you for this Git, can't wait to get everything up and running)

ARM Support

Hello,

Is there any way to compile samples in ARM configuration (not ARM64)?