Can this example work on hololens2 (arm) ? about hololenswithopencvforunityexample

[cookieofcode]

@EnoxSoftware The example works on HoloLens 2 with some modifications. The problem is the changed HoloLensCameraStream library and the newly introduced dependency on the CameraIntrinsics:

• Either use VulcanTechnologies/HoloLensCameraStream (and remove the CameraIntrinsicsChecker) or modify the following line in camnewnham/HoloLensCameraStream VideoCapture.cs:
  static readonly MediaStreamType STREAM_TYPE = MediaStreamType.VideoRecord; to static readonly MediaStreamType STREAM_TYPE = MediaStreamType.VideoPreview;. This change is necessary because I found that the HoloLens 2 does not receive the Camera and View Transformation Matrix using VideoRecord stream type.
• It works using Release|ARM. However, there are some UnityExceptions in the Debug window unrelated to this issue.

[cookieofcode]

None of the examples are working. I tried to follow all the instructions from the readme. As soon as I run even the License scene I see an error message on hololens in red. It says: "UnityException: destroy can only be called from main thread."
For other scenes, I am getting ArgumentOutOfRangeException for the parameter named index.
I am running using ARM64. I have turned on webcam capability too. In unity editors everything is fine.
I had one concern. When I added the CamStream from https://github.com/camnewnham/HoloLensCameraStream
I was prompted by unity that it was using obsolete API and unity would automatically update it. I don't know if that is the reason. Also I am using MRTK 2.4

One problem could be the API Update, as HoloLensCameraStream contains classes named the same as in the Unity API (e.g. https://docs.unity3d.com/ScriptReference/Windows.WebCam.VideoCapture.html). Cancel the API Updater dialog and manually fix the errors.

[EnoxSoftware]

This is a great post!
https://github.com/VulcanTechnologies/HoloLensCameraStream/issues/63#issuecomment-718216694

[nous-]

@EnoxSoftware Note on that post above, I neglected to mention to transpose the output of the HololensCameraStream TryGetWorldCoords to get a column major matrix which was required for all the other steps. Updated the original post to show this happening in the plugin

[aioniosfoititis]

I can confirm that the changes that @cookieofcode proposes do work for holoLens 2 for most of the examples of this guide.

Although I noticed two more problems.The first one is that in the face recognition examples only in the left-Lens of the glasses the rectangles appear, and the second one is that HoloLens PhotoCapture example seams not to be working.

The total examples work fine in the unity editor.

[cookieofcode]

The type or namespace name 'GestureRecognizer' could not be found

Looks like a missing import of the "UnityEngine.XR.WSA.Input" (instead of UnityEngine.VR.WSA.Input). Not sure about this one.

The type or namespace name 'WebCam' does not exist in the namespace 'UnityEngine.XR.WSA'

Unity moved UnityEngine.XR.WSA.WebCam to UnityEngine.Windows.WebCam. I recommend to do the updates manually, as the API Updater may interfere with HoloLensCamStream.

As you can see for the 896x504 case cx,cy are: 1123, 614. And the imageWidth, imageHeight from IntrinsicsMatrix is 2272x1278. .
These values are not making sense to me. Would really appreciate if you could comment on this too. Thank you.

I'm not familiar with this scene, but if something with the intrinisic is wrong, check my comment with the code suggestions above (#29 (comment)).

[aioniosfoititis]

@Abdul-Mukit hi, go into the mixed reality settings and go for single pass instead of single pass instanced. That fixed the issue for me

[cookieofcode]

The video preview of CameraCapture works on the HoloLens2. Could not test it with HoloLensWithOpenCVForUnityExample and OpenCVForUnity yet.

[EnoxSoftware]

The video preview of CameraCapture works on the HoloLens2. Could not test it with HoloLensWithOpenCVForUnityExample and OpenCVForUnity yet.

Thank you for the useful information.
Unfortunately, the current functionality of CameraCapture does not allow fast access to the ByteArray of raw pixel data, so there is a problem with the OpenCV integration.

[ssyplot]

The video preview of CameraCapture works on the HoloLens2. Could not test it with HoloLensWithOpenCVForUnityExample and OpenCVForUnity yet.

Thank you for the useful information.
Unfortunately, the current functionality of CameraCapture does not allow fast access to the ByteArray of raw pixel data, so there is a problem with the OpenCV integration.

HoloLens 2 Research Mode enables access to the raw streams on device (depth camera, gray-scale cameras, IMU). https://github.com/microsoft/HoloLens2ForCV

[EnoxSoftware]

Hi There

Thank you for the helpful information.
I found a very interesting repository.
https://github.com/qian256/HoloLensARToolKit
This repository seems to include how to get the video preview and camera matrices of the camera on the Hololens2 device I was looking for.

[cookieofcode]

The video preview of CameraCapture works on the HoloLens2. Could not test it with HoloLensWithOpenCVForUnityExample and OpenCVForUnity yet.

Thank you for the useful information.
Unfortunately, the current functionality of CameraCapture does not allow fast access to the ByteArray of raw pixel data, so there is a problem with the OpenCV integration.

HoloLens 2 Research Mode enables access to the raw streams on device (depth camera, gray-scale cameras, IMU). https://github.com/microsoft/HoloLens2ForCV

The Research Mode is interesting, but it needs to be specifically enabled and currently needs Windows Insider Preview.

Hi There

Thank you for the helpful information.
I found a very interesting repository.
https://github.com/qian256/HoloLensARToolKit
This repository seems to include how to get the video preview and camera matrices of the camera on the Hololens2 device I was looking for.

I've also discovered the HoloLensARToolKit repository, but it seems rather bloated for my purpose, as I don't need Trackers.

Aquiring and displaying a video stream with HoloLensCameraStream using MediaCapture works fine on the HoloLens 2. I'm not sure if all transformations work yet.

[EnoxSoftware]

The best scenario for me is for the HololensCameraStream developers to incorporate code that supports Hololens2.
From looking at the latest HololensARToolKit code, it looks like there are tips to properly support Hololens2.

[EnoxSoftware]

This repository shows that the CameraStream plugin works successfully on hololens2.
https://github.com/abist-co-ltd/hololens-opencv-laserpointer
Basically it seems that you just need to change the build architecture to ARM64.

Anyway, I don't have hololens2 at hand, so I'll put this issue on hold for a while.

[EnoxSoftware]

@cookieofcode Thank you for the helpful information.

In the test I did on Hololens1, It was found that the combining static readonly MediaStreamType STREAM_TYPE = MediaStreamType.VideoPreview and mediaCapture.AddVideoEffectAsync () slows down significantly.
Therefore, I changed to camnewnham/HoloLensCameraStream, but it was unexpected that side effects appeared on Hololens2.

[Abdul-Mukit]

None of the examples are working. I tried to follow all the instructions from the readme. As soon as I run even the License scene I see an error message on hololens in red. It says: "UnityException: destroy can only be called from main thread."
For other scenes, I am getting ArgumentOutOfRangeException for the parameter named index.
I am running using ARM64. I have turned on webcam capability too. In unity editors everything is fine.
I had one concern. When I added the CamStream from https://github.com/camnewnham/HoloLensCameraStream
I was prompted by unity that it was using obsolete API and unity would automatically update it. I don't know if that is the reason. Also I am using MRTK 2.4

[Abdul-Mukit]

Hi, @cookieofcode. Thank you for your answer. Here are a bit more details.
These are the updates that auto-updated was talking about:
For MatrixUsageApp.cs, ProjectionExample.cs and VideoPanelApp.cs;
error CS0246: The type or namespace name 'GestureRecognizer' could not be found (are you missing a using directive or an assembly reference?)
and for ProjectionExample.cs; error CS0246: The type or namespace name 'GestureRecognizer' could not be found (are you missing a using directive or an assembly reference?)
I don't think they have anything to do with the Hololens error due to not calling destroy from the main thread I was originally getting.
So, I imported the CamStream folder without the examples folder and I was able to avoid this API update issue.

After that, it appeared again when importing OpenCVForUnityExamples.unitypackage. These were the problems.
1. Assets\HoloLensWithOpenCVForUnityExample\HoloLensPhotoCaptureExample\HoloLensPhotoCaptureExample.cs(12,54): error CS0234: The type or namespace name 'WebCam' does not exist in the namespace 'UnityEngine.XR.WSA' (are you missing an assembly reference?)
2. Assets\HoloLensWithOpenCVForUnityExample\HoloLensPhotoCaptureExample\HoloLensPhotoCaptureExample.cs(11,26): error CS0234: The type or namespace name 'WebCam' does not exist in the namespace 'UnityEngine.XR.WSA' (are you missing an assembly reference?)
3. Assets\HoloLensWithOpenCVForUnityExample\HoloLensPhotoCaptureExample\HoloLensPhotoCaptureExample.cs(99,42): error CS0246: The type or namespace name 'PhotoCapture' could not be found (are you missing a using directive or an assembly reference?)
4.
Assets\HoloLensWithOpenCVForUnityExample\HoloLensPhotoCaptureExample\HoloLensPhotoCaptureExample.cs(105,31): error CS0246: The type or namespace name 'PhotoCapture' could not be found (are you missing a using directive or an assembly reference?)
5. Assets\HoloLensWithOpenCVForUnityExample\HoloLensPhotoCaptureExample\HoloLensPhotoCaptureExample.cs(135,30): error CS0246: The type or namespace name 'PhotoCapture' could not be found (are you missing a using directive or an assembly reference?)
6. Assets\HoloLensWithOpenCVForUnityExample\HoloLensPhotoCaptureExample\HoloLensPhotoCaptureExample.cs(135,70): error CS0246: The type or namespace name 'PhotoCaptureFrame' could not be found (are you missing a using directive or an assembly reference?)
7. Assets\HoloLensWithOpenCVForUnityExample\HoloLensPhotoCaptureExample\HoloLensPhotoCaptureExample.cs(200,30): error CS0246: The type or namespace name 'PhotoCapture' could not be found (are you missing a using directive or an assembly reference?)
8. Assets\HoloLensWithOpenCVForUnityExample\HoloLensPhotoCaptureExample\HoloLensPhotoCaptureExample.cs(25,9): error CS0246: The type or namespace name 'PhotoCapture' could not be found (are you missing a using directive or an assembly reference?)

I triggered auto-update and the errors went away. But the original error of "UnityException: destroy can only be called from the main thread." is still there in hololens and none of the examples work. Do you still think manually fixing might be the answer?

[Abdul-Mukit]

Only the "Check Intrinsic" scene worked, sort of. I got the following output. I am not sure if the values are correct for 896x504 case.
Following is a snapshot:

As you can see for the 896x504 case cx,cy are: 1123, 614. And the imageWidth, imageHeight from IntrinsicsMatrix is 2272x1278. .
These values are not making sense to me. Would really appreciate if you could comment on this too. Thank you.

[Abdul-Mukit]

Thanks again. I manually fixed the API update issues. Just needed to change to UnityEngine.Windows.WebCam in HoloLensPhotoCaptureExample.cs. However, it didn't solve the issue. Still getting the original unityException.

A little progress. Along the lines of your suggestion of using the MediaStreamType.VideoPreview, I edited the CameraIntrinsicsChecker class to use it instead of VideoRecord type. And now I get the correct intrinsic. Here is a snapshot:

I'll try out the same in the PluginProject and see if works. :)

[Abdul-Mukit]

I rebuilt the plugin as suggested in #29 (comment)
Finally, all examples were working. Thank you, @cookieofcode.
So the summary is that by changing the plugin and the CameraIntrinsicsCheckerHelper.cs (for the intrinsic scene) got the project working on HL2 for me. One difference is that it only worked on the ARM64, never worked on ARM. As was mentioned by @aioniosfoititis the display is only on the left eye. Didn't check the calibration using Aruco marker scene.
Overall, in all scenes, I got around 10-30fps. Works for me. Thank you.
:)

[A00107408]

Thank you @Abdul-Mukit @cookieofcode and @EnoxSoftware. Following your comments I got it working on HL2, with some exceptions:

In Summary:

I made the change to camnewnham/HoloLensCameraStream suggested in #29 (comment) and built the dlls.

Importing HoloLensWithOpenCVForUnityExample.unitypackage in Unity 2018.4.28f1 didn't trigger the API updater, so happy days, no errors to fix manually.

Built for ARM.

Problems:

ArUco scene works fine in both eyes, until I turn on camera preview. It flickers once or twice, then stops working.

Comic filter scene works in left eye only?

Hope this helps. Thanks again.

I can confirm that this works in Unity 2019.4.15f with MRTK 2.5.4.

[Abdul-Mukit]

Still getting webcam preview only on the left eye.

[Abdul-Mukit]

Thank you. That solved the problem.

[Rutherfordum]

Tested on HoloLens x86 and HoloLens2 ARM64
Master/Release

1. change the following line in camnewnham / HoloLensCameraStream VideoCapture.cs:
   static readonly MediaStreamType STREAM_TYPE = MediaStreamType.VideoRecord; to static readonly MediaStreamType STREAM_TYPE = MediaStreamType.VideoPreview; and collect the solution. Replace the resulting HoloLensCameraStream.dll in the Unity Assets \ CamStream \ Plugins \ WSA project

2. change the line, in the Unity project CameraIntrinsicsCheckerHelper.cs,
   static readonly MediaStreamType STREAM_TYPE = MediaStreamType.VideoRecord; to static readonly MediaStreamType STREAM_TYPE = MediaStreamType.VideoPreview

Thank you very much @cookieofcode and @Abdul-Mukit

[EnoxSoftware]

I found a very interesting Hololens2 and OpenCVForUnity integration example project.
It contains a concise and simple code for using the Locatable camera on Hololens2.
https://github.com/cookieofcode/LabAssistVision

[AlexanderRodecape]

Thanks to the contributions here I was able to get this to run on a hololens 2 built for arm64. However, on the face recognition example, the red box overlays don't seem to be in the right place when looking through the hololens. On the device portal live preview the overlay looks to be in the correct place, though. Any suggestions? Thanks for creating this repo.