enoxsoftware / hololenswithopencvforunityexample

HoloLens With OpenCVforUnity Example (Support for Hololens1 and Hololens2)

HoloLens With OpenCVForUnity Example

What's new

• Support for Hololens1 and Hololens2. (XR System: Legacy Built-in XR / XR Plugin Management WindowsMR / XR Plugin Management OpenXR)

Demo Video (old version)

Environment

• Hololens1 10.0.17763.3532 / Hololens2 20348.1522
• Windows 10 SDK 10.0.19041.0
• Visual Studio 2019
• Unity 2019.4.31f1 / 2020.3.38f1
• Microsoft Mixed Reality Toolkit v2.8.2
• OpenCV for Unity 2.5.1+
• EnoxSoftware/HoloLensCameraStream

Setup

1. Download the latest release unitypackage. HoloLensWithOpenCVForUnityExample.unitypackage
2. Create a new project. (HoloLensWithOpenCVForUnityExample)
   • Change the platform to UWP in the "Build Settings" window.
3. Import the OpenCVForUnity.
   • Select MenuItem[Tools/OpenCV for Unity/Open Setup Tools].
   • Click the [Move StreamingAssets Folder] button.
   • Leave the following files and delete the rest. ("StreamingAssets/OpenCVForUnity/objdetect/haarcascade_frontalface_alt.xml", "lbpcascade_ frontalface.xml")
4. Clone HoloLensCameraStream repository.
   • Copy the "HoloLensCameraStream/HoloLensVideoCaptureExample/Assets/CamStream/" folder to the "Assets/" folder.
5. Import the Microsoft Mixed Reality Toolkit. (Recommend using MixedRealityFeatureTool)
   • Follow the MRTK2 configuration dialog to set up the project.
6. Import the HoloLensWithOpenCVForUnityExample.unitypackage.
7. Add the "Assets/HoloLensWithOpenCVForUnityExample/*.unity" files to the "Scenes In Build" list in the "Build Settings" window.
8. Configure settings in the "Project Settings" window.
   • Add Define Symbols: the following to Scripting Define Symbols depending on the XR system used in your project; Legacy built-in XR: BUILTIN_XR; XR Plugin Management (Windows Mixed Reality): XR_PLUGIN_WINDOWSMR; XR Plugin Management (OpenXR):XR_PLUGIN_OPENXR.
   • Enable WebCam Capabilties in Publishing settings tab.
9. (Optional) Setup a performance environment suitable for Holorens. (See https://docs.microsoft.com/en-us/windows/mixed-reality/develop/unity/recommended-settings-for-unity)
10. Build the project: You can now build the Unity project, which generates a Visual Studio Solution (which you will then have to also build). With the Build Settings window still open, click Build. In the explorer window that appears, make a new folder called App, which should live as a sibling next to the 'Assets` folder. Then click Select Folder to generate the VS solution in that folder. Then wait for Unity to build the solution.
11. Open the VS Solution: When the solution is built, a Windows explorer folder will open. Open the newly-built VS solution, which lives in App/HoloLensWithOpenCVForUnityExample.sln. This is the solution that ultimately gets deployed to your HoloLens.
12. Configure the deploy settings: In the Visual Studio toolbar, change the solution platform from ARM to x86 if you are building for Hololens1 or to ARM64 if you are building for Hololens2; Change the deploy target (the green play button) to Device (if your HoloLens is plugged into your computer), or Remote Machine (if your HoloLens is connected via WiFi).
13. Run the app: Go to Debug > Start Debugging. Once the app is deployed to the HoloLens, you should see some confirmation output in the Output window.
    • (Print the AR marker "CanonicalMarker-d10-i1-sp500-bb1.pdf" on an A4 size paper)

Project Assets	Build Settings

Player Settings

ScreenShot (old version)

hololenswithopencvforunityexample's Issues

Error when building for Holo Lens 1 (x86)

The example runs fine in the editor (Unity 2019.4.17f1, also 2018.4.28f1), but when I build, I get the following error:

ArgumentException: The Assembly UnityEditor is referenced by OpenCVForUnityCSharpDLL ('Assets/OpenCVForUnity/org/OpenCVForUnityCSharpDLL.dll'). But the dll is not allowed to be included or could not be found.
UnityEditor.AssemblyHelper.AddReferencedAssembliesRecurse (System.String assemblyPath, System.Collections.Generic.List1[T] alreadyFoundAssemblies, System.String[] allAssemblyPaths, System.String[] foldersToSearch, System.Collections.Generic.Dictionary2[TKey,TValue] cache, UnityEditor.BuildTarget target) (at <2f1c602eae0d45c293fff3e3aef759fa>:0)
UnityEditor.AssemblyHelper.AddReferencedAssembliesRecurse (System.String assemblyPath, System.Collections.Generic.List1[T] alreadyFoundAssemblies, System.String[] allAssemblyPaths, System.String[] foldersToSearch, System.Collections.Generic.Dictionary2[TKey,TValue] cache, UnityEditor.BuildTarget target) (at <2f1c602eae0d45c293fff3e3aef759fa>:0)
UnityEditor.AssemblyHelper.FindAssembliesReferencedBy (System.String[] paths, System.String[] foldersToSearch, UnityEditor.BuildTarget target) (at <2f1c602eae0d45c293fff3e3aef759fa>:0)
UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr)

I tried to build the "HoloLensArUcoExample" scene of your example. Also an empty scene could not be build because of the same error.

What am I missing?

Issue deploying to Hololens 1

The example runs fine in the editor (Unity 2018.4.32), but when I build and deploy through Visual Studio 2019 (16.9.0), I get the following error:

RemoteCommandException: Windows cannot install package Template3D_1.0.0.0_x86__pzq3xp76mxafg because this package is not compatible with the device. The package requires OS version 10.0.18362.0 or higher on the Windows.XboxSRA device family. The device is currently running OS version 10.0.17763.1757.

My unity settings are the same as the provided screenshots, where Minimum Platform version is 10.0.10240. My Hololens is running 10.0.17763, where there is no available OS updates in Settings. Both Windows 18362 and 17763 SDK are installed in Visual Studio.

I then tried building an empty scene only using MRTK with the same build settings, this error does not occur.

What am I missing??

Help

In the case of face recognition, I only have the left eye to render the picture

ONNX model opset version 11 Integration - Not possible with Baraccuda/ Winml --- is it possible with OpenCV

Repected People,

Is there a possiblity to integrate a custom Object Detection ONNX model opset Version 11 with OpenCv for unity and is there a possiblity for it to integrate into hololens 2? Please do help me with the Question

Red video camera icon

Is there any way to run this code without the annoying flashing red video camera icon in the top left corner of my FOV?

FPS is very low

I followed the steps to set up and build the project in to hololens. It works but the fps is very low! Around 05 fps.
Is there anything wrong?

WorldManager deprecated

Hello,
I'm facing an issue with Unity 2021.2.16f1. It seems that the native UnityEngine.XR library has been deprecated.
So UnityEngine.XR.WSA.WorldManager is not recognized anymore.

For example in HololensCameraStreamToMatHelper (line 419) WorldManager raises an exception at compile time.

Is there any replacement method for this ?
Thank you for your support.

Question: Is the pluggin compatible with Hololens 2?

Hi, I´m trying the ArUco scene example with the Hololens 2 and I'm unable to compile the project, so I would like to know if the sample scene is compatible with Hololens 2.

Thanks!

.

.

Sme

Need help for the OpenPose for multiple persons using OpenCV and Unity.

Hi, I am using OpenCV for my Unity project in that I was working on the OpenPose. I am able to get the OpenPose for a single person. I need help for the OpenPose for multiple persons.

Can you please help me or share some of the references to get the OpenPose for multiple persons using Unity.

Thanks in advance.

How do I record while using Opencv in Hololnes?

How do I record while using Opencv in Hololnes?
pleases...

Problem with CameraIntrinsics

Hi,
I am starting on unity and openCV. When I import your project and import the other unitypackage needed I get this error. Do you have an idea how to solve it?


Paul

Fail to build the scenes on HL2 follow your instructions

Dear sir, I'm trying to build this repo following your 'README' instruction, and my environment settings are:

• Hololens2 20348.1522
• Windows 10 SDK 10.0.19041.0
• Visual Studio 2019
• Unity 2020.3.35f1
• OpenXR 1.6.0
• MRTK v2.8.3
• OpenCV for Unity 2.5.0
• EnoxSoftware/HoloLensCameraStream (.dll with 30K size)
  However, I failed many times on the "Reference Rewriter" error, could you give me any suggestions?
  Since the release note of v1.0.5 presents this repo supports HL2 well, thus I still hope to try it through.
  
  

Are there any chance to support HoloLens 2?

ArUco Example - Head Movement inverted

I am using the HoloLensWithOpenCVForUnityExample with Unity2019.4.2 and with the newest MixedRealityToolkit 2.4.0.
I have followed the steps 1-7 closely and think it should be setup correctly, as there are no build errors.

But somehow in the ArUco example my head movement seams to be exaggerated:
https://youtu.be/vrrV-by1Fpg

If you look closely the marker in the CameraFeed seams to be correct...

P.S.
I have tried with Unity 2018.4.24f and with the MRTK 2.2.0 and there this problem does not exist.

Hololens camera does not display images in PreviewQuad

Development environment：
opencv for untiy 2.2.0 unity 5.6.2f1 hololtoolkit 1.5.8
I build and run is success on PC but does not display images in PreviewQuad.
The Build Setting-> Capabilities -> Webcam is Checked.
I do not know which step is wrong configuration, I did not change any code.

Unable to load DLL 'opencvforunity'

I am getting this error in the log, I followed the setup instructions. So where am I going wrong?

Exception thrown: 'System.DllNotFoundException' in Assembly-CSharp.dll DllNotFoundException: Unable to load DLL 'opencvforunity': The specified module could not be found. (Exception from HRESULT: 0x8007007E) at OpenCVForUnity.Mat.core_Mat_n_1Mat__III(Int32 rows, Int32 cols, Int32 type) at HoloLensWithOpenCVForUnityExample.WebCamTextureToMatHelper.<init>d__23.MoveNext() at UnityEngine.SetupCoroutine.InvokeMoveNext(IEnumerator enumerator, IntPtr returnValueAddress) at UnityEngine.SetupCoroutine.$Invoke1(Int64 instance, Int64* args) at UnityEngine.Internal.$MethodUtility.InvokeMethod(Int64 instance, Int64* args, IntPtr method) (Filename: <Unknown> Line: 0)

ArUco tracking on Hololens 2 is not accurate

When trying to track an ArUco marker on the Hololens 2, the tracking is fairly close, but is not entirely accurate. I'm using the cameraInsrinsicsChecker scene to capture the correct CamMatrix, but it is slightly off. I'm using a 5 foot by 4 foot desk as my object, and when capturing the marker, the desk is tilted around 30 degrees.

Is there a better way to determine the correct CamMatrix settings? Or perhaps predetermined settings for the Hololens 2

Camera image of Hololens2 is not be displayed

Hi.
I want to use HololensWithOpenCVForUnityExample with Hololens2.
It works well in unity editor but when built to Hololens2, camera image of Hololens2 is not be displayed.
(I want to try Comic filter Example.)
I set up according to Readme of this, HololensWithOpenCVForUnityExample.
How can I use Hololens2's camera with HololensWithOpenCVForUnityExample??
In there any special procedure when using Hololens2, not Hololens??

I read #29 and one said we should change code in Videocapture.cs. But Videocapture.cs is not in Camstream folder in HoloLensCameraStream so I think Videocapture.cs is not used when using HololensWithOpenCVForUnityExample and it seems meaningless....

using version:
Hololens2
Unity 2019.4.23f1
Visual Studio 2019
OpenCVForUnity 2.4.4
HololensWithOpenCVForUnityExample 1.0.4
camnewnham/HoloLensCameraStream

Face recognition is not working

Hi,
I followed the exact steps to get the setup but I can't get the face recognition to work. All other examples are fine. The only difference might be the current version of OpenCV for Unity as I don't think I can get that older version 2.2.1.

Any ideas what to check? Example like HoloLensPhotoCaptureExample is taking picture but it doesn't select faces.

Has anyone had success with unity2020 lts

Has anyone had success with unity2020 lts.
Hololens2

Hitting Play on any scene crashes the editor in Unity 2017.2

I tried using this in Unity 2017.2 with the 2017.2 branch of HoloToolkit and it crashes the editor instantly when the play button is hit in any scene.

Please add RealTime FaceRecognition Example.

Hi,

In the RealTime FaceRecognition Example, it does not use multi-threading. If you have time, please update the RealTime FaceRecognition Example used by HoloLens, thank you.

obtaining CameraIntrinsics in photo mode for a different camera size (resolution)

Hi,

I'm trying to get the intrinsic Parameters of my HoloLens at a resolution of 2048x1152 (PhotoMode).
Modifying CameraIntrinsicsCheckerHelper.cs
Line 165 to: x.Width == 2048 &&
and Line 166 to: x.Height == 1152 &&

yields to following Problem in the VisualStudio output window:

Exception thrown: 'System.ArgumentException' in Assembly-CSharp.dll
Unhandled 'Platform.InvalidArgumentException' exception caught! - 'The parameter is incorrect.
Value does not fall within the expected range.', Sender: ''. Missing try/catch blocks.
(Filename: C:\buildslave\unity\build\PlatformDependent/MetroPlayer/MetroDebug.cpp Line: 46)

I guess this is because the camera doesn't support video mode in that resolution as doucemented here in the first table.

Is it possible to get the intrinsics in photo mode instead of video mode?

PrecompiledAssemblyException: Multiple precompiled assemblies with the same name nunit.framework.dll included or the current platform. Only one assembly with the same name is allowed per platform. Assembly paths: C:/Program Files/Unity/Hub/Editor/2018.4.28f1/Editor/Data/UnityExtensions/Unity/TestRunner/net35/unity-custom/nunit.framework.dll

Hi,

I would like to try out one of the examples, compile it and deploy it to the Hololens.

The scene runs fine in the Unity Editor Play Mode but when trying to build, I get these errors:

What are the steps necessary to fix them?

ArUco Example Cube Jumping Problem

When I tried the ArUco example, the AR game object(Red cube) was jumping around. The more far away from the marker, the more shaking.

ArUco Scene Not Building

I have found that the example will not build when the ArUco scene is included.

When I trying to build, the Building Player will freeze while trying to build HoloLensArucoExample, and never advance beyond that point and no errors are reported.

I was able to build to HoloLens Example when that scene is removed (even the ArUCo Calibration scene is fine), but it happens to be the scene that I needed. I am trying to discern what element in the scene might be causing the problem.

(It turned out that restarting the Unity Editor fixed the problem. I should have tried that before posting here).

The HololensArUcoExample is slow and has lacks

I tried running the HololensArUcoExample and was not very happy with my experience. It was very slow and had some lacks. When I moved the marker, the cube would move in steps. Is there a way to optimize this?

Works in Unity, not on HoloLens 2

I have followed the setup instructions.

Unity 2018.4.28f
MRTK 2.2

The scenes run fine in Unity with a web camera on PC, which should mean everything is configured correctly.

But when I build an deploy this to the HL2, the menus all work fine but nothing else works. For example, when I enable camera preview in the ArUco scene, I just get a blank white quad with no video feed.

In the player Publishing Settings I have explicitly enabled WebCam capabilities.

Some hints:

On the HL2, the scene quads remain white and appear in my left eye only.
Although using MRTK 2.2 in Unity 2019.4.15f, the quad did have the oval appearance in the comic filter scene. Otherwise they are square and white and in my left eye only.

The camera intrinsics scene scrollable window seems to show data on four cameras.

Am I missing some setting? Does the headset need to be in research mode?

I have also tried Unity version 2019.3.16f, 2019.4.15f, and 2020.2.4f and I have also tried MRTK 5.1.

In Visual Studio I build for ARM.

Any suggestion greatly appreciated.

GetNativeISpatialCoordinateSystemPtr() obsolete in 2021

https://docs.unity3d.com/ScriptReference/XR.WSA.WorldManager.GetNativeISpatialCoordinateSystemPtr.html says that

UnityEngine.XR.WSA.WorldManager.GetNativeISpatialCoordinateSystemPtr() is obsolete.

It is used on line 421 of HololensCameraStreamToMatHelper.cs.

I cannot find the new alternative for this, and it is the only thing preventing me from running this project after following all of the setup instructions.

What is the new up to date alternative for this line of code?

No Tools menu

I am trying to follow the instruction on this repo. Is it an issue is the OpenCvForUnity package doesn't create the Tool menu?

Is OpenCV available for custom object detection?

Hi,
I am interested in deploying OpenCV on HL2 for custom object detection.
Is OpenCV available to do that?
If so, are there some reference tutorial videos?
(I notice that OpenCV can detect human faces, and I would like to detect other things, like construction vehicles.)
Thank you so much

I am also facing same issues with

I am also facing same issues with
trial OpenCvUnity on window 10 and getting same error

Originally posted by @vikashsingh009 in #35 (comment)

The HoloLens camera doesn't work with OpenCV for Unity 2.1.5

Hi,
I'm trying to use the HoloLens camera with OpenCV for Unity 2.1.5 and HoloLensWithOpenCVForUnityExample, but it doesn't work.
I also heard that another person who had used it with OpenCVForUnityUWP_Beta3.unitypackage couldn't use it with OpenCV for Unity 2.1.5. What's the difference between them?

Missing Prefabs for each scene

What i did?

• Cloned project;
• Opened it via Unity 5.5.3
• Imported latest OpenCV For Unity from Asset store

What is wrong?
In each of scenes there are missing prefabs

Could You please:

• Add these prefabs to version control ? if not possible - provide what should be where via README or screenshots
• force text serialization (Edit->Project Settings -> Editor -> Asset Serialization -> Force Text

Unity Version

Hi,

I am having issues in regard to what version of Unity I should be using. When using the suggested version (Unity 2017.1.3f1), I am getting XR namespace doesn't exist errors. Apparently only Unity versions 2017.2 and after use XR and any previous version uses the VR namespace. Is there any updates as to which version of Unity we should use?

Thanks!

NullReferenceException error within WindowsMixedRealityToolkit

I followed the setup instructions 1-7. Then, I connected my Hololens through Window > XR > Holographic Emulation > "Remote to Device". However, when I press Play in Unity, it seems that the application is not connected to Hololens and instead access my PC's webcam. The following NullReferenceException error occurs.

NullReferenceException: Object reference not set to an instance of an object
Microsoft.MixedReality.Toolkit.WindowsMixedReality.WindowsMixedRealityCameraSettings.InitializeReprojectionUpdater () (at Assets/MixedRealityToolkit.Providers/WindowsMixedReality/WindowsMixedRealityCameraSettings.cs:101)
Microsoft.MixedReality.Toolkit.WindowsMixedReality.WindowsMixedRealityCameraSettings.Enable () (at Assets/MixedRealityToolkit.Providers/WindowsMixedReality/WindowsMixedRealityCameraSettings.cs:46)
Microsoft.MixedReality.Toolkit.BaseDataProviderAccessCoreSystem.Enable () (at Assets/MixedRealityToolkit/Services/BaseDataProviderAccessCoreSystem.cs:39)
Microsoft.MixedReality.Toolkit.CameraSystem.MixedRealityCameraSystem.Enable () (at Assets/MixedRealityToolkit.Services/CameraSystem/MixedRealityCameraSystem.cs:154)
Microsoft.MixedReality.Toolkit.MixedRealityToolkit+<>c.b__62_0 (Microsoft.MixedReality.Toolkit.IMixedRealityService service) (at Assets/MixedRealityToolkit/Services/MixedRealityToolkit.cs:934)
Microsoft.MixedReality.Toolkit.MixedRealityToolkit.ExecuteOnAllServices (System.Collections.Generic.IEnumerable1[T] services, System.Action1[T] execute) (at Assets/MixedRealityToolkit/Services/MixedRealityToolkit.cs:1024)
Microsoft.MixedReality.Toolkit.MixedRealityToolkit.ExecuteOnAllServicesInOrder (System.Action`1[T] execute) (at Assets/MixedRealityToolkit/Services/MixedRealityToolkit.cs:1010)
Microsoft.MixedReality.Toolkit.MixedRealityToolkit.EnableAllServices () (at Assets/MixedRealityToolkit/Services/MixedRealityToolkit.cs:934)
Microsoft.MixedReality.Toolkit.MixedRealityToolkit.OnEnable () (at Assets/MixedRealityToolkit/Services/MixedRealityToolkit.cs:620)

I've searched around and most solutions suggest fixing NullReferenceException error by changing the script contents but I am not sure if I am supposed to do that since the scripts are downloaded as a part of Microsoft's Mixed Reality Toolkit. Any ideas on how to fix this problem? I am trying to get the app to run on the Hololens instead of using my PC's Webcam.

Repeatedly enabling/disabling camera crashes the app.

I need to enable/disable marker detection repeatedly in my app. Doing this in this example app (switch between Aruco example and main menu back and forth) crashes after a while, unless I build the app in Debug configuration. How to safely disable/enable marker tracking repeatedly?

How to track a real world object

How to tweak the Face recognition code to detect some other real world object. I have the cascade XML.

Calib3d.findChessboardCorners always returns true

Hi,

Using OptimizationWebCamTextureToMatHelper for HoloLens, I call findChessboardCorners as Calib3d.findChessboardCorners(gray, boardDimensions, corners, flagsCorner) once the . It always returns true even though there is no checssboard in the gray mat.

Codes:

In Update() following as your example,

Mat rgbaMat = webCamTextureToMatHelper.GetDownScaleMat(webCamTextureToMatHelper.GetMat());
Imgproc.cvtColor(rgbaMat, grayMat, Imgproc.COLOR_RGBA2GRAY);
MatOfPoint2f points = new MatOfPoint2f();
patternFound = getCorners(grayMat, points); <-- patternFound is always true;

bool getCorners(Mat gray, MatOfPoint2f corners)
{
if (!Calib3d.findChessboardCorners(gray, boardDimensions, corners, flagsCorner))
return false;
Imgproc.cornerSubPix(gray, corners, winSize, zoneSize, criteria);
return true;
}

Thanks for your help,
Bobby

Can you only get the camera in Gray Mode?

How can I make it detect colors and adapt the "HoloLensComicFilterExample" example to be equal to "MultiObjectTrackingBasedOnColorExample".

An example of how to detect colors would be great.
Thanks!

Scripts not attached to scenes

Hi,

I've been trying to open up the scenes, following the instructions on the readme file, but it looks like none of the scripts were attached when they were imported. Do you know what I may be doing wrong?

Identify Cylinder shapes of different dimension

Not really an issue but a query. Can this example be used to identify cylinders of different dimensions? If so which file should I have to add/change?

Using Examples on The HoloLens2

A previous issue describes steps to make the examples work on the HoloLens2. This issue raises some questions for me.

When i follow the steps i get multiple errors for missing assembly references. photo capture, webcam, hololensCamerastream, gesturerecogniser, VR and VideoCapture are all unrecognised.
Unity also tried to update my API's but this failed.

To be fair i'm using the latest LTS version of unity (2020.3.6f1) and MRTK 2.6.
Could this be causing the errors? Please advise.

Previous issue: #29 (comment)

The type or namespace name 'CameraIntrinsics' could not be found

I followed the setup in Readme.md, but the error in the picture below will always appear, unable to find 'CameraIntrinsics', I would like to ask which step may be the problem?

Environment I use:
HoloLens 2
Visual Studio 2019
Unity 2019.4.7f1
Microsoft Mixed Reality Toolkit v2.2.0
OpenCV for Unity 2.4.7

Can this example work on hololens2 (arm) ?

Could someone try out if this native plugin can work with Hololens2 and replace HololensCameraStream?
https://github.com/carmines/UnityPluginCollection/tree/master/CameraCapture

I haven't been able to check it out as I don't have Hololens2 at hand, but I have confirmed that I can get video preview and camera matrices successfully on Hololens.

What is the specific method you are using for face recognition processing?

HoloLensArUcoExample as Markerless Example

Hi,

I'm looking for an Example like the HoloLensArUcoExample but with free picures like the Markerless example in the normal OpenCV for Unity samples.

And what is in your opinion best practice to program this tracking. I thing in each sample there use a other workfolw.

Thank you

Facial detection overlays not working

I can build and run the samples on the HoloLens but when I run the Face Detection Overlays example and then open up images of faces on my PC monitor, I'm not getting the expected red boxes over detected faces as shown in your screenshots. I do have haarcascade_frontalface_alt.xml and lbpcascade_frontalface.xml in StreamingAssets as mentioned in #13.