enoxsoftware / hololenswithopencvforunityexample Goto Github PK
View Code? Open in Web Editor NEWHoloLens With OpenCVforUnity Example (Support for Hololens1 and Hololens2)
License: MIT License
HoloLens With OpenCVforUnity Example (Support for Hololens1 and Hololens2)
License: MIT License
Repected People,
Is there a possiblity to integrate a custom Object Detection ONNX model opset Version 11 with OpenCv for unity and is there a possiblity for it to integrate into hololens 2? Please do help me with the Question
Hi,
I'm looking for an Example like the HoloLensArUcoExample but with free picures like the Markerless example in the normal OpenCV for Unity samples.
And what is in your opinion best practice to program this tracking. I thing in each sample there use a other workfolw.
Thank you
How do I record while using Opencv in Hololnes?
pleases...
The example runs fine in the editor (Unity 2018.4.32), but when I build and deploy through Visual Studio 2019 (16.9.0), I get the following error:
RemoteCommandException: Windows cannot install package Template3D_1.0.0.0_x86__pzq3xp76mxafg because this package is not compatible with the device. The package requires OS version 10.0.18362.0 or higher on the Windows.XboxSRA device family. The device is currently running OS version 10.0.17763.1757.
My unity settings are the same as the provided screenshots, where Minimum Platform version is 10.0.10240. My Hololens is running 10.0.17763, where there is no available OS updates in Settings. Both Windows 18362 and 17763 SDK are installed in Visual Studio.
I then tried building an empty scene only using MRTK with the same build settings, this error does not occur.
What am I missing??
Has anyone had success with unity2020 lts.
Hololens2
UnityEngine.XR.WSA.WorldManager.GetNativeISpatialCoordinateSystemPtr() is obsolete.
It is used on line 421 of HololensCameraStreamToMatHelper.cs.
I cannot find the new alternative for this, and it is the only thing preventing me from running this project after following all of the setup instructions.
What is the new up to date alternative for this line of code?
When I tried the ArUco example, the AR game object(Red cube) was jumping around. The more far away from the marker, the more shaking.
How to tweak the Face recognition code to detect some other real world object. I have the cascade XML.
I am using the HoloLensWithOpenCVForUnityExample with Unity2019.4.2 and with the newest MixedRealityToolkit 2.4.0.
I have followed the steps 1-7 closely and think it should be setup correctly, as there are no build errors.
But somehow in the ArUco example my head movement seams to be exaggerated:
https://youtu.be/vrrV-by1Fpg
If you look closely the marker in the CameraFeed seams to be correct...
P.S.
I have tried with Unity 2018.4.24f and with the MRTK 2.2.0 and there this problem does not exist.
When trying to track an ArUco marker on the Hololens 2, the tracking is fairly close, but is not entirely accurate. I'm using the cameraInsrinsicsChecker scene to capture the correct CamMatrix, but it is slightly off. I'm using a 5 foot by 4 foot desk as my object, and when capturing the marker, the desk is tilted around 30 degrees.
Is there a better way to determine the correct CamMatrix settings? Or perhaps predetermined settings for the Hololens 2
Hi,
I'm trying to use the HoloLens camera with OpenCV for Unity 2.1.5 and HoloLensWithOpenCVForUnityExample, but it doesn't work.
I also heard that another person who had used it with OpenCVForUnityUWP_Beta3.unitypackage couldn't use it with OpenCV for Unity 2.1.5. What's the difference between them?
Hi,
I am having issues in regard to what version of Unity I should be using. When using the suggested version (Unity 2017.1.3f1), I am getting XR namespace doesn't exist errors. Apparently only Unity versions 2017.2 and after use XR and any previous version uses the VR namespace. Is there any updates as to which version of Unity we should use?
Thanks!
Hi,
I've been trying to open up the scenes, following the instructions on the readme file, but it looks like none of the scripts were attached when they were imported. Do you know what I may be doing wrong?
What i did?
What is wrong?
In each of scenes there are missing prefabs
Could You please:
The example runs fine in the editor (Unity 2019.4.17f1, also 2018.4.28f1), but when I build, I get the following error:
ArgumentException: The Assembly UnityEditor is referenced by OpenCVForUnityCSharpDLL ('Assets/OpenCVForUnity/org/OpenCVForUnityCSharpDLL.dll'). But the dll is not allowed to be included or could not be found.
UnityEditor.AssemblyHelper.AddReferencedAssembliesRecurse (System.String assemblyPath, System.Collections.Generic.List1[T] alreadyFoundAssemblies, System.String[] allAssemblyPaths, System.String[] foldersToSearch, System.Collections.Generic.Dictionary
2[TKey,TValue] cache, UnityEditor.BuildTarget target) (at <2f1c602eae0d45c293fff3e3aef759fa>:0)
UnityEditor.AssemblyHelper.AddReferencedAssembliesRecurse (System.String assemblyPath, System.Collections.Generic.List1[T] alreadyFoundAssemblies, System.String[] allAssemblyPaths, System.String[] foldersToSearch, System.Collections.Generic.Dictionary
2[TKey,TValue] cache, UnityEditor.BuildTarget target) (at <2f1c602eae0d45c293fff3e3aef759fa>:0)
UnityEditor.AssemblyHelper.FindAssembliesReferencedBy (System.String[] paths, System.String[] foldersToSearch, UnityEditor.BuildTarget target) (at <2f1c602eae0d45c293fff3e3aef759fa>:0)
UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr)
I tried to build the "HoloLensArUcoExample" scene of your example. Also an empty scene could not be build because of the same error.
What am I missing?
I tried using this in Unity 2017.2 with the 2017.2 branch of HoloToolkit and it crashes the editor instantly when the play button is hit in any scene.
Hi, I am using OpenCV for my Unity project in that I was working on the OpenPose. I am able to get the OpenPose for a single person. I need help for the OpenPose for multiple persons.
Can you please help me or share some of the references to get the OpenPose for multiple persons using Unity.
Thanks in advance.
.
I have followed the setup instructions.
Unity 2018.4.28f
MRTK 2.2
The scenes run fine in Unity with a web camera on PC, which should mean everything is configured correctly.
But when I build an deploy this to the HL2, the menus all work fine but nothing else works. For example, when I enable camera preview in the ArUco scene, I just get a blank white quad with no video feed.
In the player Publishing Settings I have explicitly enabled WebCam capabilities.
Some hints:
On the HL2, the scene quads remain white and appear in my left eye only.
Although using MRTK 2.2 in Unity 2019.4.15f, the quad did have the oval appearance in the comic filter scene. Otherwise they are square and white and in my left eye only.
The camera intrinsics scene scrollable window seems to show data on four cameras.
Am I missing some setting? Does the headset need to be in research mode?
I have also tried Unity version 2019.3.16f, 2019.4.15f, and 2020.2.4f and I have also tried MRTK 5.1.
In Visual Studio I build for ARM.
Any suggestion greatly appreciated.
Could someone try out if this native plugin can work with Hololens2 and replace HololensCameraStream?
https://github.com/carmines/UnityPluginCollection/tree/master/CameraCapture
I haven't been able to check it out as I don't have Hololens2 at hand, but I have confirmed that I can get video preview and camera matrices successfully on Hololens.
I can build and run the samples on the HoloLens but when I run the Face Detection Overlays example and then open up images of faces on my PC monitor, I'm not getting the expected red boxes over detected faces as shown in your screenshots. I do have haarcascade_frontalface_alt.xml
and lbpcascade_frontalface.xml
in StreamingAssets as mentioned in #13.
How can I make it detect colors and adapt the "HoloLensComicFilterExample" example to be equal to "MultiObjectTrackingBasedOnColorExample".
An example of how to detect colors would be great.
Thanks!
Dear sir, I'm trying to build this repo following your 'README' instruction, and my environment settings are:
Hi,
I'm trying to get the intrinsic Parameters of my HoloLens at a resolution of 2048x1152 (PhotoMode).
Modifying CameraIntrinsicsCheckerHelper.cs
Line 165 to: x.Width == 2048 &&
and Line 166 to: x.Height == 1152 &&
yields to following Problem in the VisualStudio output window:
Exception thrown: 'System.ArgumentException' in Assembly-CSharp.dll
Unhandled 'Platform.InvalidArgumentException' exception caught! - 'The parameter is incorrect.
Value does not fall within the expected range.', Sender: ''. Missing try/catch blocks.
(Filename: C:\buildslave\unity\build\PlatformDependent/MetroPlayer/MetroDebug.cpp Line: 46)
I guess this is because the camera doesn't support video mode in that resolution as doucemented here in the first table.
Is it possible to get the intrinsics in photo mode instead of video mode?
Hi, I´m trying the ArUco scene example with the Hololens 2 and I'm unable to compile the project, so I would like to know if the sample scene is compatible with Hololens 2.
Thanks!
A previous issue describes steps to make the examples work on the HoloLens2. This issue raises some questions for me.
When i follow the steps i get multiple errors for missing assembly references. photo capture, webcam, hololensCamerastream, gesturerecogniser, VR and VideoCapture are all unrecognised.
Unity also tried to update my API's but this failed.
To be fair i'm using the latest LTS version of unity (2020.3.6f1) and MRTK 2.6.
Could this be causing the errors? Please advise.
Previous issue: #29 (comment)
Hello,
I'm facing an issue with Unity 2021.2.16f1. It seems that the native UnityEngine.XR library has been deprecated.
So UnityEngine.XR.WSA.WorldManager is not recognized anymore.
For example in HololensCameraStreamToMatHelper (line 419) WorldManager raises an exception at compile time.
Is there any replacement method for this ?
Thank you for your support.
I need to enable/disable marker detection repeatedly in my app. Doing this in this example app (switch between Aruco example and main menu back and forth) crashes after a while, unless I build the app in Debug configuration. How to safely disable/enable marker tracking repeatedly?
Development environment:
opencv for untiy 2.2.0 unity 5.6.2f1 hololtoolkit 1.5.8
I build and run is success on PC but does not display images in PreviewQuad.
The Build Setting-> Capabilities -> Webcam is Checked.
I do not know which step is wrong configuration, I did not change any code.
In the case of face recognition, I only have the left eye to render the picture
Hi,
I followed the exact steps to get the setup but I can't get the face recognition to work. All other examples are fine. The only difference might be the current version of OpenCV for Unity as I don't think I can get that older version 2.2.1.
Any ideas what to check? Example like HoloLensPhotoCaptureExample is taking picture but it doesn't select faces.
I am trying to follow the instruction on this repo. Is it an issue is the OpenCvForUnity package doesn't create the Tool menu?
Hi,
Using OptimizationWebCamTextureToMatHelper for HoloLens, I call findChessboardCorners as Calib3d.findChessboardCorners(gray, boardDimensions, corners, flagsCorner) once the . It always returns true even though there is no checssboard in the gray mat.
Codes:
In Update() following as your example,
Mat rgbaMat = webCamTextureToMatHelper.GetDownScaleMat(webCamTextureToMatHelper.GetMat());
Imgproc.cvtColor(rgbaMat, grayMat, Imgproc.COLOR_RGBA2GRAY);
MatOfPoint2f points = new MatOfPoint2f();
patternFound = getCorners(grayMat, points); <-- patternFound is always true;
bool getCorners(Mat gray, MatOfPoint2f corners)
{
if (!Calib3d.findChessboardCorners(gray, boardDimensions, corners, flagsCorner))
return false;
Imgproc.cornerSubPix(gray, corners, winSize, zoneSize, criteria);
return true;
}
Thanks for your help,
Bobby
I tried running the HololensArUcoExample and was not very happy with my experience. It was very slow and had some lacks. When I moved the marker, the cube would move in steps. Is there a way to optimize this?
I have found that the example will not build when the ArUco scene is included.
When I trying to build, the Building Player will freeze while trying to build HoloLensArucoExample, and never advance beyond that point and no errors are reported.
I was able to build to HoloLens Example when that scene is removed (even the ArUCo Calibration scene is fine), but it happens to be the scene that I needed. I am trying to discern what element in the scene might be causing the problem.
(It turned out that restarting the Unity Editor fixed the problem. I should have tried that before posting here).
I am getting this error in the log, I followed the setup instructions. So where am I going wrong?
Exception thrown: 'System.DllNotFoundException' in Assembly-CSharp.dll DllNotFoundException: Unable to load DLL 'opencvforunity': The specified module could not be found. (Exception from HRESULT: 0x8007007E) at OpenCVForUnity.Mat.core_Mat_n_1Mat__III(Int32 rows, Int32 cols, Int32 type) at HoloLensWithOpenCVForUnityExample.WebCamTextureToMatHelper.<init>d__23.MoveNext() at UnityEngine.SetupCoroutine.InvokeMoveNext(IEnumerator enumerator, IntPtr returnValueAddress) at UnityEngine.SetupCoroutine.$Invoke1(Int64 instance, Int64* args) at UnityEngine.Internal.$MethodUtility.InvokeMethod(Int64 instance, Int64* args, IntPtr method) (Filename: <Unknown> Line: 0)
Is there any way to run this code without the annoying flashing red video camera icon in the top left corner of my FOV?
I followed the setup instructions 1-7. Then, I connected my Hololens through Window > XR > Holographic Emulation > "Remote to Device". However, when I press Play in Unity, it seems that the application is not connected to Hololens and instead access my PC's webcam. The following NullReferenceException error occurs.
NullReferenceException: Object reference not set to an instance of an object
Microsoft.MixedReality.Toolkit.WindowsMixedReality.WindowsMixedRealityCameraSettings.InitializeReprojectionUpdater () (at Assets/MixedRealityToolkit.Providers/WindowsMixedReality/WindowsMixedRealityCameraSettings.cs:101)
Microsoft.MixedReality.Toolkit.WindowsMixedReality.WindowsMixedRealityCameraSettings.Enable () (at Assets/MixedRealityToolkit.Providers/WindowsMixedReality/WindowsMixedRealityCameraSettings.cs:46)
Microsoft.MixedReality.Toolkit.BaseDataProviderAccessCoreSystem.Enable () (at Assets/MixedRealityToolkit/Services/BaseDataProviderAccessCoreSystem.cs:39)
Microsoft.MixedReality.Toolkit.CameraSystem.MixedRealityCameraSystem.Enable () (at Assets/MixedRealityToolkit.Services/CameraSystem/MixedRealityCameraSystem.cs:154)
Microsoft.MixedReality.Toolkit.MixedRealityToolkit+<>c.b__62_0 (Microsoft.MixedReality.Toolkit.IMixedRealityService service) (at Assets/MixedRealityToolkit/Services/MixedRealityToolkit.cs:934)
Microsoft.MixedReality.Toolkit.MixedRealityToolkit.ExecuteOnAllServices (System.Collections.Generic.IEnumerable1[T] services, System.Action
1[T] execute) (at Assets/MixedRealityToolkit/Services/MixedRealityToolkit.cs:1024)
Microsoft.MixedReality.Toolkit.MixedRealityToolkit.ExecuteOnAllServicesInOrder (System.Action`1[T] execute) (at Assets/MixedRealityToolkit/Services/MixedRealityToolkit.cs:1010)
Microsoft.MixedReality.Toolkit.MixedRealityToolkit.EnableAllServices () (at Assets/MixedRealityToolkit/Services/MixedRealityToolkit.cs:934)
Microsoft.MixedReality.Toolkit.MixedRealityToolkit.OnEnable () (at Assets/MixedRealityToolkit/Services/MixedRealityToolkit.cs:620)
I've searched around and most solutions suggest fixing NullReferenceException error by changing the script contents but I am not sure if I am supposed to do that since the scripts are downloaded as a part of Microsoft's Mixed Reality Toolkit. Any ideas on how to fix this problem? I am trying to get the app to run on the Hololens instead of using my PC's Webcam.
I followed the setup in Readme.md, but the error in the picture below will always appear, unable to find 'CameraIntrinsics', I would like to ask which step may be the problem?
Environment I use:
HoloLens 2
Visual Studio 2019
Unity 2019.4.7f1
Microsoft Mixed Reality Toolkit v2.2.0
OpenCV for Unity 2.4.7
Not really an issue but a query. Can this example be used to identify cylinders of different dimensions? If so which file should I have to add/change?
Hi,
I am interested in deploying OpenCV on HL2 for custom object detection.
Is OpenCV available to do that?
If so, are there some reference tutorial videos?
(I notice that OpenCV can detect human faces, and I would like to detect other things, like construction vehicles.)
Thank you so much
I followed the steps to set up and build the project in to hololens. It works but the fps is very low! Around 05 fps.
Is there anything wrong?
Hi,
In the RealTime FaceRecognition Example, it does not use multi-threading. If you have time, please update the RealTime FaceRecognition Example used by HoloLens, thank you.
I am also facing same issues with
trial OpenCvUnity on window 10 and getting same error
Originally posted by @vikashsingh009 in #35 (comment)
Hi.
I want to use HololensWithOpenCVForUnityExample with Hololens2.
It works well in unity editor but when built to Hololens2, camera image of Hololens2 is not be displayed.
(I want to try Comic filter Example.)
I set up according to Readme of this, HololensWithOpenCVForUnityExample.
How can I use Hololens2's camera with HololensWithOpenCVForUnityExample??
In there any special procedure when using Hololens2, not Hololens??
I read #29 and one said we should change code in Videocapture.cs. But Videocapture.cs is not in Camstream folder in HoloLensCameraStream so I think Videocapture.cs is not used when using HololensWithOpenCVForUnityExample and it seems meaningless....
using version:
Hololens2
Unity 2019.4.23f1
Visual Studio 2019
OpenCVForUnity 2.4.4
HololensWithOpenCVForUnityExample 1.0.4
camnewnham/HoloLensCameraStream
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.