qian256 / hololensartoolkit Goto Github PK

Hi Long,

as I mentioned in issue #26 the App only works with .NET scripting backend. Now I would like to integrate your project into another project which has dependencies that require IL2CPP as scripting backend. Is it possible to make the two work togehter somehow?

Best regards

Emulator of HoloLens does not run the sample, probably because the video configuration is fixed to 896x504.

Can you tell me how to use my own Markers?
I am new in developing on the Hololens and need some help.
Thanks.

The creator recommends unity 5.6
while Microsoft insist on using latest version of unity
I tried to compile solution with hiro sample and got

System.IO.DirectoryNotFoundException: cannot find part of path,C:\Users\USER\Documents\Unity Projects\HoloensARToolKit-master\HoloLensARSample\Builds\Hiro\HoloLensARToolKit\GLTFSerializationUWP.dll".
2> w System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
2> w System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
2> w System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
2> w Mono.Cecil.ModuleDefinition.ReadModule(String fileName, ReaderParameters parameters)
2> w Unity.LoadModulesStep.Execute()
2> w Unity.Step.Execute(OperationContext operationContext, IStepContext previousStepContext)
2> w Unity.Operation.Execute()
2> w Unity.Program.Main(String[] args)

请问想自己建立 unitypankage ，使用artoolkit5.0 源码 建立自己的SDK ，请问我应该怎么做哪？ 是建立一个可以在unity软件上使用的sdk？

Hello, excellent library, thank you very much!
I'm walking through the code, I see it calls aruwpUpdate with a frame from the hololens camera to detect any markers present....then calls UpdateTrackingInfo on each marker, which calls the
ARUWP.aruwpQueryMarker to pull the 4x3 matrix from the dll, question, is this matrics
in world coordinates around the player already? Or is there another transform to be done into world space??
How does aruwpUpdate know the worldspace transform to return the markers in? Is there a setup or is more info passed in other than with Detect(p)?
Thanks

你好 ，我使用了您的代码，这在hololens1中表现的很好 ，但是当我尝试将其在hololens2中使用的时候，左上角的片元没有被着色，ar功能也没有开启 。随后我尝试将plugins中的ARTOOLKITUWP.all 改为
arm架构 ，不幸的是，这并没有什么用。我想请问你关于将这个插件适配到hololens2上的计划是怎样的，还有如果方便的话 ，我也想自己对其进行适配，但是我缺少相关的知识，你能告诉我一些modify的途径吗？如果可能的话。

Instead of finding the main camera via name I'd suggest finding it via Tag
holoLensCamera = GameObject.FindWithTag("MainCamera");
Seems like a better practice to me as it's default and grants out of the box compatiblity with the HoloToolkit and own naming conventions.

when I rebuild the scene in my own project, why the camera cannot get work? I made every setting as same as it was done in this project. There is no error in deploy but the camera indicator was not on when running the solution. There are no other access package except HoloToolkit and ARToolKitUWP.unitypackage. Can you help me with this?

Hi,

has someone checked the compatibility with Unity2017.1 and does it work?

Hello, I've tried single scene, and when I start it in my hololens the cube is positioned 5-6 cm above and 5-6 cm closer to me than the marker. I've controlled marker size and settings in app and it's ok: 80mm.

Does this toolkit work with a cylindrical object?

Hi, qian256.
I used your example, which is very cool, but I did not find a way to make the files of *.patt and *.bat (such as hiro. patt, multi-barcode-4x3. Dat).
I found the arpattsave.c file and the arPattSave(...) method, but I didn't find anywhere to use them.
How do I make *.patt files?

Hi,

When the marker is not visible, the virtual displayed cube is anchored to the world as expected. I was now trying to setup a new virtual object in the scene that would have the same behavior as the marker but for some strange reason it is not working.

I created a new virtual object in the Unity scene and in a separate script set the object up in the same way as you set up for the marker: Defined an equivalent to dummyGameObject that has the hololensCamera as a parent as you did in the Setup() inside ARUWPMarker, and then calculate its pose as you do in the Update() function (use the latestTransMatrix to calculate its pose when visible and don't do anything when not visible). The new virtual object then overlays the marker cube when the marker is visible, but unfortunately it just keeps being anchored to the head whenever the marker is not visible.

Any thoughts on why this could be happening? Is there anything else I am missing in the ARUWPMarker script or in another script?

Continuation of discussion from - #8
(background)
WebCamTexture doesn't stop automatically and hence I have been experimenting with it and have added two methods to (ARController.cs script) in my local clone of the repository

 public void stopWebcam()
    {
        if(webcamTexture != null)
        {
            webcamTexture.Stop();
        }
    }

    public void restartWebcam()
    {
        if(webcamTexture == null)
        {
            startWebcam();
        } else
        {
            webcamTexture.Stop();
            webcamTexture.Play();
        }
    }

Below are my observations:

• Calling stopWebcam() does stop the web camera and this works fine the first time.

• When trying to start the camera after stopping it from the code results in no video feed, which I suspect is due to being indirectly related to this issue: Unity/Linux - Unity does not free webcam when stopped

• I also tried to destroy the object that has ARController and ARMarker scripts attached, this does solve the issue but is a hack and definitely not a permanent fix

Hi,
I just started using Hololens ARToolkit and was playing with the samples. I tried the sample for single marker and was looking at Hiro marker to see what happens...nothing happened. I turned my head several times, then was able to see a cube far in the distance, which was there for a second and then disappeared. I ran the code on VS 2017 to see what's happening, and realized the marker is not visible for more that 1 second, although neither me nor the marker move in the interval.
Am I doing something wrong? My unity version is 2018.3.11 and these are the messages I get in the console,back to back:

(Filename: C:\buildslave\unity\build\Runtime/Export/Debug.bindings.h Line: 45)

ARController (native): [info]Marker 0 no longer visible

(Filename: C:\buildslave\unity\build\Runtime/Export/Debug.bindings.h Line: 45)

ARController (native): [info]Marker 0 now visible

Hi,

I have been using the HololensARToolKit a bit more now and noticed that even though the tracking latency is around 30-40 ms and the rendering between 40-50 ms, the total latency between object detection and overlay of the virtual object on top of the real object appears to be around 500 ms to 1 s. I noticed that particularly when moving the head from side to side and noticing that the virtual object lags behind the real one for a quite long time, or when one suddenly moves the object. I was wondering if you had looked into this issue? I guess that there is quite some unaccounted time between acquiring the image and display of the virtual object for that image, besides the rendering and tracking latency you are calculating at the moment?

Would be interesting to see if such time could be reduced, but perhaps the best would have to implement a filter taking into account the head pose from the Hololens (or the IMU data directly once it gets exposed in the API).

Does anybody have the same problem that the virtual object tracked by Hiro marker is not stable. When I remove the marker, the object will stay of course. But when I move my camera to other place and come back, the object disappeared. The object can jump any random place. I don't know why. But the kanji marker tracked is very stable. When I remove the marker, move the camera somewhere and come back, the cube is stay there. Who can tell me why?

Hello Long Qian,

first, thank you so much for developing this kit.
Is it necessary to calibrate each hololense I'm using separately or can I just use the calibration parameters you provided with the kit?

Thank you very much.
Michael

Are there any plans of sharing the calibration code used in https://arxiv.org/abs/1703.05834 with the academic community?

Hi Long,

Thanks for this amazing project and open sourcing it.

I am trying to use markers for defining initial position of holograms using following workflow

1. start the app
2. walk to a stationary marker (on a surface) and scan it
3. use the marker position to place the gameObject / hologram

I have couple of questions regarding this.

• After some quick Google-fu I found out that I can get marker position using ARToolkit (using the low level unity API), is it possible to do the same when using HoloLensARToolkit?
• When running the sample project with single marker tracking (Assets/Sample/HoloLensARToolKitSingle.unity) the frame rate is dropping down to 7-9 fps, is that expected? I haven't made any modifications to the code.
• Is there a way to control the lifecycle of video capture / marker tracking so that it can be triggered based on user input. I am trying to keep the FPS stable and I am assuming that this might be helpful when doing that.

Any other options and suggestions are definitely appreciated.

Hello, I tried to change the .dat files of the project with mine, but it does not seem to work. The .meta files are not recreating.
Can someone help me please?

Library doesn't work in IL2CPP.

I know some methods require [MonoPInvokeCallback(typeof(XXX))] attributes be added to the callback methods passed into native functions.

Not sure what else needs to be changed though.

Hi!
I was wondering what could be the best way to stop the tracking and the execution of HoloLensARToolkit. I am using your code as one part of my application and at some point I need to stop its execution and free the resources, in order to continue with my application (e.g.: I need to access the webcam, but I can not do it if I do not stop HoloLensARToolkit before). Thank you!

Hello!
Thank you for your work on bringing ARToolKit to HoloLens! I just downloaded and tested the sample unity project. It seems to work very well, except there is quite some offset between the marker location and the cube when looking through HoloLens. From reading the other issues, I think that this can probably be resolved by tuning the "magic function", right?
Some questions regarding this:

1. What are the magicMatrix1 and magicMatrix2 doing? What transformations are these? I don't really know how to tune them, since I don't understand what they do? :/
2. Also, if the magic function is taking care of the transformation between RGB camera and the virtual camera, wouldn't it be possible (and much easier) to use "CameraToWorldMatrix" from PhotoCapture class in UnityEngine.XR.WSA.WebCam API instead? (described here: https://docs.microsoft.com/en-us/windows/mixed-reality/locatable-camera) Or is this a different coordinate space tranformation?

I would very much appreciate your help with my questions!
I'm kinda lost to solve the problem with the visual offset and don't know how to make it align...

I am attempting to run the sample scene HoloLensARToolKitSingle and I am encountering a strange effect where tracking performance is extremely bad when the video preview is toggled off.

My HoloLens is currently on 10.0.17763.134 (I have also encountered this on 10.0.17134.345, but have not yet tried other earlier versions). My Unity version is 2018.2.17f1.

I am able to run the HoloLensARToolKitSingle scene successfully: I see the video preview, and when I display PDFs of the Hiro and Kanji markers on my monitor the virtual cubes appear superimposed relatively close to the locations (I will look into your calibration methods in your paper after I've gotten this immediate issue figured out).

When the video preview is on (default), the reported FPS stats are:

• Render: about 50fps
• Video: about 21 fps
• Track: about 20 fps

But when I use the clicker and toggle off the video preview, I get much worse tracking performance!

• Render: about 60 fps
• Video: about 5 fps
• Track: about 4 fps

This seems strange to me; why should the video preview have to be displaying and copying an extra bitmap in order for tracking to be of semi-reasonable performance?

I can tell that the tracking is still happening -- if I move around the PDF windows of the markers on my desktop, the virtual cube positions update -- it's just unworkably slow.

Please let me know if there is anything you can think of to overcome this issue.

If needed I'm willing to downgrade my HoloLens OS version, but I'd rather know beforehand if that has a chance of fixing the issue before going to the trouble.

I know you had mentioned that performance of tracking was degraded with RS4 ( #26 (comment) ). Is this still the case?

I've been reading your code and I can't seem to understand what you are doing with your "magic function". Do you mind sharing or commenting on what is the approach that you are taking on this, and also where are you getting all the values for "magicMatrix1" and "magicMatrix2" from (considering you are hard coding those values in)?

Also, I was reading your blog and you mentioned that all of your code is using left-handed coordinate systems. If I understand correctly you are using "LHMatrixFromRHMatrix" to achieve this, however, in your code this specific function is commented out. Are you ignoring this function and changing the coordinate system in your own way?

As an extra note, I want to point out that when you use arunity all the frames are already changed to work properly inside of Unity and there is no need to modify the marker configuration files for them to be displayed correctly.

Hello,

I have a problem with your sample during live preview from Device Portal (Mixed Reality Capture) - I do not get any video and 3d objects. Are you familiar with that issue? Maybe there is something that I should configure additionally?

Best regards,
Piotr

您好，想请教您一个问题。在运行了您的ARToolKitUWP demo感觉挺不错，但是当我把MainCamera换成HololensCamera的时候，刚进入程序就闪退了，请问这个是什么原因？您知道吗？

For the new HoloLens 2, this will need to be built under ARM32/ARM64

Hi
I am finding ways to get video stream from HL2. I just loaded the demo for HL2 and deployed it to my device. There is no image on top left coner and Video/Track FPS display show "Infinity ms".

Unity asked me if I want to update Asset Database Version from 1.0 to 2.0. I tried both Version 1 and 2. In version 1 preview window return me a white screen and in version 2 there is nothing.

I am not sure why. Or what adjustment I need to do to make it work?

Unity: 2019.3.15f1
MRTK: 2.4 (I also tried to use original Unity project without MRTK)

I have tried both running the samples in unity and running a compiled version to the Hololens. The program runs, but the video preview is just a white box and the the tracking of the patterns does not work. Any thoughts on what could be going wrong?

I have tried using both unity 2017 and unity 5.6.f3. Have also tried both Visual studio 2015 and 2017.

Hi!
First, thank your for open sourcing your awesome code.
I'm absolutely new beginner to Unity, ARToolKit and Hololens.
As for my situation, I have to make hololens detect some object in augmented reality(this object will be the same as marker) and make something appear around or on this marker.
So I thought I have to use opencv at first, but after searching I found your code and decided to use it.

But this is the first time, I had a problem even at the basic step..
The following image is taken on hololens emulator.

I walked through the process for running the sample.
I think I followed all the steps you mentioned.
But I just got the result above.
I think the camera is not working correctly because I got a white box in which there was captured image taken by camera in the sample video.. (my laptop has camera)
Is there any other configuration I should have set?

I think I miss something so basic but I'm truly new to this, so I want you to help me...
I'm waiting for your answer.
Thank you for reading.

Hi,
I've deployed the sample with scene HoloLensARToolKitCube onto hololens, but when I open the app, it crashes after showing the "HoloLensARToolKit Sample scene" title. ( I'm using Unity 5.6.1f1 and visual studio2017)
Any idea about what causes the crash and any suggestion to fix it ?
Thank you so much!!!

Hi, I'm running the cube sample and I've made a printed cube using 'cube00-05-hamming63-a4.pdf'
When I run the example it only track on of the pattern and does not recognise the rest. could you upload a pdf file that you used to print and make your cube shown in the example?
Thanks!

Is there a way to interact with the matched 3D Object and if so how can I do it?
Thanks

Hi,
I noticed that there is a small one-frame judder that happens when the user's head moves. It's noticeable when a marker is kept in the same location and the head rotates back and forth (and it does not appear when the head is kept still and the marker is moved back and forth).

The cause seems to be the OnFrameArrived() function in ARUWPVideo.cs, where the line:

Interlocked.Exchange(ref _cameraToWorldMatrix, cameraToWorldMatrixAsFloat);

is being called too early. It leads to the case where the locatable-camera-to-marker matrix is being set one frame before the world-to-locatable-camera matrix is set, which causes the judder.

In my own tests I tried moving it to just before setting signalTrackingUpdated:

Interlocked.Exchange(ref _cameraToWorldMatrix, cameraToWorldMatrixAsFloat);
signalTrackingUpdated = true;

and the issue seemed to be resolved.

Hi,
I am trying to ensure I have identical development conditions from when you determined your compatibility in your README:

1. Unity 5.6.0f3
2. Visual Studio 2017 (Toolset v141)
3. ARToolKit 5.3.2
4. Windows 10 SDK 10.0.10240.0 to 10.0.14393.0

However, when VS2017.2 changed to VS2017.3, there was an incompatibility that was introducted when targeting UWP projects using .NET. See here for details. A fix was apparently introduced into Unity with 5.6.3.

However, what this means is that if a user tries to use the latest version of VS2017 (note that old versions of VS2017, e.g. VS2017.2, are not made available by Microsoft anymore -- it's a forced upgrade), they will not be able to build the HoloLensARToolKit samples with Unity 5.6.0f3.

I was wondering if you would be able to rerun your code on an updated development environment and determine the current version of Unity and VS2017 that a successful test has been run on.

Hello! Thanks again for all your work on this,
I'm trying to understand the 4x3 matrics which is returned by aruwpUpdate,
aruwpUpdate is passed a ARUint8 *frame, and ends up creating a matrics from the frame in
latestTrackingInfo.trans in Update....
What does the matrics represent? Is it the screen coordinate of the icon in the frame? Then that gets translated to world coordinates?
The matrics is 4x3 and I don't see a lot of zeros, so it must have some meaning other than the screen points where the image was seen?

Simply put, what does the returned tranform from aruwpQueryMarkerTransformation represent? What coordinate system is it in?

Thanks!

Hi Long,

many thanks for this awesome project. I would like to integrate the functionality of this project into another project of mine. However its not working yet.
I did load the asset file and setted everything in my project as in the sample files (creating ARUWPController, attaching scripts, etc.). When I try to build, it gives me this errors:

Assets\Scripts\ARUWPVideo.cs(190,29): error CS1983: The return type of an async method must be void, Task or Task<T>
Assets\Scripts\ARUWPVideo.cs(190,18): error CS0433: The type 'Task<TResult>' exists in both System.Threading.Tasks, Version=4.0.10.0 and System.Threading, Version=1.0.2856.102

The Sample Apps with the same scripts attached are working fine, but somehow if I want to build with my project these scripts produce some errors and I dont know why.
Are there any steps I have to consider when integrating this project into another one?
Many thanks!

I'm running the HoloLenARToolKitSingle demo. I'd say it works fine so far, marker is recognized and it exactly follows any rotation or tilt of the marker. However it is roughly 20-30cm away from the marker. When I hold the marker in my hand and extend the arm it's about halfway between hand and eye.

Is it possible to adjust for that offset and can you advice where and how to do it?

Edit: Hmm it's depending on the distance of the Hololens camera to the marker so there's probably no easy fix.

I use HoloLensARToolKitSingle scene and found that four scripts under object ARUWPController are missing. I can only see ClickerInput.cs.

Please tell me how to resolve that. I download unity 5.6.0f3 to run this project and does not work.

Hi,

when I open the sample project in unity (5.5.1f1) I get the following errors, which i have no idea how to get rid of. I can't find any networking in the scripts. Does someone have a clue?

error1:
UNetWeaver error: Exception :Mono.Cecil.AssemblyResolutionException: Failed to resolve assembly: 'UnityEngine.Networking, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'
at Mono.Cecil.BaseAssemblyResolver.Resolve (Mono.Cecil.AssemblyNameReference name, Mono.Cecil.ReaderParameters parameters) [0x00000] in :0
at Mono.Cecil.BaseAssemblyResolver.Resolve (Mono.Cecil.AssemblyNameReference name) [0x00000] in :0
at Mono.Cecil.DefaultAssemblyResolver.Resolve (Mono.Cecil.AssemblyNameReference name) [0x00000] in :0
at Mono.Cecil.MetadataResolver.Resolve (Mono.Cecil.TypeReference type) [0x00000] in :0
at Mono.Cecil.ModuleDefinition.Resolve (Mono.Cecil.TypeReference type) [0x00000] in :0
at Mono.Cecil.TypeReference.Resolve () [0x00000] in :0
at Unity.UNetWeaver.Weaver.ResolveMethod (Mono.Cecil.TypeReference t, System.String name) [0x00027] in C:\buildslave\unity\build\Extensions\Networking\Weaver\UNetWeaver.cs:1109
at Unity.UNetWeaver.Weaver.SetupTargetTypes () [0x00826] in C:\buildslave\unity\build\Extensions\Networking\Weaver\UNetWeaver.cs:1398
at Unity.UNetWeaver.Weaver.Weave (System.String assName, IEnumerable1 dependencies, IAssemblyResolver assemblyResolver, System.String unityEngineDLLPath, System.String unityUNetDLLPath, System.String outputDir) [0x0004b] in C:\buildslave\unity\build\Extensions\Networking\Weaver\UNetWeaver.cs:1713 at Unity.UNetWeaver.Weaver.WeaveAssemblies (IEnumerable1 assemblies, IEnumerable1 dependencies, IAssemblyResolver assemblyResolver, System.String outputDir, System.String unityEngineDLLPath, System.String unityUNetDLLPath) [0x0006e] in C:\buildslave\unity\build\Extensions\Networking\Weaver\UNetWeaver.cs:1837 UnityEngine.Debug:LogError(Object) Unity.UNetWeaver.Log:Error(String) (at C:/buildslave/unity/build/Extensions/Networking/Weaver/Program.cs:20) Unity.UNetWeaver.Weaver:WeaveAssemblies(IEnumerable1, IEnumerable1, IAssemblyResolver, String, String, String) (at C:/buildslave/unity/build/Extensions/Networking/Weaver/UNetWeaver.cs:1844) Unity.UNetWeaver.Program:Process(String, String, String, String[], String[], IAssemblyResolver, Action1, Action`1) (at C:/buildslave/unity/build/Extensions/Networking/Weaver/Program.cs:34)
UnityEditor.Scripting.Serialization.Weaver:WeaveUnetFromEditor(String, String, String, String, Boolean)

error2:
Failure generating network code.
UnityEditor.Scripting.Serialization.Weaver:WeaveUnetFromEditor(String, String, String, String, Boolean)

Hi, was wondering what is the scaling applied to convert between the object defined in Unity and displayed in the Hololens? For example, if the marker cube is changed to another size, how can I change it in Unity so that it becomes with the correct dimensions when displayed in the Hololens?