ms-iot / ros_msft_mrtk

This repository contains ROS2 bindings for the Mixed Reality Toolkit - allowing you to create VR and AR applications which reflect sensor data from a Robot or provide command and control

Mixed Reality Toolkit for ROS2 Overview

The MRTK for ROS is being developed in the open: Feature requests are very much welcome at the issues page.

The Robot Operating system - also called ROS ¹ - is an open source middleware for creating Robots. This repository implements glue which connects your Mixed Reality Robotics application with Robots and infrastructure using ROS2. ROS2 runs directly on the Hololens without going through ROS Bridge - reducing latency and improving network efficiency. The ROS2 MRTK extension works with ROS2.net² within the Unity Game engine, allowing you to write ROS2-like code directly in Unity.

The Mixed Reality Toolkit is an Open Source framework for building Virtual Reality and Augmented Reality applications, which target numerous platforms - including Hololens.

Getting started with ROS2 on Hololens 2.

This depends on ROS2 Foxy UPM supplied by Microsoft. Windows is required for Hololens and Windows Mixed Reality development. MRTK for ROS2 development will not be supported on other platforms.

Prerequsitites

• Unity 2020.4 or later
• Microsoft Mixed Reality Toolkit
• Nuget for Unity 3.04 or later
• Microsoft Mixed Reality QR Nuget
• Download the Unity release for the ROS2 native from http://aka.ms/ros/mrtk_native
• Download the Unity release for the Mixed Reality Toolit for ROS2 from http://aka.ms/ros/mrtk
• Long Pathes enabled on Windows. Run this command from an administrative command prompt reg add HKLM\SYSTEM\CurrentControlSet\Control\FileSystem /v LongPathsEnabled /t REG_DWORD /d 1 /f'

Create your application

• Create a 3D Unity application

• Use the Microsoft Mixed Reality Feature Tool, and select the following features:

  • Azure Mixed Reality Services
    • Azure Spatial Anchors SDK Core
    • Azure Spatial Anchors SDK for iOS
    • Azure Spatial Anchors SDK for Windows
  • Mixed Reality Toolkit
    • Mixed Reality Toolkit Extensions
    • Mixed Reality Toolkit Foundation
    • Mixed Reality Standard Assets
    • Mixed Reality Toolkit Tools
  • Platform Support
    • Mixed Reality OpenXR Plugin
  • Spatial Audio
    • Microsoft Spatializer
  • World Locking Tools
    • WLT Core

• Wait for Unity to deploy these components.

  • If asked about the new Input System, select Yes.

• When Unity restarts, you should now see a window titled MRTK Project Configurator

  • Select Unity OpenXR Plugin
  • Select Show XR Plug-in Management
    • In the new window, with the monitor icon on the tab, select OpenXR.
    • Switch to the Microsoft icon, ans select OpenXR
    • Close the Unity Configuration Window
  • Return to the MRTK Project Configurator Window and select Apply Settings, then select Next
  • Select Microsoft Spatializer in the Audio Spatializer dropdown, then click Apply

• When Unity Restarts (again), Select the Mixed Reality Menu item, Toolkit, Add to Scene and Configure

• Add Nuget for unity to your application by double clicking the the NuGetForUnity.3.0.2.unitypackage. Import the assets into your project.

• In the Menu for Unity, select NuGet, then Manage Packages.

• In the Search edit box, enter Microsoft.MixedReality.QR, then click search. Install 0.5.3 or later.

• In the Window Menu, select Package Manager.

• Select +, then Add Package from tarball. select the Unity release for ROS2 Native downloaded above.

• Select +, then Add Package from tarball. select the Unity release for the Mixed Reality Toolkit for ROS2 downloaded above.

• In the Project tree view in Unity, expand ROS2 and ROS2.net Native components for Unity and locate the link.xml file. Drag and copy this file into your asset folder.

  • This is needed to prevent optimizations which break ROS2.net Messages.

• Configure your project for Hololens:

  • Select Edit then Project Settings
  • Select Player. Select the Windows icon.
  • Under Capabilities, ensure that the following are checked:
    • InternetClient
    • InternetClientServer
    • PrivateNetworkClientServer
    • WebCam
    • Microphone
    • SpatialPerception
  • Select File Then Build Settings
    • Select Add Open Scenes to add your current scenes to the project.
    • Select Universal Windows Platform and Switch Plaform if needed.
    • Select Build
    • In the save dialog, right click on the background and select New Folder, name it App, ensure App is selected, then click Select Folder
  • Open the resulting Visual Studio Solution in the App folder
  • Configure Visual Studio
    • Select the Release build type. (This is important, as the ROS2 binaries are only delivered as release)
    • Select ARM64.
    • Build and Deploy

You can now add scenes from the ROS MRTK extension or use individual components in your application.

Sample Scene to bootstrap your application

One you have configured the application above, you can now add components to your scene to define your application. A Sample scene is provided which demonstrates spatial pinning and Lidar visualization.

• In the Unity Package Manager Window, select ROS2 and ROS2.net UI components for Unity.
• Under the Samples dropdown, select Basic ROS World and select Import.
• Drag this scene to the Unity Hierarchy Window.
• Remove any existing scenes.

Contributions

Contributions are welcome. Refer to the Contribution Process for more details.

Footnotes

1. ROS is a trademark of Open Robotics. ↩

2. ROS2.net is maintained by the ROS2.net maintainers and community. ↩

ros_msft_mrtk's Issues

Nav2 Integration, including Waypoint visualization

How are people installing this?

Tried to install this recently and encounter a ahem few problems.

Lets start with some of the prerequsitites, namely the ROS2 native and Mixed Reality Toolkit for ROS2 links that just redirect you to GitHub repositories instead of giving you a download.

So after a few hours of trying to work out how to get .tgz files from GitHub repositories, I was finally able to add them to my Unity project. OR SO I THOUGHT, except Unity didn't like it because the package.json files in both repositories weren't in the root directory and Unity couldn't find them. This is two counts for the instructions being incorrect.

Next thing I tried was importing directly from GitHub using Unity's "Add package from git..." option. Funny story, THE SAME THING HAPPENED for both repositories.

So finally I had to resort to a very painful process. I downloaded the zip files for both, dealt with path too long errors despite having longpaths enabled, went into Unity, used the "Add packages from disk..." option and manually navigated to the packages.json files. This actually work to my surprise and I finally had the packages in my project.

And that's how it ended. PSYCHE.

Upon adding this repository I get 40 errors for a bunch of namespaces that don't seem to exist, and when trying to use the ONE IMPORT BUTTON AVAILABLE, I find the folder it's referencing doesn't exist. I even tried to find the link.xml mentioned in the instructions and lo and behold, it doesn't exist either. That's strike three for the instructions.

So yes. I am very VERY curious to know how people are installing and using this software, the process for which has been a worse experience than dealing with government services.

3D objects & the handmenu from the example are not displayed

Hi,
I followed your instructions and wanted to try out Basic ROS World.
In the GameView in Unity and when I run it on the Hololens, the handmenu from the example and other 3D objects are not displayed.
What is the reason for this and does anyone have a solution?

URDF Integration

File paths too long

Describe the bug
Whilst setting up a project using the README instructions I noticed that I had a path name too long error in the console window. This was for the world locking tools package. Aside from that console message which was tempting to ignore I ended up with no MRTK shader in the project and hence no materials on the ui elements. I suspect every time some tooling enumerates that folder it quietly fails and skips some other operations (in this case copying the MRTK shader from it's package).

To Reproduce
Steps to reproduce the behavior:

1. Follow the instructions in the README
2. Create your project in a reasonable-length path location (I used something like c:\users\myname\repos\localprojects\myproject)
3. Look out for console errors and missing materials, etc.

Expected behavior
no errors/missing materials
I remade the project at c:\ROS and it works as expected

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• Unity version: 2020.3.26f1

Additional context
Add any other context about the problem here.

Android Support

MoveIt2 Trajectory Preview

Optimize Lidar Visualization using Unity VFX graph

https://www.youtube.com/watch?v=P5BgrdXis68

Automatic GameObject layout using TF2 graph

Usage Guide

Hi,

Thank you for the hard work that has gone into these projects. I'm attempting to follow the instructions provided to get ROS2 Foxy setup and compiled for mixed reality targeting the Hololens 2. My understanding of the process is as follows:

1. Install ROS2 Foxy for Windows
2. Compile ROS# for UWP
3. Bootstrap the example Unity project (possibly add references to compiled DLL's)

Is this correct or is there something that I am missing? Thank you in advance for any assistance.

Ros node on HoloLens2 cannot be detected.

Dear team,

Thanks a lot for sharing this nice project. I run the Unity project on my laptop and the ros node can be detected by ros2 node list. However, once I deploy the app to HoloLens2, it can not be detected any more. My laptop and Hololens2 are on the same network. Could you please provide some help? Thanks in advance.

I am using MRTK 2.8 + Unity 2021 + ROS2 humble, and the Unity app is built and deployed in Release mode

PS. If I run a ROS node on Hololens2 by ROS-TCP-CONNECTER, the node can be detected.

Implement ROSLog visualizer

Azure Spatial Anchor multiviewer support

Pointcloud 2 Visualization

iOS support

UWP Topics are not shown

I was able to build your system from code based on foxy.
The windows solution works well.

However, I found tons of problems for the UWP solution.
Now at least I am being able to execute a simple node in UWP, however topics are not shown in the ROS2 network.
The only I could do in the node is creating a subscriber and a subscriber inside this same process/node.

I put all the required capabilities: Internet (Client&Server), Internet(Client), Private Network.
Do you have any Idea why this is happening?

Humble Support

How to receive/send audio data and use non-built-in packages

Hi, I want to send and receive audio data between hololens2 and a ros2 port. Is it possible to include packages like audio-common? Or do you have another idea?
Thanks!

Rolling Support

Galactic support

ROS2 node does not get properly destroyed

I was using the example listener script in the package to subscribe for my Lidar pointcloud2 data, while I was able to properly subscribe to it and visualize the points, every time I terminated the application in both Unity editor and hololens2 the node is not being properly terminated and became a zombie node in the background. This can be verified by using "ros2 node list" in the terminal, and these zombie nodes can only be destroyed if I manually close the unity editor/reboot hololens device. Do you know is there any way to cleanly close the application?

p.s. I was also looking at other ros2 Unity packages such as this one: https://github.com/RobotecAI/ros2cs, they do have to call a shutdown function to destroy the nodes properly so my guess is such a function might be missing

Nuget Version

Describe the bug
The version of NugetForUnity specified in the readme https://github.com/ms-iot/ros_msft_mrtk#prerequsitites doesn't work for me. I needed to use 3.0.4

To Reproduce
Steps to reproduce the behavior:

1. Follow the documented setup
2. Search for the QR code package in NugetForUnity UI
3. Press Install
4. See error stating that the package cannot be found

Expected behavior
Package should be found and installed
I tried NugetForUnity v3.0.4 and that worked as expected

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• Unity version: 2020.3.26f1

Additional context
Add any other context about the problem here.