MultiKinect2Grabber packets lost without displaying point cloud? about libfreenect2pclgrabber

[giacomodabisias]

Hi :)
There are a few things I could think of:
Are you linking the libfreenect2 libusb library?
Did you try to build a different processor? (Cuba, opencl,opengl?)
Did you try with more memory for USB buffer?

You can pass two processors as additional parameters
Il 14/Mar/2016 21:28, "Sky Rolnick" [email protected] ha scritto:

Hi,

I've been following your work on the libfreenect2pclgrabber. Its great
work and I'd like to contribute if I can. I have checked out the latest
master branch of your code and of libfreenect2 and successfully built both.
I can run the single Kinect2Grabber and see the visualizer. When I run the
MultiKinect2Grabber, it never gets past the k2g_1.getCloud() line. Instead
I see a lot of packets lots.

I am running the multi-kinect grabber using the following

./MultiKinect2Grabber 3 031862135147 13249453374

[Info] [Freenect2Impl] enumerating devices...
[Info] [Freenect2Impl] 15 usb devices connected
[Info] [Freenect2Impl] found valid Kinect v2 @2 https://github.com/2:7
with serial 031862135147
[Info] [Freenect2Impl] found valid Kinect v2 @2 https://github.com/2:8
with serial 132494533747
[Info] [Freenect2Impl] found 2 devices

Then I see some more messages ...

creating Cpu processor
libva info: VA-API version 0.38.1
libva info: va_getDriverName() returns -1
libva error: va_getDriverName() failed with unknown libva
error,driver_name=(null)
[Error] [VaapiRgbPacketProcessorImpl] vaInitialize(display, &major_ver,
&minor_ver): unknown libva error
[Info] [Freenect2DeviceImpl] opening...

[Error] [usb::TransferPool] failed to submit transfer: LIBUSB_ERROR_IO
Input/Output Error
[Error] [usb::TransferPool] failed to submit transfer: LIBUSB_ERROR_IO
Input/Output Error
[Error] [usb::TransferPool] failed to submit transfer: LIBUSB_ERROR_IO
Input/Output Error
[Error] [usb::TransferPool] failed to submit transfer: LIBUSB_ERROR_IO
Input/Output Error
[Error] [usb::TransferPool] failed to submit transfer: LIBUSB_ERROR_IO
Input/Output Error
[Error] [usb::TransferPool] failed to submit transfer: LIBUSB_ERROR_IO
Input/Output Error
[Error] [usb::TransferPool] failed to submit transfer: LIBUSB_ERROR_IO
Input/Output Error
[Error] [usb::TransferPool] failed to submit transfer: LIBUSB_ERROR_IO
Input/Output Error
[Error] [usb::TransferPool] failed to submit transfer: LIBUSB_ERROR_IO
Input/Output Error
[Info] [Freenect2DeviceImpl] transfer pool sizes rgb: 20_16384 ir: 60_
8*33792

And then I see that many packets are lost.

[Info] [DepthPacketStreamParser] 30 packets were lost
[Info] [DepthPacketStreamParser] 32 packets were lost
[Info] [DepthPacketStreamParser] 33 packets were lost
[Info] [TurboJpegRgbPacketProcessor] avg. time: 18.1056ms -> ~55.2315Hz

I have modified my usbfs_memory_mb as well.

more /sys/module/usbcore/parameters/usbfs_memory_mb
64

I should also mention I am using Ubuntu 14.04 and a recent build of the
master branch of pcl.
Protonect runs fine on my system.

Just wondering if you encountered this and what your solution was?

Thanks!

—
Reply to this email directly or view it on GitHub
#17.

[skymeson]

Are you linking the libfreenect2 libusb library?

I'm not sure if I have the right libusb libraries or not. I tried the following.

cd /home/skymeson/kinect/libfreenect2/depends
sudo ./instal_libusb.sh

When I check which usblib is being used by default I see the following.
ldd MultiKinect2Grabber | grep usb
libusb-1.0.so.0 => /lib/x86_64-linux-gnu/libusb-1.0.so.0 (0x00007f4f39b3d000)

I also tried this.

LD_LIBRARY_PATH=/home/skymeson/kinect/libfreenect2/depends/libusb/lib ./MultiKinect2Grabber 3 031862135147 132494533747

Still getting same errors.

Did you try to build a different processor? (Cuda, opencl,opengl?)

I'm guessing you would do this by modifying the CMakeLists.txt file?

option(WITH_CUDA "adds cuda support for freenect2" OFF)
to
option(WITH_CUDA "adds cuda support for freenect2" ON)

builds ok:)

option(WITH_OPENCL "adds opencl support for freenect2" OFF)
to
option(WITH_OPENCL "adds opencl support for freenect2" ON)

I get a build error.

[ 33%] Building CXX object CMakeFiles/Kinect2Grabber.dir/test.cpp.o
In file included from /home/skymeson/kinect/libfreenect2pclgrabber/test.cpp:19:0:
/home/skymeson/kinect/libfreenect2pclgrabber/k2g.h: In constructor ‘K2G::K2G(processor, bool, std::string)’:
/home/skymeson/kinect/libfreenect2pclgrabber/k2g.h:76:46: error: expected type-specifier
dev_ = freenect2_.openDefaultDevice(new libfreenect2::OpenCLPacketPipeline());
^
/home/skymeson/kinect/libfreenect2pclgrabber/k2g.h:78:47: error: expected type-specifier
dev_ = freenect2_.openDevice(serial, new libfreenect2::OpenCLPacketPipeline());
^
make[2]: *** [CMakeFiles/Kinect2Grabber.dir/test.cpp.o] Error 1
make[1]: *** [CMakeFiles/Kinect2Grabber.dir/all] Error 2
make: *** [all] Error 2

I may not have OpenCL installed properly on my system.

Now when I run the following, I at least see a window pop up but I see no point cloud.

./MultiKinect2Grabber 2 031862135147 132494533747

When I try the following, I see no window but fewer packets are being lost now.

./MultiKinect2Grabber 3 031862135147 132494533747

Kinect #1 serial 031862135147 Kinect #2 serial 132494533747
getting cloud
[Info] [DepthPacketStreamParser] 7 packets were lost
[Info] [CudaDepthPacketProcessor] avg. time: 1.2077ms -> ~828.02Hz
[Info] [TurboJpegRgbPacketProcessor] avg. time: 16.1827ms -> ~61.7944Hz
[Info] [DepthPacketStreamParser] 1 packets were lost
[Info] [DepthPacketStreamParser] 1 packets were lost
[Info] [CudaDepthPacketProcessor] avg. time: 1.16965ms -> ~854.957Hz
[Info] [TurboJpegRgbPacketProcessor] avg. time: 16.4651ms -> ~60.7345Hz
[Info] [CudaDepthPacketProcessor] avg. time: 1.17064ms -> ~854.231Hz

I'm not sure I understood your suggestions fully though. Is that what you meant, or should I try something else?

Thanks!

[giacomodabisias]

Hi,
the libusb version seems ok.
What kernel version are you using?

Don't edit the cmake file but use ccmake to set options :) This way you can build the grabber with opencv/cuda/opengl based on what you used whe building libfreenect2 (you should use ccmake also to build that).

Can you open both kinect2s separately?
an possible issue is that both kinects are on the same usb3 bus and the bus capacity is not enough. Try to put them on separate board is possible (usb3 ports far away from each other)

[skymeson]

Yeah, I realized I sort of hacked my way through the CMakeLists.txt without using ccmake. I was able to get it to compile but it may have been missing something in the configuration.

Now I use ccmake and set build options for latest libfreenect2 directory (/home/skymeson/freenect2/include) and the latest pcl 1.8 directory (/usr/local/include/pcl-1.8)

It builds fine but I get seg faults when I run the MultiKinect2Grabber. The single Kinect2Grabber seems to work fine. I've tested each individual kinect withKinect2Grabber and they both work fine. I can see 2 devices listed and can get their valid serial #.

[Info] [Freenect2Impl] found valid Kinect v2 @2:12 with serial 013415745147
[Info] [Freenect2Impl] found valid Kinect v2 @2:13 with serial 053961435147
[Info] [Freenect2Impl] found 2 devices

I get seg faults regardless of what mode I am in, 0, 1, 2, 3.
./MultiKinect2Grabber 3 013415745147 053961435147

I'm thinking it must have something to do with not finding libusb-1.0, but not sure exactly why? Do I have the configuration correct?

ls /usr/include/libusb-1.0/
libusb.h
ls /usr/lib/x86_64-linux-gnu/libusb-1.0.so
/usr/lib/x86_64-linux-gnu/libusb-1.0.so

LIBUSB_1_INCLUDE_DIR /usr/include
LIBUSB_1_LIBRARY /usr/lib/x86_64-linux-gnu/libusb-1.0.so
OPENCV_FOUND ON
RSSDK_DIR RSSDK_DIR NOTFOUND
RSSDK_LIBRARY RSSDK_LIBRARY-NOTFOUND
WITH_CUDA ON
WITH_OPENCL ON
WITH_SERIALIZATION OFF

When I configure I see a few different warning messages and also notice an issue with my libusb-1.0 so it may be disabled.

** WARNING ** io features related to ensenso will be disabled

** WARNING ** io features related to davidSDK will be disabled

** WARNING ** io features related to dssdk will be disabled

** WARNING ** io features related to pcap will be disabled

** WARNING ** io features related to png will be disabled

** WARNING ** io features related to libusb-1.0 will be disabled

** WARNING ** visualization features related to ensenso will be disabled

** WARNING ** visualization features related to davidSDK will be disabled

** WARNING ** visualization features related to dssdk will be disabled

** WARNING ** visualization features related to rssdk will be disabled

I've also tried toggling WITH_SERIALIZATION but no luck.

Is there something I am missing with my libusb-1.0 setup? That could explain the segfaults.

Thanks!

[giacomodabisias]

segfaults look odd. Try adding some std::cout in the test.cpp code to see where it happens

[skymeson]

I modified the following lines.

std::cout << "Initialize multi K2G: " << std::endl;
K2G k2g_1(freenectprocessor, argc == 5 ? true : false, argv[2]); //"005157353647"
std::cout << "Kinect with processorID " << argv[2] << " initialized." << endl;
K2G k2g_2(freenectprocessor, argc == 5 ? true : false, argv[3]); //"500320441942"
std::cout << "Kinect with processorID " << argv[3] << " initialized." << endl;

The output before it segfaults is

skymeson@skymeson:~/kinect/libfreenect2pclgrabber/libfreenect2pclgrabber/build$ ./run_multi_kinect.sh
Syntax is: ./MultiKinect2Grabber [-processor 0|1|2] -processor options 0,1,2,3 correspond to CPU, OPENCL, OPENGL, CUDA respectively
followed by the two kinect serials
Initialize multi K2G:
[Info] [Freenect2Impl] enumerating devices...
[Info] [Freenect2Impl] 15 usb devices connected
[Info] [Freenect2Impl] found valid Kinect v2 @2:12 with serial 013415745147
[Info] [Freenect2Impl] found valid Kinect v2 @2:13 with serial 053961435147
[Info] [Freenect2Impl] found 2 devices
creating Cuda processor
[Info] [CudaDepthPacketProcessorImpl] device 0: GeForce GTX 980M @ 1126MHz Memory 4087MB
[Info] [CudaDepthPacketProcessorImpl] selected device 0
libva info: VA-API version 0.38.1
libva info: va_getDriverName() returns -1
libva error: va_getDriverName() failed with unknown libva error,driver_name=(null)
[Error] [VaapiRgbPacketProcessorImpl] vaInitialize(display, &major_ver, &minor_ver): unknown libva error
[Info] [Freenect2DeviceImpl] opening...
[Info] [Freenect2DeviceImpl] transfer pool sizes rgb: 20_16384 ir: 60_8*33792
[Info] [Freenect2DeviceImpl] opened
[Info] [Freenect2DeviceImpl] starting...
[Info] [Freenect2DeviceImpl] submitting rgb transfers...
[Info] [Freenect2DeviceImpl] submitting depth transfers...
[Info] [Freenect2DeviceImpl] started
[Error] [RgbPacketStreamParser] packetsize or sequence doesn't match!
Kinect with processorID 013415745147 initialized.
[Info] [Freenect2Impl] enumerating devices...
[Info] [Freenect2Impl] 15 usb devices connected
[Info] [Freenect2Impl] found valid Kinect v2 @2:12 with serial 013415745147
[Info] [Freenect2Impl] found valid Kinect v2 @2:13 with serial 053961435147
[Info] [Freenect2Impl] found 2 devices
creating Cuda processor
[Info] [CudaDepthPacketProcessorImpl] device 0: GeForce GTX 980M @ 1126MHz Memory 4087MB
[Info] [CudaDepthPacketProcessorImpl] selected device 0
libva info: VA-API version 0.38.1
libva info: va_getDriverName() returns -1
libva error: va_getDriverName() failed with unknown libva error,driver_name=(null)
[Error] [VaapiRgbPacketProcessorImpl] vaInitialize(display, &major_ver, &minor_ver): unknown libva error
[Info] [Freenect2DeviceImpl] opening...
[Error] [protocol::UsbControl] failed to set configuration! LIBUSB_ERROR_OTHER Other error. Try debugging with environment variable: export LIBUSB_DEBUG=3 .
[Info] [Freenect2DeviceImpl] closing...
[Error] [protocol::CommandTransaction] bulk transfer failed: LIBUSB_ERROR_IO Input/Output Error
[Error] [protocol::CommandTransaction] bulk transfer failed: LIBUSB_ERROR_IO Input/Output Error
[Info] [Freenect2DeviceImpl] deallocating usb transfer pools...
[Info] [Freenect2DeviceImpl] closing usb device...
[Info] [Freenect2DeviceImpl] closed
[Error] [Freenect2Impl] failed to open Kinect v2: @2:13
./run_multi_kinect.sh: line 1: 19529 Segmentation fault (core dumped) ./MultiKinect2Grabber 3 013415745147 053961435147

I think that means it is failing on this line.
K2G k2g_1(freenectprocessor, argc == 5 ? true : false, argv[2]); //"005157353647"

[giacomodabisias]

failed to set configuration! LIBUSB_ERROR_OTHER Other error. Try debugging with environment variable: export LIBUSB_DEBUG=3
The error seems to come from libusb
I solved checking here:
OpenKinect/libfreenect2#458
have a look and let me know

[giacomodabisias]

Also if possible as already mentioned try different usb ports

[skymeson]

Yeah, tried that. Both kinects work fine in the single mode. I've tried on different usb ports with same results. My computer has 4 usb3.0 ports and tried them all. Each time it recognizes 2 devices found.

[giacomodabisias]

Yeah but on the same bus they could saturate the bandwidth

[skymeson]

Okay, so perhaps there is nothing I can do on this machine then. I would need to test on a different machine with separate usb bus.

Is there a way to know if this is what is happening for certain? Its a fairly new laptop but possible that usb share same bus.

Should I be concerned that build tells me
** WARNING ** io features related to libusb-1.0 will be disabled

[giacomodabisias]

What laptop is it?
No that is only a warning.

[skymeson]

This is probably not your issue.

I just tried with a modified Protonect.cpp from libfreenect2 and I get the same segfault.

That means it is either libfreenect2 issue or my laptop usb bus bandwidth. Probably going to go with my laptop bus is the culprit.

Laptop I'm using is a asus rog g751jy. It has 4 x USB 3.0, 24 Gb ram, and 4 Gb Nvidia GTX 980M, if that means anything to you.

I think I'm just limited to one kinect per machine for now until I can test on a different machine.

You can close the issue if you like.

Thanks for all your help!

[giacomodabisias]

Ok, seems a rock solid machine. I will try with my laptop to check a bit the issue. If there are any news let me know.

Bests

[cforntu]

Excuse me, I met almost the same problem. After I change the option during ccmake to :
WITH_CUDA ON
WITH_OPENCL OFF
WITH_PCL ON
WITH_ROS OFF
and when I enter the command "make" the output is
[ 50%] Building CXX object CMakeFiles/Kinect2Grabber.dir/test.cpp.o
In file included from /home/ccc/pcl/libfreenect2pclgrabber/test.cpp:19:0:
/home/ccc/pcl/libfreenect2pclgrabber/include/k2g.h: In constructor ‘K2G::K2G(Processor, bool, std::string)’:
/home/ccc/pcl/libfreenect2pclgrabber/include/k2g.h:101:46: error: expected type-specifier
dev_ = freenect2_.openDefaultDevice(new libfreenect2::CudaPacketPipeline());
^
/home/ccc/pcl/libfreenect2pclgrabber/include/k2g.h:103:47: error: expected type-specifier
dev_ = freenect2_.openDevice(serial, new libfreenect2::CudaPacketPipeline());
^
make[2]: *** [CMakeFiles/Kinect2Grabber.dir/test.cpp.o] Error 1
make[1]: *** [CMakeFiles/Kinect2Grabber.dir/all] Error 2
make: *** [all] Error 2
Could you please tell me what is means by "expected type-specifier"

[giacomodabisias]

Do you have the last libfreenect2 version? try to make a git pull in the libfreenect2 folder and reinstall the library

[cforntu]

I'm sorry that I tried the git pull and reinstall the library but the error is still there. Could you give me more suggestions.

[cforntu]

I tried the dmesg command and there is nothing to to with the bandwidth and here are some messages from the log
[30226.115866] xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff880084a2aad8
[30226.115867] xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff880084a2aa90
[30226.178124] xhci_hcd 0000:00:14.0: WARN Event TRB for slot 30 ep 2 with no TDs queued?
[30879.902837] ipheth 1-3:4.2: ipheth_rcvbulk_callback: urb status: -71
[30879.902942] usb 1-3: USB disconnect, device number 13

[giacomodabisias]

Did you build libfreenect2 with cuda support?

[cforntu]

I built libfreenect2 with cuda support and here are the options I chose while
~/Desktop/install/libfreenect2/build/ccmake ..

and there are still errors there when I build libfreenect2pclgrabber

ccc@ccc-Z170X-UD3:~/pcl/libfreenect2pclgrabber/build$

An interesting thing is that when I run Protonect under /Desktop/install/libfreenect2/build/bin$ ./Protonect
I could get the image but the output showed that there are many packets lost.
[Info] [Freenect2Impl] found valid Kinect v2 @2:5 with serial 004535452747
[Info] [Freenect2Impl] found valid Kinect v2 @2:4 with serial 003399454047
[Info] [Freenect2Impl] found 2 devices
libva info: VA-API version 0.39.0
libva info: va_getDriverName() returns -1
libva error: va_getDriverName() failed with unknown libva error,driver_name=(null)
[Error] [VaapiRgbPacketProcessorImpl] vaInitialize(display, &major_ver, &minor_ver): unknown libva error
[Info] [Freenect2DeviceImpl] opening...
[Info] [Freenect2DeviceImpl] transfer pool sizes rgb: 20_16384 ir: 60_8*33792
[Info] [Freenect2DeviceImpl] opened
[Info] [Freenect2DeviceImpl] starting...
[Debug] [Freenect2DeviceImpl] status 0x090000: 9855
[Debug] [Freenect2DeviceImpl] status 0x090000: 9855
[Info] [Freenect2DeviceImpl] submitting rgb transfers...
[Info] [Freenect2DeviceImpl] submitting depth transfers...
[Info] [Freenect2DeviceImpl] started
device serial: 004535452747
device firmware: 4.0.3911.0
[Debug] [DepthPacketStreamParser] not all subsequences received 0
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Info] [DepthPacketStreamParser] 14 packets were lost
Is there any relationship between these two issues. I appreciate your expertise, Thanks.

[giacomodabisias]

To me it looks like you didn't compile libfreenect2 with cuda support. Can you run the Protonect with the cuda packet processor?

[cforntu]

Excuse me, I am new to kinect, could you please tell me how to run the Protonect with the cuda packet processor.

[giacomodabisias]

./Protonect cuda
then check in the output if there is written something like this:
[Info] [CudaDepthPacketProcessorImpl] device 0: GeForce GTX 960 @ 1291MHz Memory 2044MB
[Info] [CudaDepthPacketProcessorImpl] selected device 0

[cforntu]

Thank you for your reply. I tried the command and I got the similar output.
[Info] [CudaDepthPacketProcessorImpl] device 0: GeForce GTX 980 Ti @ 1240MHz Memory 6140MB
[Info] [CudaDepthPacketProcessorImpl] selected device 0

[cforntu]

But there are still many packets lost.

[cforntu]

Hello, when I tried dmesg after the the "make" command, I saw the following messages:
[ 43.748904] usb 2-5.1: Not enough bandwidth for new device state.
[ 43.748905] usb 2-5.1: Not enough bandwidth for altsetting 1
[ 43.749288] usb 2-5.1: 3:1: usb_set_interface failed (-28)
[ 43.749935] usb 2-5.1: Not enough bandwidth for new device state.
[ 43.749936] usb 2-5.1: Not enough bandwidth for altsetting 1
[ 43.750321] usb 2-5.1: 3:1: usb_set_interface failed (-28)
and
[10018.021877] usb 2-5.1: reset SuperSpeed USB device number 9 using xhci_hcd
[10018.097418] xhci_hcd 0000:00:14.0: WARN Event TRB for slot 14 ep 2 with no TDs queued?
[10018.499213] usb 2-5.1: reset SuperSpeed USB device number 9 using xhci_hcd
[10018.521291] usb 2-5.1: usbfs: process 4232 (USB) did not claim interface 1 before use
[10018.521326] usb 2-5.1: usbfs: process 4232 (USB) did not claim interface 0 before use
[10018.521338] usb 2-5.1: usbfs: process 4232 (USB) did not claim interface 0 before use
[10018.521348] usb 2-5.1: usbfs: process 4232 (USB) did not claim interface 0 before use
[10018.521358] usb 2-5.1: usbfs: process 4232 (USB) did not claim interface 0 before use
[10018.522227] usb 2-5.1: usbfs: process 4232 (USB) did not claim interface 0 before use
[10018.522251] usb 2-5.1: usbfs: process 4232 (USB) did not claim interface 0 before use
[10018.522263] usb 2-5.1: usbfs: process 4232 (USB) did not claim interface 0 before use
I'm sorry for the previous messages saying that there is no problem with the bandwidth.
Could you please tell me how to solve this?

[giacomodabisias]

Hi,
you have a compilation problem. That could be a run time problem. If you have read the readme I have written the solution to that problem:

execute "echo 64 > /sys/module/usbcore/parameters/usbfs_memory_mb". If this works you can apply this patch permanently as follows:

Open the /etc/default/grub file in any text editor. Find and replace: GRUB_CMDLINE_LINUX_DEFAULT="quiet splash" with this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash usbcore.usbfs_memory_mb=1000"

Update grub with these settings: $ sudo update-grub

[cforntu]

Hello, I am sorry to bother you again. The error is still there but never mind, I can use the single Kinect2 instead. Could you please tell me how to export and save the raw depth data grabbed by the kinect sensor?

[giacomodabisias]

You need to compile the libraryWITH_SERIALIZATION.
and then call enableSerialization() on the grabber.
This will store all cloud data in the local folder

2016-07-01 4:50 GMT+02:00 cforntu [email protected]:

Hello, I am sorry to bother you again. The error is still there but never
mind, I can use the single Kinect2 instead. Could you please tell me how to
export and save the raw depth data grabbed by the kinect sensor?

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
#17 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AFm4zBOXIErsxhGO-LxdHJym7q_6f6n7ks5qRICOgaJpZM4HwcML
.

Eng G. Dabisias, PhD Student
Percro Lab, (Laboratory of Perceptual Robotics)
Scuola Superiore Sant'Anna
via Luigi Alamanni 13D, San Giuliano Terme 56010 (PI), Italy
mob.: [phone]
skype: giacomo.dabisias2

[cforntu]

Thank you for your reply, and how about the rgb and the IR data

[cforntu]

Hello sir,
could you tell me whether this grabber could save the IR image.
Regards,

[giacomodabisias]

Hi,
I did not add that code but you can just add a function which is almost the same as the get depth function but asks for the IR image

void getDepth(cv::Mat depth_mat){
        listener_.waitForNewFrame(frames_);
        libfreenect2::Frame * depth = frames_[libfreenect2::Frame::Depth];

        cv::Mat depth_tmp(depth->height, depth->width, CV_32FC1, depth->data);

        if(mirror_ == true){
            cv::flip(depth_tmp, depth_mat, 1);
        }else
        {
            depth_mat = depth_tmp.clone();
        }
        listener_.release(frames_);
    }

Instead of [libfreenect2::Frame::Depth] you have to ask for IR.