mrcfschr / organizar_mar_demo

demo video

Pipeline Overview.pdf

HoloLens 2 Sensor Streaming

HoloLens 2 server software and Python client library for streaming sensor data via TCP. Created to stream HoloLens data in real time over WiFi to a Linux machine for research purposes but also works on Windows and macOS. The server is offered as a standalone application (appxbundle) or Unity plugin (dll).

Supported interfaces

• Research Mode Visible Light Cameras (640x480 @ 30 FPS, Grayscale, H264 or HEVC encoded)
  • Left Front
  • Left Left
  • Right Front
  • Right Right
• Research Mode Depth
  • AHAT (512x512 @ 45 FPS, 16-bit Depth + 16-bit AB, H264 or HEVC encoded or Lossless* Zdepth for Depth)
  • Long Throw (320x288 @ 5 FPS, 16-bit Depth + 16-bit AB, PNG encoded)
• Research Mode IMU
  • Accelerometer (m/s^2)
  • Gyroscope (deg/s)
  • Magnetometer
• Front Camera (1920x1080 @ 30 FPS, RGB, H264 or HEVC encoded)
• Microphone (2 channels @ 48000 Hz, 16-bit PCM, AAC encoded or 5 channels @ 48000 Hz, 32-bit Float)
• Spatial Input (30 Hz)
  • Head Tracking
  • Eye Tracking
  • Hand Tracking
• Spatial Mapping (3D Meshes)
• Scene Understanding (3D Meshes + Semantic labels for planar surfaces)
• Voice Input
• Extended Eye Tracking (30, 60, or 90 FPS)
• Extended Audio (Microphone + Application audio, 2 channels @ 48000 Hz, 16-bit PCM, AAC encoded)
  • Internal Microphone mirror
  • External USB-C Microphone
• Extended Video
  • Front Camera mirror (no Mixed Reality Capture)
  • External USB-C Camera

Additional features

• Download calibration data (e.g., camera intrinsics, extrinsics, undistort maps) for the Front Camera and Research Mode sensors (except RM IMU Magnetometer).
• Optional per-frame pose for the Front Camera and Research Mode sensors.
• Support for Mixed Reality Capture (Holograms in Front Camera video) and Shared capture.
• Client can configure the bitrate and properties of the H264, HEVC, and AAC encoded streams.
• Client can configure the resolution and framerate of the Front Camera. See here for a list of supported configurations.
• Client can configure the focus, white balance, and exposure of the Front Camera. See here.
• Frame timestamps can be converted to Windows FILETIME (UTC) for external synchronization. See here.
• Client can send messages to a Unity application using the plugin.
• Server application can run in background (alongside other applications) when running in flat mode.
• C++ client library.
• MATLAB client (MEX).
• hl2da plugin: access sensor data from Unity, Unreal, and native UWP apps running on the HoloLens.

Technical Report

Our paper provides an overview of the code, features, and examples for the first released version of the application (1.0.11.0). For newer versions, please refer to the examples in the viewer directory. If hl2ss is useful for your research, please cite our report:

@article{dibene2022hololens,
  title={HoloLens 2 Sensor Streaming},
  author={Dibene, Juan C and Dunn, Enrique},
  journal={arXiv preprint arXiv:2211.02648},
  year={2022}
}

Preparation

Before using the server software configure your HoloLens as follows:

1. Update your HoloLens: Settings -> Update & Security -> Windows Update.
2. Enable developer mode: Settings -> Update & Security -> For developers -> Use developer features.
3. Enable device portal: Settings -> Update & Security -> For developers -> Device Portal.
4. Enable research mode: Refer to the Enabling Research Mode section in HoloLens Research Mode.

Please note that enabling Research Mode on the HoloLens increases battery usage.

Installation (sideloading)

The server application is distributed as a single appxbundle file and can be installed using one of the two following methods.

Method 1 (local)

1. On your HoloLens, open Microsoft Edge and navigate to this repository.
2. Download the latest appxbundle.
3. Open the appxbundle and tap Install.

Method 2 (remote)

1. Download the latest appxbundle.
2. Go to the Device Portal and navigate to Views -> Apps. Under Deploy apps, select Local Storage, click Browse, and select the appxbundle.
3. Click Install, wait for the installation to complete, then click Done.

You can find the server application (hl2ss) in the All apps list.

Permissions

The first time the server runs it will ask for the necessary permissions to access sensor data. If there are any issues please verify that the server application (hl2ss.exe) has access to:

• Camera (Settings -> Privacy -> Camera).
• Eye tracker (Settings -> Privacy -> Eye tracker).
• Microphone (Settings -> Privacy -> Microphone).
• User movements (Settings -> Privacy -> User movements).

Python client

The Python scripts in the viewer directory demonstrate how to connect to the server, receive the data, unpack it, and decode it in real time. Additional samples show how to associate data from multiple streams. Run the server on your HoloLens and set the host variable of the Python scripts to your HoloLens IP address.

Interfaces

• RM VLC: viewer/client_stream_rm_vlc.py
• RM Depth AHAT: viewer/client_stream_rm_depth_ahat.py
• RM Depth Long Throw: viewer/client_stream_rm_depth_longthrow.py
• RM IMU: viewer/client_stream_rm_imu.py
• Front Camera: viewer/client_stream_pv.py
• Microphone: viewer/client_stream_microphone.py
• Spatial Input: viewer/client_stream_si.py
• Remote Configuration: viewer/client_ipc_rc.py
• Spatial Mapping: viewer/client_ipc_sm.py
• Scene Understanding: viewer/client_ipc_su.py
• Voice Input: viewer/client_ipc_vi.py
• Unity Message Queue: viewer/client_ipc_umq.py (Plugin Only)
• Extended Eye Tracking: viewer/client_stream_eet.py
• Extended Audio: viewer/client_stream_extended_audio.py
• Extended Video: viewer/client_stream_extended_video.py

Required packages

• OpenCV pip install opencv-python
• PyAV pip install av
• NumPy pip install numpy

Optional packages

• pynput pip install pynput
• Open3D pip install open3d
• PyAudio pip install PyAudio

Unity plugin

For streaming sensor data from a Unity application. A sample Unity project (2020.3.42f1) can be found in the hl2ss_unity directory.

Build and run the sample project

1. Open the project in Unity. If the MRTK Project Configurator window pops up just close it.
2. Go to Build Settings (File -> Build Settings).
3. Switch to Universal Windows Platform.
4. Set Target Device to HoloLens.
5. Set Architecture to ARM64.
6. Set Build and Run on Remote Device (via Device Portal).
7. Set Device Portal Address to your HoloLens IP address (e.g., https://192.168.1.7) and set your Device Portal Username and Password.
8. Click Build and Run. Unity may ask for a Build folder. You can create a new one named Build.

Adding the plugin to an existing project

1. Download the latest plugin zip file and extract the Assets folder into your Unity project folder.
2. In the Unity Editor configure the hl2ss, Eye Tracking, and Scene Understanding DLLs as UWP ARM64.
   1. In the Project window navigate to Assets/Plugins/WSA, select the DLL, and then go to the Inspector window.
   2. Set SDK to UWP.
   3. Set CPU to ARM64.
   4. Click Apply.
3. Add the Hololens2SensorStreaming.cs script to the Main Camera.
4. Enable the following capabilities (Edit -> Project Settings -> Player -> Publishing Settings):
   • InternetClientServer
   • InternetClient
   • PrivateNetworkClientServer
   • Webcam
   • Microphone
   • Spatial Perception
   • Gaze Input
5. The plugin also requires the perceptionSensorsExperimental and backgroundSpatialPerception capabilities, which are not available in the Publishing Settings capabilities list. The Editor folder in the plugin zip file contains a script (BuildPostProcessor.cs) that adds the capabilities automatically after building the project. Just extract the Editor folder into the Assets folder of your Unity project. Alternatively, you can manually edit the Package.appxmanifest after building. See here for an example.

Remote Unity Scene

The plugin has basic support for creating and controlling 3D primitives and text objects via TCP for the purpose of sending feedback to the HoloLens user. See the unity_sample Python scripts in the viewer directory for some examples. Some of the supported features include:

• Create primitive: sphere, capsule, cylinder, cube, plane, and quad.
• Set active: enable or disable game object.
• Set world transform: position, rotation, and scale.
• Set local transform: position, rotation, and scale w.r.t. Main Camera.
• Set color: rgba with support for semi-transparency.
• Set texture: upload png or jpg file.
• Create text: creates a TextMeshPro object.
• Set text: sets the text, font size and color of a TextMeshPro object.
• Text to speech: upload text.
• Remove: destroy game object.
• Remove all: destroy all game objects created by the plugin.

To enable this functionality add the RemoteUnityScene.cs script to the Main Camera and set the Material field to BasicMaterial.

Unreal plugin

For streaming sensor data from an Unreal application. A sample Unreal project (4.27.2) can be found in the hl2ss_unreal directory.

Build and run the sample project

1. Open the project in Unreal and rebuild all missing modules. Ignore the Level_BuiltData error.
2. Open Project Settings (Edit -> Project Settings). Navigate to Platforms -> HoloLens. Under Packaging -> Signing Certificate, click Generate New. In the Create Private Key Password window that appears click None. Close Project Settings.
3. Package the project for HoloLens (File -> Package Project -> HoloLens). Unreal may ask for a destination folder. You can create a new one named Package.
4. Install the hl2ss_unreal.appxbundle (generated in the Package/HoloLens folder) on your HoloLens.
5. Run the hl2ss unreal app (located in the All apps list).

Adding the plugin to an existing project

1. Download the latest plugin zip file and extract the Plugins folder into your Unreal project folder.
2. Enable the hl2ss plugin (Edit -> Plugins). Restart the Editor if prompted.
3. Add "hl2ss" to PublicDependencyModuleNames in the project .Build.cs.
4. Enable the following capabilities (Edit -> Project Settings -> Platforms -> HoloLens):
   • Internet Client
   • Internet Client Server
   • Private Network Client Server
   • Microphone
   • Webcam
   • Gaze Input
   • Spatial Perception
5. Add +DeviceCapabilityList=backgroundSpatialPerception to Config/HoloLens/HoloLensEngine.ini (see here for an example).

Build from source and deploy

Building the server application and the plugin requires a Windows 10 machine.

1. Install the tools.
2. Open the Visual Studio solution (sln file in the hl2ss folder) in Visual Studio 2022.
3. Set build configuration to Release ARM64. Building for x86 and x64 (HoloLens emulator), and ARM is not supported.
4. Right click the hl2ss project and select Properties. Navigate to Configuration Properties -> Debugging and set Machine Name to your HoloLens IP address.
5. Build (Build -> Build Solution). If you get an error saying that hl2ss.winmd does not exist, copy the hl2ss.winmd file from etc into the hl2ss\ARM64\Release\hl2ss folder.
6. Run (Remote Machine). You may need to pair your HoloLens first.

The server application will remain installed on the HoloLens even after power off. The plugin is in the hl2ss\ARM64\Release\plugin folder. If you wish to create the server application appxbundle, right click the hl2ss project and select Publish -> Create App Packages.

Known issues and limitations

• Multiple streams can be active at the same time but only one client per stream is allowed.
• Occasionally, the server might crash when accessing the Front Camera and RM Depth Long Throw streams simultaneously. See microsoft/HoloLens2ForCV#142.
• The RM Depth AHAT and RM Depth Long Throw streams cannot be accessed simultaneously.
• Spatial Input is not supported in flat mode.

References

This project uses the HoloLens 2 Research Mode API and the Cannon library, both available at the HoloLens2ForCV repository. Lossless* depth compression enabled by the Zdepth library.

organizar_mar_demo's Issues

still getting to many detections after filtering torch.Size([4, 118508])

73]]
occurance count tensor([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31],
device='cuda:0', dtype=torch.int32)
occurance thres value tensor(6, device='cuda:0', dtype=torch.int32)
after filtering torch.Size([4, 118508])
num_ins_points_after_filtering tensor([2943, 2670, 478, 540], device='cuda:0')
DEBUG torch.Size([4, 118508]) [tensor(1), tensor(0), tensor(2), tensor(2)]
0
tensor([1])

if grid[i,j] == 0: IndexError: index 100 is out of bounds for axis 1 with size 100

DEBUG Before Pooling 0.02396082878112793
DEBUG Pooling takes time 0.09056544303894043
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.5692, dtype=torch.float64)
DEBB (21539, 2)
DEBUG visibility mask 5499
DEBUG calculating visibility 0.002176523208618164
DEBUG get final masks 0.0009484291076660156
DEBUG whole function time 0.11830520629882812
number of 3d mask points: [4792] number of 2d masks: [93236]
9%|█████▍ | 5/55 [00:00<00:06, 8.04it/s]-------------------------frame 5 -------------------------
DEBUG Before projection 0.010017633438110352
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (21980, 2)
within_mask shape: (1, 21980)
(9413, 2)
DEBUG lowest depth 1000.0 0.58408165785756
DEBUG Before Pooling 0.031357765197753906
DEBUG Pooling takes time 0.09698843955993652
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.5841, dtype=torch.float64)
DEBB (21980, 2)
DEBUG visibility mask 5337
DEBUG calculating visibility 0.003312349319458008
DEBUG get final masks 0.001657247543334961
DEBUG whole function time 0.13464879989624023
number of 3d mask points: [4790] number of 2d masks: [90158]
11%|██████▌ | 6/55 [00:00<00:06, 7.61it/s]-------------------------frame 6 -------------------------
DEBUG Before projection 0.009968757629394531
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (19024, 2)
within_mask shape: (1, 19024)
(11294, 2)
DEBUG lowest depth 1000.0 0.571054776868206
DEBUG Before Pooling 0.031726837158203125
DEBUG Pooling takes time 0.05864310264587402
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.5711, dtype=torch.float64)
DEBB (19024, 2)
DEBUG visibility mask 6838
DEBUG calculating visibility 0.0020432472229003906
DEBUG get final masks 0.0008783340454101562
DEBUG whole function time 0.09383773803710938
number of 3d mask points: [5896] number of 2d masks: [118560]
13%|███████▋ | 7/55 [00:00<00:05, 8.13it/s]-------------------------frame 7 -------------------------
DEBUG Before projection 0.00921010971069336
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (17190, 2)
within_mask shape: (1, 17190)
(9155, 2)
DEBUG lowest depth 1000.0 0.4790997796252028
DEBUG Before Pooling 0.020475387573242188
DEBUG Pooling takes time 0.09473991394042969
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.4791, dtype=torch.float64)
DEBB (17190, 2)
DEBUG visibility mask 5755
DEBUG calculating visibility 0.002029895782470703
DEBUG get final masks 0.0008814334869384766
DEBUG whole function time 0.11882472038269043
number of 3d mask points: [4945] number of 2d masks: [119630]
15%|████████▋ | 8/55 [00:00<00:05, 8.01it/s]-------------------------frame 8 -------------------------
DEBUG Before projection 0.01035618782043457
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (10302, 2)
within_mask shape: (1, 10302)
(3940, 2)
DEBUG lowest depth 1000.0 0.5237359526563959
DEBUG Before Pooling 0.02102041244506836
DEBUG Pooling takes time 0.05348396301269531
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.5237, dtype=torch.float64)
DEBB (10302, 2)
DEBUG visibility mask 4139
DEBUG calculating visibility 0.002334117889404297
DEBUG get final masks 0.001134634017944336
DEBUG whole function time 0.07872176170349121
number of 3d mask points: [3339] number of 2d masks: [111018]
-------------------------frame 9 -------------------------
DEBUG Before projection 0.011635780334472656
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (11734, 2)
within_mask shape: (1, 11734)
(3128, 2)
DEBUG lowest depth 1000.0 0.43410286555715594
DEBUG Before Pooling 0.012127876281738281
DEBUG Pooling takes time 0.07071137428283691
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.4341, dtype=torch.float64)
DEBB (11734, 2)
DEBUG visibility mask 3311
DEBUG calculating visibility 0.002080678939819336
DEBUG get final masks 0.0009095668792724609
DEBUG whole function time 0.08660340309143066
number of 3d mask points: [2645] number of 2d masks: [100078]
18%|██████████▋ | 10/55 [00:01<00:04, 9.04it/s]-------------------------frame 10 -------------------------
DEBUG Before projection 0.0015628337860107422
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (19229, 2)
within_mask shape: (1, 19229)
(6738, 2)
DEBUG lowest depth 1000.0 0.21643779011472275
DEBUG Before Pooling 0.018944263458251953
DEBUG Pooling takes time 0.04829549789428711
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.2164, dtype=torch.float64)
DEBB (19229, 2)
DEBUG visibility mask 4830
DEBUG calculating visibility 0.002758502960205078
DEBUG get final masks 0.0012826919555664062
DEBUG whole function time 0.0720663070678711
number of 3d mask points: [4350] number of 2d masks: [93316]
-------------------------frame 11 -------------------------
DEBUG Before projection 0.002200603485107422
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (21292, 2)
within_mask shape: (1, 21292)
(5250, 2)
DEBUG lowest depth 1000.0 0.2884138195764257
DEBUG Before Pooling 0.015159368515014648
DEBUG Pooling takes time 0.08796453475952148
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.2884, dtype=torch.float64)
DEBB (21292, 2)
DEBUG visibility mask 4665
DEBUG calculating visibility 0.002557039260864258
DEBUG get final masks 0.001047372817993164
DEBUG whole function time 0.10738444328308105
number of 3d mask points: [3917] number of 2d masks: [79855]
22%|████████████▊ | 12/55 [00:01<00:04, 9.48it/s]-------------------------frame 12 -------------------------
DEBUG Before projection 0.004877805709838867
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (15480, 2)
within_mask shape: (1, 15480)
(7074, 2)
DEBUG lowest depth 1000.0 0.2804977707347812
DEBUG Before Pooling 0.017422914505004883
DEBUG Pooling takes time 0.07146549224853516
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.2805, dtype=torch.float64)
DEBB (15480, 2)
DEBUG visibility mask 5383
DEBUG calculating visibility 0.002557516098022461
DEBUG get final masks 0.0012166500091552734
DEBUG whole function time 0.09366869926452637
number of 3d mask points: [4754] number of 2d masks: [98740]
-------------------------frame 13 -------------------------
DEBUG Before projection 0.010124444961547852
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (14574, 2)
within_mask shape: (1, 14574)
(7016, 2)
DEBUG lowest depth 1000.0 0.27882038849008933
DEBUG Before Pooling 0.01720452308654785
DEBUG Pooling takes time 0.06715941429138184
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.2788, dtype=torch.float64)
DEBB (14574, 2)
DEBUG visibility mask 5276
DEBUG calculating visibility 0.0024957656860351562
DEBUG get final masks 0.0012173652648925781
DEBUG whole function time 0.08896493911743164
number of 3d mask points: [4674] number of 2d masks: [99493]
25%|███████████████ | 14/55 [00:01<00:04, 9.67it/s]-------------------------frame 14 -------------------------
DEBUG Before projection 0.0019071102142333984
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (13641, 2)
within_mask shape: (1, 13641)
(6779, 2)
DEBUG lowest depth 1000.0 0.2752623842545029
DEBUG Before Pooling 0.018152952194213867
DEBUG Pooling takes time 0.08957862854003906
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.2753, dtype=torch.float64)
DEBB (13641, 2)
DEBUG visibility mask 4999
DEBUG calculating visibility 0.0019419193267822266
DEBUG get final masks 0.0008246898651123047
DEBUG whole function time 0.11109495162963867
number of 3d mask points: [4483] number of 2d masks: [98471]
27%|████████████████ | 15/55 [00:01<00:04, 9.46it/s]-------------------------frame 15 -------------------------
DEBUG Before projection 0.010465621948242188
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (16644, 2)
within_mask shape: (1, 16644)
(7456, 2)
DEBUG lowest depth 1000.0 0.5620193761065477
DEBUG Before Pooling 0.024161815643310547
DEBUG Pooling takes time 0.07147693634033203
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.5620, dtype=torch.float64)
DEBB (16644, 2)
DEBUG visibility mask 5167
DEBUG calculating visibility 0.0023915767669677734
DEBUG get final masks 0.0010521411895751953
DEBUG whole function time 0.0997765064239502
number of 3d mask points: [4512] number of 2d masks: [106723]
29%|█████████████████▏ | 16/55 [00:01<00:04, 9.34it/s]-------------------------frame 16 -------------------------
DEBUG Before projection 0.008830547332763672
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (10712, 2)
within_mask shape: (1, 10712)
(5485, 2)
DEBUG lowest depth 1000.0 0.5593881174414153
DEBUG Before Pooling 0.014219522476196289
DEBUG Pooling takes time 0.10079622268676758
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.5594, dtype=torch.float64)
DEBB (10712, 2)
DEBUG visibility mask 6481
DEBUG calculating visibility 0.002121448516845703
DEBUG get final masks 0.0011744499206542969
DEBUG whole function time 0.1192631721496582
number of 3d mask points: [4503] number of 2d masks: [128357]
31%|██████████████████▏ | 17/55 [00:01<00:04, 8.88it/s]-------------------------frame 17 -------------------------
DEBUG Before projection 0.009118795394897461
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (11673, 2)
within_mask shape: (1, 11673)
(886, 2)
DEBUG lowest depth 1000.0 1.6559878165581818
DEBUG Before Pooling 0.00614166259765625
DEBUG Pooling takes time 0.08616375923156738
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(1.6560, dtype=torch.float64)
DEBB (11673, 2)
DEBUG visibility mask 927
DEBUG calculating visibility 0.001905679702758789
DEBUG get final masks 0.0008389949798583984
DEBUG whole function time 0.09574460983276367
number of 3d mask points: [651] number of 2d masks: [9717]
33%|███████████████████▎ | 18/55 [00:02<00:04, 9.01it/s]-------------------------frame 18 -------------------------
DEBUG Before projection 0.013391733169555664
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (8544, 2)
within_mask shape: (1, 8544)
(6040, 2)
DEBUG lowest depth 1000.0 0.4436077305503897
DEBUG Before Pooling 0.02183222770690918
DEBUG Pooling takes time 0.06266069412231445
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.4436, dtype=torch.float64)
DEBB (8544, 2)
DEBUG visibility mask 4339
DEBUG calculating visibility 0.0024595260620117188
DEBUG get final masks 0.001039266586303711
DEBUG whole function time 0.08896327018737793
number of 3d mask points: [4083] number of 2d masks: [157458]
35%|████████████████████▍ | 19/55 [00:02<00:03, 9.17it/s]-------------------------frame 19 -------------------------
DEBUG Before projection 0.0017843246459960938
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (8455, 2)
within_mask shape: (1, 8455)
(6007, 2)
DEBUG lowest depth 1000.0 0.44310142439316513
DEBUG Before Pooling 0.013291120529174805
DEBUG Pooling takes time 0.08449935913085938
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.4431, dtype=torch.float64)
DEBB (8455, 2)
DEBUG visibility mask 4356
DEBUG calculating visibility 0.0019347667694091797
DEBUG get final masks 0.0010089874267578125
DEBUG whole function time 0.10172247886657715
number of 3d mask points: [4094] number of 2d masks: [158518]
36%|█████████████████████▍ | 20/55 [00:02<00:03, 9.27it/s]-------------------------frame 20 -------------------------
DEBUG Before projection 0.010192632675170898
DEBUG (110421, 2) (3, 360, 640)
inbound_shape: (10695, 2)
within_mask shape: (3, 10695)
(5558, 2)
DEBUG lowest depth 1000.0 0.41424859483546916
DEBUG Before Pooling 0.016143798828125
DEBUG Pooling takes time 0.0678560733795166
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.4142, dtype=torch.float64)
DEBB (10695, 2)
DEBUG visibility mask 5943
DEBUG calculating visibility 0.00212860107421875
DEBUG get final masks 0.0022742748260498047
DEBUG whole function time 0.08954644203186035
number of 3d mask points: [2118 2058 3974] number of 2d masks: [ 90213 88582 125756]
38%|██████████████████████▌ | 21/55 [00:02<00:03, 9.43it/s]-------------------------frame 21 -------------------------
DEBUG Before projection 0.003696441650390625
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (15649, 2)
within_mask shape: (1, 15649)
(7608, 2)
DEBUG lowest depth 1000.0 0.49864375718683696
DEBUG Before Pooling 0.03847789764404297
DEBUG Pooling takes time 0.08025741577148438
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.4986, dtype=torch.float64)
DEBB (15649, 2)
DEBUG visibility mask 4357
DEBUG calculating visibility 0.002032756805419922
DEBUG get final masks 0.0008835792541503906
DEBUG whole function time 0.12238001823425293
number of 3d mask points: [3999] number of 2d masks: [117564]
40%|███████████████████████▌ | 22/55 [00:02<00:03, 8.91it/s]-------------------------frame 22 -------------------------
DEBUG Before projection 0.01019597053527832
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (14140, 2)
within_mask shape: (1, 14140)
(6975, 2)
DEBUG lowest depth 1000.0 0.43070284880135784
DEBUG Before Pooling 0.019932985305786133
DEBUG Pooling takes time 0.09044051170349121
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.4307, dtype=torch.float64)
DEBB (14140, 2)
DEBUG visibility mask 4523
DEBUG calculating visibility 0.0022864341735839844
DEBUG get final masks 0.0011093616485595703
DEBUG whole function time 0.1144411563873291
number of 3d mask points: [3993] number of 2d masks: [124024]
42%|████████████████████████▋ | 23/55 [00:02<00:03, 8.60it/s]-------------------------frame 23 -------------------------
DEBUG Before projection 0.009198188781738281
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (14828, 2)
within_mask shape: (1, 14828)
(5785, 2)
DEBUG lowest depth 1000.0 0.5747968276349804
DEBUG Before Pooling 0.03376889228820801
DEBUG Pooling takes time 0.09518074989318848
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.5748, dtype=torch.float64)
DEBB (14828, 2)
DEBUG visibility mask 5180
DEBUG calculating visibility 0.0022776126861572266
DEBUG get final masks 0.0011284351348876953
DEBUG whole function time 0.13306927680969238
number of 3d mask points: [4466] number of 2d masks: [112895]
44%|█████████████████████████▋ | 24/55 [00:02<00:03, 8.04it/s]-------------------------frame 24 -------------------------
DEBUG Before projection 0.00807642936706543
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (9392, 2)
within_mask shape: (1, 9392)
(4932, 2)
DEBUG lowest depth 1000.0 0.14045408278662533
DEBUG Before Pooling 0.013437747955322266
DEBUG Pooling takes time 0.09040474891662598
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.1405, dtype=torch.float64)
DEBB (9392, 2)
DEBUG visibility mask 4250
DEBUG calculating visibility 0.001697540283203125
DEBUG get final masks 0.0007815361022949219
DEBUG whole function time 0.10690879821777344
number of 3d mask points: [3846] number of 2d masks: [111322]
45%|██████████████████████████▊ | 25/55 [00:02<00:03, 8.20it/s]-------------------------frame 25 -------------------------
DEBUG Before projection 0.009161949157714844
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (9015, 2)
within_mask shape: (1, 9015)
(1196, 2)
DEBUG lowest depth 1000.0 0.5140668383525226
DEBUG Before Pooling 0.007866859436035156
DEBUG Pooling takes time 0.07358455657958984
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.5141, dtype=torch.float64)
DEBB (9015, 2)
DEBUG visibility mask 762
DEBUG calculating visibility 0.002032756805419922
DEBUG get final masks 0.001108407974243164
DEBUG whole function time 0.08803868293762207
number of 3d mask points: [574] number of 2d masks: [25611]
-------------------------frame 26 -------------------------
DEBUG Before projection 0.0058286190032958984
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (11423, 2)
within_mask shape: (1, 11423)
(6649, 2)
DEBUG lowest depth 1000.0 0.14174714826801632
DEBUG Before Pooling 0.01754164695739746
DEBUG Pooling takes time 0.07451605796813965
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.1417, dtype=torch.float64)
DEBB (11423, 2)
DEBUG visibility mask 4676
DEBUG calculating visibility 0.0021047592163085938
DEBUG get final masks 0.0009217262268066406
DEBUG whole function time 0.09593892097473145
number of 3d mask points: [4229] number of 2d masks: [113796]
49%|████████████████████████████▉ | 27/55 [00:03<00:03, 8.85it/s]-------------------------frame 27 -------------------------
DEBUG Before projection 0.011690139770507812
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (11246, 2)
within_mask shape: (1, 11246)
(6556, 2)
DEBUG lowest depth 1000.0 0.13899582613807174
DEBUG Before Pooling 0.020305633544921875
DEBUG Pooling takes time 0.05367541313171387
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.1390, dtype=torch.float64)
DEBB (11246, 2)
DEBUG visibility mask 4694
DEBUG calculating visibility 0.004099607467651367
DEBUG get final masks 0.0016450881958007812
DEBUG whole function time 0.08128762245178223
number of 3d mask points: [4268] number of 2d masks: [114399]
-------------------------frame 28 -------------------------
DEBUG Before projection 0.007021427154541016
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (11206, 2)
within_mask shape: (1, 11206)
(6747, 2)
DEBUG lowest depth 1000.0 0.1411047703589184
DEBUG Before Pooling 0.01813030242919922
DEBUG Pooling takes time 0.0928030014038086
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.1411, dtype=torch.float64)
DEBB (11206, 2)
DEBUG visibility mask 4706
DEBUG calculating visibility 0.0021796226501464844
DEBUG get final masks 0.0011582374572753906
DEBUG whole function time 0.1152181625366211
number of 3d mask points: [4373] number of 2d masks: [116036]
53%|███████████████████████████████ | 29/55 [00:03<00:02, 8.95it/s]-------------------------frame 29 -------------------------
DEBUG Before projection 0.004651069641113281
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (9263, 2)
within_mask shape: (1, 9263)
(385, 2)
DEBUG lowest depth 1000.0 0.5087860458685247
DEBUG Before Pooling 0.004960060119628906
DEBUG Pooling takes time 0.07980775833129883
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.5088, dtype=torch.float64)
DEBB (9263, 2)
DEBUG visibility mask 282
DEBUG calculating visibility 0.002179384231567383
DEBUG get final masks 0.0010504722595214844
DEBUG whole function time 0.08874964714050293
number of 3d mask points: [218] number of 2d masks: [13696]
-------------------------frame 30 -------------------------
DEBUG Before projection 0.005127429962158203
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (11429, 2)
within_mask shape: (1, 11429)
(6951, 2)
DEBUG lowest depth 1000.0 0.14857363214054897
DEBUG Before Pooling 0.013332366943359375
DEBUG Pooling takes time 0.064453125
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.1486, dtype=torch.float64)
DEBB (11429, 2)
DEBUG visibility mask 4852
DEBUG calculating visibility 0.002452373504638672
DEBUG get final masks 0.0012001991271972656
DEBUG whole function time 0.08234286308288574
number of 3d mask points: [4437] number of 2d masks: [116049]
56%|█████████████████████████████████▎ | 31/55 [00:03<00:02, 9.55it/s]-------------------------frame 31 -------------------------
DEBUG Before projection 0.0017282962799072266
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (24703, 2)
within_mask shape: (1, 24703)
(7932, 2)
DEBUG lowest depth 1000.0 0.8143798478406982
DEBUG Before Pooling 0.0349421501159668
DEBUG Pooling takes time 0.07274341583251953
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.8144, dtype=torch.float64)
DEBB (24703, 2)
DEBUG visibility mask 3789
DEBUG calculating visibility 0.002514362335205078
DEBUG get final masks 0.001065969467163086
DEBUG whole function time 0.11227560043334961
number of 3d mask points: [2909] number of 2d masks: [72857]
58%|██████████████████████████████████▎ | 32/55 [00:03<00:02, 9.34it/s]-------------------------frame 32 -------------------------
DEBUG Before projection 0.004822731018066406
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (19123, 2)
within_mask shape: (1, 19123)
(6445, 2)
DEBUG lowest depth 1000.0 0.7871566076297801
DEBUG Before Pooling 0.020070791244506836
DEBUG Pooling takes time 0.08843231201171875
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.7872, dtype=torch.float64)
DEBB (19123, 2)
DEBUG visibility mask 1403
DEBUG calculating visibility 0.0022542476654052734
DEBUG get final masks 0.000978708267211914
DEBUG whole function time 0.11229467391967773
number of 3d mask points: [1112] number of 2d masks: [50288]
60%|███████████████████████████████████▍ | 33/55 [00:03<00:02, 9.09it/s]-------------------------frame 33 -------------------------
DEBUG Before projection 0.009968280792236328
DEBUG (110421, 2) (4, 360, 640)
inbound_shape: (24469, 2)
within_mask shape: (4, 24469)
(11872, 2)
DEBUG lowest depth 1000.0 1.1390172371242695
DEBUG Before Pooling 0.032949209213256836
DEBUG Pooling takes time 0.06800532341003418
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(1.1390, dtype=torch.float64)
DEBB (24469, 2)
DEBUG visibility mask 3372
DEBUG calculating visibility 0.0033216476440429688
DEBUG get final masks 0.0038552284240722656
DEBUG whole function time 0.10911679267883301
number of 3d mask points: [ 600 2173 1824 969] number of 2d masks: [ 9556 63127 53554 30584]
62%|████████████████████████████████████▍ | 34/55 [00:03<00:02, 8.86it/s]-------------------------frame 34 -------------------------
DEBUG Before projection 0.006492137908935547
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (27617, 2)
within_mask shape: (1, 27617)
(10627, 2)
DEBUG lowest depth 1000.0 1.3050651205090658
DEBUG Before Pooling 0.02699589729309082
DEBUG Pooling takes time 0.06447672843933105
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(1.3051, dtype=torch.float64)
DEBB (27617, 2)
DEBUG visibility mask 2743
DEBUG calculating visibility 0.003755331039428711
DEBUG get final masks 0.0016429424285888672
DEBUG whole function time 0.10183501243591309
number of 3d mask points: [2131] number of 2d masks: [51146]
64%|█████████████████████████████████████▌ | 35/55 [00:03<00:02, 8.92it/s]-------------------------frame 35 -------------------------
DEBUG Before projection 0.005423784255981445
DEBUG (110421, 2) (3, 360, 640)
inbound_shape: (26974, 2)
within_mask shape: (3, 26974)
(12054, 2)
DEBUG lowest depth 1000.0 1.2758236909538445
DEBUG Before Pooling 0.032683610916137695
DEBUG Pooling takes time 0.06321859359741211
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(1.2758, dtype=torch.float64)
DEBB (26974, 2)
DEBUG visibility mask 3781
DEBUG calculating visibility 0.002564668655395508
DEBUG get final masks 0.0023674964904785156
DEBUG whole function time 0.10149908065795898
number of 3d mask points: [ 934 2103 1691] number of 2d masks: [13065 54246 44379]
65%|██████████████████████████████████████▌ | 36/55 [00:04<00:02, 8.99it/s]-------------------------frame 36 -------------------------
DEBUG Before projection 0.00901484489440918
DEBUG (110421, 2) (2, 360, 640)
inbound_shape: (26912, 2)
within_mask shape: (2, 26912)
(10481, 2)
DEBUG lowest depth 1000.0 1.2778260361696159
DEBUG Before Pooling 0.03102397918701172
DEBUG Pooling takes time 0.07111716270446777
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(1.2778, dtype=torch.float64)
DEBB (26912, 2)
DEBUG visibility mask 2722
DEBUG calculating visibility 0.003500699996948242
DEBUG get final masks 0.002439737319946289
DEBUG whole function time 0.10907340049743652
number of 3d mask points: [2096 1568] number of 2d masks: [54077 42483]
67%|███████████████████████████████████████▋ | 37/55 [00:04<00:02, 8.80it/s]-------------------------frame 37 -------------------------
DEBUG Before projection 0.008684396743774414
DEBUG (110421, 2) (2, 360, 640)
inbound_shape: (27121, 2)
within_mask shape: (2, 27121)
(10472, 2)
DEBUG lowest depth 1000.0 1.2899229778889052
DEBUG Before Pooling 0.03079366683959961
DEBUG Pooling takes time 0.053566932678222656
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(1.2899, dtype=torch.float64)
DEBB (27121, 2)
DEBUG visibility mask 2723
DEBUG calculating visibility 0.0032949447631835938
DEBUG get final masks 0.002137899398803711
DEBUG whole function time 0.09078621864318848
number of 3d mask points: [2102 1704] number of 2d masks: [53414 44756]
69%|████████████████████████████████████████▊ | 38/55 [00:04<00:01, 9.09it/s]-------------------------frame 38 -------------------------
DEBUG Before projection 0.0022029876708984375
DEBUG (110421, 2) (2, 360, 640)
inbound_shape: (27013, 2)
within_mask shape: (2, 27013)
(10357, 2)
DEBUG lowest depth 1000.0 1.2846881033637327
DEBUG Before Pooling 0.05275249481201172
DEBUG Pooling takes time 0.045641183853149414
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(1.2847, dtype=torch.float64)
DEBB (27013, 2)
DEBUG visibility mask 2685
DEBUG calculating visibility 0.0026578903198242188
DEBUG get final masks 0.002019166946411133
DEBUG whole function time 0.10381793975830078
number of 3d mask points: [1614 2040] number of 2d masks: [43383 52935]
71%|█████████████████████████████████████████▊ | 39/55 [00:04<00:01, 9.14it/s]-------------------------frame 39 -------------------------
DEBUG Before projection 0.004935503005981445
DEBUG (110421, 2) (2, 360, 640)
inbound_shape: (26075, 2)
within_mask shape: (2, 26075)
(10608, 2)
DEBUG lowest depth 1000.0 1.2129342150648028
DEBUG Before Pooling 0.033954620361328125
DEBUG Pooling takes time 0.0586550235748291
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(1.2129, dtype=torch.float64)
DEBB (26075, 2)
DEBUG visibility mask 3308
DEBUG calculating visibility 0.0029866695404052734
DEBUG get final masks 0.002106189727783203
DEBUG whole function time 0.09833574295043945
number of 3d mask points: [1764 876] number of 2d masks: [52500 12861]
73%|██████████████████████████████████████████▉ | 40/55 [00:04<00:01, 9.26it/s]-------------------------frame 40 -------------------------
DEBUG Before projection 0.010252952575683594
DEBUG (110421, 2) (2, 360, 640)
inbound_shape: (23319, 2)
within_mask shape: (2, 23319)
(7683, 2)
DEBUG lowest depth 1000.0 0.9780230205001833
DEBUG Before Pooling 0.02130413055419922
DEBUG Pooling takes time 0.05342364311218262
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.9780, dtype=torch.float64)
DEBB (23319, 2)
DEBUG visibility mask 2813
DEBUG calculating visibility 0.0049114227294921875
DEBUG get final masks 0.0026121139526367188
DEBUG whole function time 0.08306550979614258
number of 3d mask points: [ 602 1629] number of 2d masks: [ 8585 55912]
-------------------------frame 41 -------------------------
DEBUG Before projection 0.0014753341674804688
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (22675, 2)
within_mask shape: (1, 22675)
(2819, 2)
DEBUG lowest depth 1000.0 1.1105391392919925
DEBUG Before Pooling 0.010732173919677734
DEBUG Pooling takes time 0.049860477447509766
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(1.1105, dtype=torch.float64)
DEBB (22675, 2)
DEBUG visibility mask 1955
DEBUG calculating visibility 0.003199338912963867
DEBUG get final masks 0.0013921260833740234
DEBUG whole function time 0.06601905822753906
number of 3d mask points: [1762] number of 2d masks: [24506]
76%|█████████████████████████████████████████████ | 42/55 [00:04<00:01, 10.40it/s]-------------------------frame 42 -------------------------
DEBUG Before projection 0.009032726287841797
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (22858, 2)
within_mask shape: (1, 22858)
(2813, 2)
DEBUG lowest depth 1000.0 1.104957325477601
DEBUG Before Pooling 0.011041402816772461
DEBUG Pooling takes time 0.05358004570007324
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(1.1050, dtype=torch.float64)
DEBB (22858, 2)
DEBUG visibility mask 1967
DEBUG calculating visibility 0.003232717514038086
DEBUG get final masks 0.001255035400390625
DEBUG whole function time 0.06992101669311523
number of 3d mask points: [1766] number of 2d masks: [24730]
-------------------------frame 43 -------------------------
DEBUG Before projection 0.008940458297729492
DEBUG (110421, 2) (3, 360, 640)
inbound_shape: (22915, 2)
within_mask shape: (3, 22915)
(5049, 2)
DEBUG lowest depth 1000.0 1.0228207453394873
DEBUG Before Pooling 0.015105724334716797
DEBUG Pooling takes time 0.05568289756774902
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(1.0228, dtype=torch.float64)
DEBB (22915, 2)
DEBUG visibility mask 1455
DEBUG calculating visibility 0.0031456947326660156
DEBUG get final masks 0.0031490325927734375
DEBUG whole function time 0.07786846160888672
number of 3d mask points: [ 823 1180 1191] number of 2d masks: [25503 36823 37135]
80%|███████████████████████████████████████████████▏ | 44/55 [00:04<00:01, 10.87it/s]-------------------------frame 44 -------------------------
DEBUG Before projection 0.003187417984008789
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (22956, 2)
within_mask shape: (1, 22956)
(414, 2)
DEBUG lowest depth 1000.0 1.1928852429559709
DEBUG Before Pooling 0.0037360191345214844
DEBUG Pooling takes time 0.09848928451538086
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(1.1929, dtype=torch.float64)
DEBB (22956, 2)
DEBUG visibility mask 349
DEBUG calculating visibility 0.002815723419189453
DEBUG get final masks 0.0012552738189697266
DEBUG whole function time 0.10693573951721191
number of 3d mask points: [200] number of 2d masks: [4596]
-------------------------frame 45 -------------------------
DEBUG Before projection 0.00949406623840332
DEBUG (110421, 2) (2, 360, 640)
inbound_shape: (21546, 2)
within_mask shape: (2, 21546)
(6357, 2)
DEBUG lowest depth 1000.0 0.9751478024545849
DEBUG Before Pooling 0.01864337921142578
DEBUG Pooling takes time 0.0688931941986084
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.9751, dtype=torch.float64)
DEBB (21546, 2)
DEBUG visibility mask 1401
DEBUG calculating visibility 0.0027298927307128906
DEBUG get final masks 0.0020599365234375
DEBUG whole function time 0.09303903579711914
number of 3d mask points: [1187 1162] number of 2d masks: [41848 40939]
84%|█████████████████████████████████████████████████▎ | 46/55 [00:05<00:00, 10.14it/s]-------------------------frame 46 -------------------------
DEBUG Before projection 0.004500865936279297
DEBUG (110421, 2) (2, 360, 640)
inbound_shape: (16302, 2)
within_mask shape: (2, 16302)
(2673, 2)
DEBUG lowest depth 1000.0 1.1222483881070424
DEBUG Before Pooling 0.013808727264404297
DEBUG Pooling takes time 0.04833579063415527
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(1.1222, dtype=torch.float64)
DEBB (16302, 2)
DEBUG visibility mask 1893
DEBUG calculating visibility 0.002696514129638672
DEBUG get final masks 0.0019791126251220703
DEBUG whole function time 0.06751036643981934
number of 3d mask points: [975 648] number of 2d masks: [20519 18074]
-------------------------frame 47 -------------------------
DEBUG Before projection 0.010534524917602539
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (19841, 2)
within_mask shape: (1, 19841)
(5196, 2)
DEBUG lowest depth 1000.0 0.9124770043967604
DEBUG Before Pooling 0.019891977310180664
DEBUG Pooling takes time 0.05407428741455078
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.9125, dtype=torch.float64)
DEBB (19841, 2)
DEBUG visibility mask 1419
DEBUG calculating visibility 0.00321197509765625
DEBUG get final masks 0.0013437271118164062
DEBUG whole function time 0.07928156852722168
number of 3d mask points: [1163] number of 2d masks: [44071]
87%|███████████████████████████████████████████████████▍ | 48/55 [00:05<00:00, 10.72it/s]-------------------------frame 48 -------------------------
DEBUG Before projection 0.00838017463684082
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (10283, 2)
within_mask shape: (1, 10283)
(1954, 2)
DEBUG lowest depth 1000.0 0.8311924935207939
DEBUG Before Pooling 0.011318206787109375
DEBUG Pooling takes time 0.05311775207519531
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.8312, dtype=torch.float64)
DEBB (10283, 2)
DEBUG visibility mask 896
DEBUG calculating visibility 0.0028362274169921875
DEBUG get final masks 0.0011515617370605469
DEBUG whole function time 0.06956958770751953
number of 3d mask points: [688] number of 2d masks: [35287]
-------------------------frame 49 -------------------------
DEBUG Before projection 0.010664939880371094
DEBUG (110421, 2) (2, 360, 640)
inbound_shape: (12288, 2)
within_mask shape: (2, 12288)
(1257, 2)
DEBUG lowest depth 1000.0 0.8363851112666612
DEBUG Before Pooling 0.00600123405456543
DEBUG Pooling takes time 0.09581279754638672
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.8364, dtype=torch.float64)
DEBB (12288, 2)
DEBUG visibility mask 493
DEBUG calculating visibility 0.0018525123596191406
DEBUG get final masks 0.0015060901641845703
DEBUG whole function time 0.10577940940856934
number of 3d mask points: [380 306] number of 2d masks: [18151 12914]
91%|█████████████████████████████████████████████████████▋ | 50/55 [00:05<00:00, 10.51it/s]-------------------------frame 50 -------------------------
DEBUG Before projection 0.010257720947265625
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (15671, 2)
within_mask shape: (1, 15671)
(2045, 2)
DEBUG lowest depth 1000.0 0.6321509442000798
DEBUG Before Pooling 0.008143901824951172
DEBUG Pooling takes time 0.05050063133239746
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.6322, dtype=torch.float64)
DEBB (15671, 2)
DEBUG visibility mask 2216
DEBUG calculating visibility 0.0054552555084228516
DEBUG get final masks 0.001058816909790039
DEBUG whole function time 0.08646273612976074
number of 3d mask points: [1930] number of 2d masks: [52773]
-------------------------frame 51 -------------------------
DEBUG Before projection 0.011245250701904297
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (20904, 2)
within_mask shape: (1, 20904)
(11651, 2)
DEBUG lowest depth 1000.0 0.3923582039963355
DEBUG Before Pooling 0.029257535934448242
DEBUG Pooling takes time 0.07211947441101074
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(0.3924, dtype=torch.float64)
DEBB (20904, 2)
DEBUG visibility mask 6355
DEBUG calculating visibility 0.0025827884674072266
DEBUG get final masks 0.0011296272277832031
DEBUG whole function time 0.10570955276489258
number of 3d mask points: [5770] number of 2d masks: [101982]
95%|███████████████████████████████████████████████████████▊ | 52/55 [00:05<00:00, 10.06it/s]-------------------------frame 52 -------------------------
DEBUG Before projection 0.010248422622680664
DEBUG (110421, 2) (3, 360, 640)
inbound_shape: (33891, 2)
within_mask shape: (3, 33891)
(6148, 2)
DEBUG lowest depth 1000.0 1.6153810853139052
DEBUG Before Pooling 0.01897573471069336
DEBUG Pooling takes time 0.09053301811218262
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(1.6154, dtype=torch.float64)
DEBB (33891, 2)
DEBUG visibility mask 3680
DEBUG calculating visibility 0.002962827682495117
DEBUG get final masks 0.0034427642822265625
DEBUG whole function time 0.11652827262878418
number of 3d mask points: [1183 1136 1195] number of 2d masks: [17211 16853 11862]
-------------------------frame 53 -------------------------
DEBUG Before projection 0.007999658584594727
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (29151, 2)
within_mask shape: (1, 29151)
(1223, 2)
DEBUG lowest depth 1000.0 1.5650419398277684
DEBUG Before Pooling 0.009886741638183594
DEBUG Pooling takes time 0.08727049827575684
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(1.5650, dtype=torch.float64)
DEBB (29151, 2)
DEBUG visibility mask 922
DEBUG calculating visibility 0.0034759044647216797
DEBUG get final masks 0.0014085769653320312
DEBUG whole function time 0.10304450988769531
number of 3d mask points: [737] number of 2d masks: [11922]
98%|█████████████████████████████████████████████████████████▉ | 54/55 [00:05<00:00, 9.43it/s]-------------------------frame 54 -------------------------
DEBUG Before projection 0.007568836212158203
DEBUG (110421, 2) (1, 360, 640)
inbound_shape: (23365, 2)
within_mask shape: (1, 23365)
(8875, 2)
DEBUG lowest depth 1000.0 1.1562889892672752
DEBUG Before Pooling 0.030913352966308594
DEBUG Pooling takes time 0.0499730110168457
DEBUG lowest depth tensor(1000., dtype=torch.float64) tensor(1.1563, dtype=torch.float64)
DEBB (23365, 2)
DEBUG visibility mask 2469
DEBUG calculating visibility 0.003454923629760742
DEBUG get final masks 0.00160980224609375
DEBUG whole function time 0.0868673324584961
number of 3d mask points: [1853] number of 2d masks: [47300]
100%|███████████████████████████████████████████████████████████| 55/55 [00:05<00:00, 9.26it/s]
sucsessfully saved point cloud
masks_to_be_merged [[0], [1], [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22, 23, 24, 25, 26, 28, 29, 30, 32, 68], [17, 27, 31], [33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 45, 46, 47, 48, 49, 52, 55, 56, 57, 59, 60, 63, 64, 65, 66, 69, 70, 73], [35, 40, 50, 51, 53, 54, 61, 62, 71, 72], [58], [67]]
occurance count tensor([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29], device='cuda:0',
dtype=torch.int32)
occurance thres value tensor(6, device='cuda:0', dtype=torch.int32)
after filtering torch.Size([3, 110421])
num_ins_points_after_filtering tensor([6157, 2176, 737], device='cuda:0')
DEBUG torch.Size([3, 110421]) [tensor(0), tensor(1), tensor(2)]
0
tensor([0])
1
tensor([1])
2
tensor([2])
/mnt/c/Users/Marc/Desktop/dev/ORganizAR_MAR_DEMO/viewer/ORganizAR_LangSAM.py:651: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
filtered_3d_masks["ins"] = torch.tensor(filtered_3d_masks["ins"][final_masks_indices,:])
instance_coords [[ 0.34139118 -1.35022748 -0.2083721 ]
[-0.12591281 -1.01535024 -0.0532721 ]
[-0.12642925 -0.9750535 -0.06792925]
...
[-1.7810851 -1.30560895 -0.57569311]
[-1.76049733 -1.30489547 -0.56593404]
[-1.76290772 -1.32530909 -0.56939982]]
start_point (60, 34)
axis: 0
get_target_pos: -2.798000000000002
axis: 1
get_target_pos: -0.3120000000000118
axis: 2
get_target_pos: 1.004000000000019
pos [-2.798000000000002, -0.3120000000000118, 1.004000000000019]
target_pos [-2.81000023 0.98351084 0.26602468]
end_point (5, 80)
using diagonal
original hull area: 62.0
[-3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11]
[71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86]
(55, 3)
bed
instance_coords [[-0.18986951 -0.1630533 -0.69035684]
[-0.1966483 -0.16994508 -0.70003759]
[-0.17341632 -0.1466927 -0.67815671]
...
[ 0.12498067 -0.10590312 -0.65121227]
[ 0.14517611 -0.1257345 -0.65052828]
[ 0.14638405 -0.14710569 -0.64988572]]
start_point (70, 51)
axis: 0
get_target_pos: -2.0159999999999627
axis: 1
get_target_pos: -0.34799999999995634
axis: 2
get_target_pos: -0.6420000000000528
pos [-2.0159999999999627, -0.34799999999995634, -0.6420000000000528]
target_pos [-2.01316562 -0.65399155 0.34202812]
end_point (25, 40)
original hull area: 62.0
[17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32]
[32 33 34 35 36 37 38 39 40 41 42 43 44 45 46]
(46, 3)
chair
instance_coords [[-0.01678365 1.08712881 -0.16333228]
[-0.02134869 1.04517185 -0.15911979]
[-0.01886359 1.08904019 -0.16383183]
...
[-0.09811437 1.10999202 -0.28293803]
[ 0.11952872 1.05044925 -0.06032217]
[ 0.09609781 1.10140505 -0.24337412]]
start_point (62, 78)
axis: 0
get_target_pos: -1.4569999999999936
axis: 1
get_target_pos: -0.4560000000000173
axis: 2
get_target_pos: 1.7690000000000055
pos [-1.4569999999999936, -0.4560000000000173, 1.7690000000000055]
target_pos [-1.47756231 1.76384851 0.40717815]
Traceback (most recent call last):
File "/mnt/c/Users/Marc/Desktop/dev/ORganizAR_MAR_DEMO/viewer/ORganizAR_LangSAM.py", line 726, in
end_point = get_starting_point(np.asarray([[target_pos_instance_grid_coords[0],target_pos_instance_grid_coords[1]]]), grid)
File "/mnt/c/Users/Marc/Desktop/dev/ORganizAR_MAR_DEMO/viewer/path_planning/path_planning.py", line 222, in get_starting_point
if grid[i,j] == 0:
IndexError: index 100 is out of bounds for axis 1 with size 100
Process _interconnect-4:
Traceback (most recent call last):
File "/root/miniconda3/envs/search/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/mnt/c/Users/Marc/Desktop/dev/ORganizAR_MAR_DEMO/viewer/hl2ss_mp.py", line 238, in run
self._process_source()
File "/mnt/c/Users/Marc/Desktop/dev/ORganizAR_MAR_DEMO/viewer/hl2ss_mp.py", line 193, in _process_source
ipc.release()
File "/root/miniconda3/envs/search/lib/python3.10/multiprocessing/managers.py", line 1050, in release
return self._callmethod('release')
File "/root/miniconda3/envs/search/lib/python3.10/multiprocessing/managers.py", line 817, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/root/miniconda3/envs/search/lib/python3.10/multiprocessing/connection.py", line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/root/miniconda3/envs/search/lib/python3.10/multiprocessing/connection.py", line 411, in _send_bytes
self._send(header + buf)
File "/root/miniconda3/envs/search/lib/python3.10/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe