keijiro / bibcam

Burnt-in barcode metadata camera

Bibcam - Burnt-in barcode metadata camera

Bibcam is a proof-of-concept project where I tried the idea of "burnt-in barcode metadata" carrying camera tracking data within a single video stream.

By using this format, you can record/edit/playback AR-ready video clips without worrying about desynchronization with external tracking data.

System Requirements

• Unity 2021 LTS
• LiDAR-enabled iOS device for recording

How to Try

Build and play the Encoder scene on a LiDAR-enabled iOS device. You can record Bibcam video clips by pressing the "Record" button. It saves recorded clips into the camera roll.

To try playback with the recorded clips, copy them into the project directory (the directory that contains Assets, Packages, etc.) and rename it to Test.mp4. Then you can play it back with the Decoder scene.

Tips

• The encoder application caps the frame rate at 30fps to reduce excessive energy consumption and thermal throttling. You can switch this behavior in the application settings (Settings -> Bibcam -> "Cap Frame Rate").

Related Repositories

• BibcamVfx -- Advanced Bibcam VFX with HDRP
• BibcamUrp -- Bibcam renderer on URP
• BibcamTimeline -- Bibcam frame-accurate playback with Timeline

bibcam's Issues

"striped screen"

Compiled from Unity 2021.2.3f1 running on iOS 15.0.2 iPhone 13 Pro. Getting a striped screen, borders turned red when I press record. Test.mp4 video fails to play in decoder:
WindowsVideoMedia error unhandled Color Standard: 0 falling back to default this may result in rendering issues

Works with lidar or realsense devices ?

Hi Keijiro, once again it's an amazing project. Any chance to see this app works with any other lidar or Realsense devices ? I don't have an IPhone 13 Max. Or is there a way to synchronize tracking data with a video clip recorded with a lidar or realsense devices.

Arigato gozaimasu.

Flickering of depth image/buffer when fps is low and device is rotated

Hi,

we are seeing some strange behavior in Bibcam that we can't get to the bottom of. Therefore, I was wondering if you've experienced the same issue:

When the iOS device's fps gets below 20 fps or so and we rotate the device (with the orientation of the device locked!), the depth image "flickers" which results in all pixels getting drawn to something almost planar. Here are two images of the issue, the first one being a correct frame, the second one, where the buffer flickers:

PS: We encode the confidence values for each pixels in the green channel of the stencil buffer. This is not causing the issue, it's also happening in a "clean" Bibcam.

Question regarding portrait mode

Hi! I just wanted to ask if you considering making Bibcam also available for recording in portrait mode?

Only run on iOS??

Hi, I am interested in your project.
And I wonder, does this only work on iOS? I mean, can't it run on Android device with ToF?

Thank you.

No communication with Camera

Hi,
I love this project, thanks tos for sharing with the world.
Noob here. I can't manage to capture any video on my device. It looks like there's simply no communication with iPhone's camera. I don't see any preview (only stripes) and even if I hit record, nothing is saved on device.
Unity Remote 5 has camera access in iOS settings.

I'm using Unity 2021.2.5f1 Personal on a MBP 16" with MacOS Big Sur.
iPhone 13 Pro Max, iOS 15.1.1 (I tried both "Any iOS device" or "iPhone 13 Pro" in Project Settings / Editor)

Am I missing something ?

Test Data

Hi,

do you have a test recording by any chance so we can play around with the decoding without actually having a LIDAR enabled device at home? That would be great!

Work wit URP?

Hello, your project is just amazing. Is there any way I can get it to work wit URP? When I change to it, the video is not displaying anymore

Realtime streaming?

Hi,

Could this video 'format' be streamed in realtime to an other device (desktop or mobile) using something like NDI ?

Thank