facebook / facebook360_dep

Facebook360 Depth Estimation Pipeline - https://facebook.github.io/facebook360_dep

---

Facebook360 Depth Estimation Pipeline (facebook360_dep) is a computational imaging software pipeline supporting on-line marker-less calibration, high-quality reconstruction, and real-time streaming and rendering of 6DoF content.

This work is based on the 2019 SIGGRAPH Asia publication "An Integrated 6DoF Video Camera and System Design" (paper, supplemental material).

Contents

• Requirements
• Documentation
• How to Contribute
• Code of Conduct
• Support
• License

📋 Requirements

facebook360_dep can be run on Windows, macOS, or Linux as your development operating system with the use of Docker. Building source natively on operating systems has been tested on Mac but is not the recommended way to use the software pipeline. Certain capabilities are restricted on the operating systems, due in part to restrictions to the GPU from Docker.

📖 Documentation

Here you can find short guides for the most common scenarios: facebook360_dep.

👏 How to Contribute

The main purpose of this repository is to continue evolving facebook360_dep core. We want to make contributing to this project as easy and transparent as possible, and we are grateful to the community for contributing bugfixes and improvements. Read below to learn how you can take part in improving facebook360_dep.

Code of Conduct

Facebook has adopted a Code of Conduct that we expect project participants to adhere to. Please read the full text so that you can understand what actions will and will not be tolerated.

Contributing Guide

Read our Contributing Guide to learn about our development process, how to propose bugfixes and improvements, and how to build and test your changes to facebook360_dep.

Good First Issues

We have a list of good first issues that contain bugs which have a relatively limited scope. This is a great place to get started, gain experience, and get familiar with our contribution process.

❓ Support

If you need help with your facebook360_dep app, the right place to go depends on the type of help that you need.

📄 License

facebook360_dep is BSD licensed, as found in the LICENSE file.

facebook360_dep's Issues

Docs: Redirection Issue in a sub-component of a footer section

Issue

1. The home page of facebook360_dep has a navbar which has nav component Github which on clicking redirects to https://github.com/facebook/facebook360_dep
2. But if I come in footer and then click Github , it redirects to https://github.com/

Describe what you expected to happen:

I expect when I click on Github in the footer section , it should also redirect to https://github.com/facebook/facebook360_dep like it does in the navbar

Snack, code example, or link to a repository:

Link to website: https://facebook.github.io/facebook360_dep/

Screenshot:

If this issue seems valid , I would like to work on this , so please assign me

Docs: Calibration files of common 360 cameras

Would it be possible to start documenting the calibration files of common 360 cameras? I'm a little head-scratchy when it comes to determining the orientation and origin of the cameras on my Insta360 Pro 2.

UPDATE: The NVIDIA VRWorks SDK has a tool that determines the calibration based on a set of photos. I ran a set from the Insta360 Pro 2 through them to see if I could match them to what the FDEP wants but they don't match up. If you can determine the proper values from them, it might be a nice way to get this data without having the camera specs? The output is attached.

out_calib.zip

Feature: Stripe among multiple machines in EC2

The last stage of the 6DoF rendering pipeline is to stripe the binaries into a single binary file. However, it is currently implemented to only be done on a single computer. This means that, regardless of the size of your cluster, the final stage will take the same amount of time, since it is all performed on a single machine. The striping, however, can be parallelized in a mergesort-like fashion.

The current behavior and new feature

The striping is performed by ConvertToBinary (source/mesh_stream/ConvertToBinary.cpp), which produces a single binary file along with a digest JSON that describes the contents of the binary. This is effectively the same as how a .TAR file is created. To stripe across multiple machines, you will need to figure out how to properly merge these binaries together, which should be nothing more than concatenating them, and merge the JSON digest, which will involve updating the offsets of where different pieces are located in this final binary.

Expected use cases

End users will benefit primarily from a great reduction (likely upwards of 5-10%) in rendering time.

The problem of too few panoramic camera matches

Steps To Reproduce

1.1.After I modified the rig.json file based on the Insta360 Pro camera parameters, I found that the system prompts too few matches and can't generate a rig_calibration.json file.

2. The generated depth map is not accurate enough.

Describe what you expected to happen:

1. Could you please help explain why are there too few matches point, and how can I get more matching point?
2. Could you help to see how the depth map can be seen more clearly？
   

Snack, code example, or link to a repository:
https://github.com/frank561/Fan-Wu.git

cmakelists.txt missing find_package(zlib REQUIRED)

Feature: Compute first N coarsest levels in a single worker during AWS renders

The current behavior and new feature

At any level of the pyramid a worker needs to download the color level for a frame from S3, perform depth estimation, and then pack and send the results back.

At the coarsest levels we end up spending more time downloading and uploading data than actually processing the frames. We can speed up the process by letting workers do the coarsest N levels (e.g. levels 9 to 4) at once.

Note that this means not being able to run the temporal filter after each of these levels, since that stage is also distributed, but it may be a good trade-off.

Expected use cases

Faster renders at the expense of worse temporal smoothing, but may be not noticeable.

Feature: LiDAR Integration

LiDAR (https://en.wikipedia.org/wiki/Lidar) is used heavily to obtain dense point cloud representations of scenes. By performing a LiDAR scan of a scene before starting captures, we can potentially greatly improve accuracy background disparity estimation and, thus, in creating accurate foreground masks.

The current behavior and new feature

Render pipeline: this would most likely integrate into the background depth estimation and foreground mask generation steps.

Expected use cases

Research effort, potentially being integrated in future shoots if results turn out to be significantly better than they are currently.

Application fails to launch: qt.qpa.xcb: could not connect to display :0

Upon attempting to launch by running "python3 -u scripts/render/run.py --flagfile res/flags/run.flags" the following error occurs:

qt.qpa.xcb: could not connect to display :0
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: eglfs, linuxfb, minimal, minimalegl, offscreen, vnc, wayland-egl, wayland, wayland-xcomposite-egl, wayland-xcomposite-glx, webgl, xcb.

Fatal Python error: Aborted

Current thread 0x00007f86a2f57740 (most recent call first):
File "dep.py", line 282 in main
File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 251 in _run_main
File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 300 in run
File "dep.py", line 315 in
Fatal Python error: Segmentation fault

Current thread 0x00007f86a2f57740 (most recent call first):
File "dep.py", line 282 in main
File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 251 in _run_main
File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 300 in run
File "dep.py", line 315 in

I have followed the Docker installation instructions. Incidentally I have previously used the application and I think I encountered the same error and managed to resolve however that was many months ago and don't recall how. I think the issue is related to Docker / X11 forwarding. This is occurring using Ubuntu 18.04.5 LTS with an Nvidia GPU.

[Dockerfile] Eigen repository moved

Eigen moved their repository from Bitbucket to Gitlab recently (the announcement).
So 69-74 lines of the Dockerfile need to be updated like this:

RUN curl -LO https://gitlab.com/libeigen/eigen/-/archive/3.3.4/eigen-3.3.4.tar.gz && \
    tar -zxf eigen-3.3.4.tar.gz && \
    cd eigen-3.3.4 && \
    mkdir build && cd build && \
    cmake .. && make install -j$(nproc) && \
    cd ../../ && rm -fr eigen-3.3.4

Feature: WebGL Viewer Support

The WebGL standard has been adopted by all modern web browsers and offers an API for rendering using the GPU on a browser. The browser is the end-all-be-all for many users, so making 6DoF consumable through the browser is a necessary step towards getting more people to experience the content.

The current behavior and new feature

GLViewer is the viewer for 6DoF content if not using a Rift headset. It is an extensive array of OpenGL and meshing code, so converting them to a similar WebGL viewer will likely be a significant undertaking. There is also the possibility of using wasm and compiling our C++ code to be ingested by a wasm-supported browser, but this seems more experimental and will likely have performance hitches.

Relevant Files:

• source/viewer/GlViewer.cpp and linked headers, i.e.:
  • source/gpu/GlfwUtil.h
  • source/render/RigScene.h
  • source/render/VideoFile.h

Expected use cases

This would have great impact in making 6DoF content more easily consumable and, therefore, more likely for people to engage with the content. However, it will likely be a significant undertaking, for which we would likely recommend working in a small group (of say 2-3 people) towards completion.

Can't load images via symbolic links?

On Windows, image file names that are symlinks give 'Missing file' errors. It would be good to test if this is also the case on *nix. While there is a good chance that part of the fault is the Boost filesystem's known difficulties with handling Windows' nasty symlink format, my reading of the project code suggests that symlinks might not even work on Linux. For example these lines in imageUtil.cpp::verifyImagePaths() would likely reject a symlink:
const bool exists = filesystem::is_regular_file(p); CHECK(exists) << "Missing file: " << p;
It is actually possible to read from Windows symlinks with Boost, but this does require some platform-specific code. If the Linux builds already use symlinks OK, I would suggest giving this issue a low priority; but if not, a high one.

Steps To Reproduce

1. copy a calibration setup where MatchCorners runs correctly
2. replace image files with symlinks to files stored elsewhere
3. run MatchCorners

expect "Missing file" failure.

Extraneous "Connected/Authenticated" output on AWS Render

Running any render job on AWS produces extraneous connection/authenticated messages (see the ones logged by transport.py).

These are caused by running commands over SSH, which is done in the scripts/aws/create.py run_ssh_command function. We wish to silence that output.

Steps To Reproduce

1. Ran

python3 scripts/render/run.py --flagfile res/flags/run.flags

Where res/flags/run.flags is:

--alsologtostderr=1
--stderrthreshold=0
--log_dir=~/logs
--project_root=s3://facebook360-dep-sample-data/room_chat/50_frames
--cache=/Volumes/cache
--csv_path=~/Downloads/accessKeys.csv

2. Clicked on "Run" on the depth estimation tab with default values except first=000100 and last=000105.

Describe what you expected to happen:

No extraneous output being produced

Feature: Semantic Segmentation in Masking

Foreground masks act as a rough proxy to objects in the scene. Significant work on semantic segmentation (re: https://github.com/facebookresearch/Detectron) has been done, although they have not been thoroughly integrated with our pipeline to improve results.

The current behavior and new feature

Render (Foreground masks): A weighted sum of a traditional foreground mask and semantic segmentation can be used to obtain a more accurate set of masks for the scene. This can even be extended to have a multi-planar image if the semantic segmentation is accurate enough.

Expected use cases

Significant research effort.

Feature: Machine Learning Based Depth Estimation

Machine learning has replaced most "traditional" computer vision algorithms, especially in domains like annotation and labelling. Attempts to apply ML to depth estimation certainly exist, but only perform consistently well in limited capture environments (i.e. those without moving bodies or humans). Applying them to subsets of our pipeline may shed more light on how they can be used fully though.

The current behavior and new feature

Render pipeline: The algorithm is fully based on "traditional" computer vision. Instead, ML approaches may attempt to improve either foreground mask generation or the depths directly.

Expected use cases

Significant research needed.

Tutorial link is 404:

Steps To Reproduce

1. click on link: https://facebook.github.io/facebook360_dep/docs/tutorial.html
2. get 404

Describe what you expected to happen:
Link should lead to documentation

Docs: license of the example data

Hi, first of all thanks for providing this software.

The software license (BSD) is clear to me, but I it is not clearly stated if the example data has the same license. Could this be clarified?

There is an active standardization project within MPEG called MPEG Immersive Video (MIV) [ https://gitlab.com/mpeg-i-visual/tmiv ], and I was wondering if the example data that you have provided could be used at least for "academic and standardization purposes".

A redirection issue in the footer of facebook360_dep 's website

Steps To Reproduce

1. The home page of facebook360_dep has a navbar which has nav component Tutorial which on clicking redirects to https://facebook.github.io/facebook360_dep/docs/rig
2. But if I come in footer and then click Tutorial , it redirects to https://facebook.github.io/facebook360_dep/docs/calibration.html

Describe what you expected to happen:

I expect when I click on Tutorial in the footer section , it should also redirect to https://facebook.github.io/facebook360_dep/docs/rig like it does in the navbar

Snack, code example, or link to a repository:

Link to website: https://facebook.github.io/facebook360_dep/

Screenshot:

If this issue seems valid , I would like to work on this , so please assign me

Multiproccessing in resize.py hangs run.py

When I try to start the UI with run.py, the script hangs when it tries to generate the color_levels. Through some investigation, I found that it was hanging in resize_frames(). I'm not sure exactly why the multiprocessing doesn't work, but changing the function to directly resize the images fixed the issue for me. Here's how I altered resize frames:

def resize_frames(src_dir, dst_dir, rig, first, last, threshold=None):
    """Resizes a frame to the appropriate pyramid level sizes. Files are saved in
    level_0/[camera], ..., level_9/[camera] in the destination directory.

    Args:
        src_dir (str): Path to the source directory.
        dst_dir (str): Path to the destination directory.
        rig (dict[str, _]): Rig descriptor object.
        first (str): Name of the first frame to render.
        last (str): Name of the last frame to render.
        threshold (int): Threshold to be used for binary thresholding. No thresholding
            is performed if None is passed in.
    """
    
    for frame in range(int(first), int(last) + 1):
        for camera in rig["cameras"]:
            verify_frame(src_dir, camera["id"], get_frame_name(frame))
            resize_camera(
                src_dir,
                dst_dir,
                camera["id"],
                camera["resolution"],
                get_frame_name(frame),
                threshold,
            )

I'm on Windows 10, using Docker with the WSL 2 configuration. I'm sure this function works on other setups, but in case anyone else sees this issue, this might help.

Docs: A redirection issue in the footer of facebook360_dep 's documentation website

Issue

1. The home page of facebook360_dep has a navbar which has nav component Tutorial which on clicking redirects to https://facebook.github.io/facebook360_dep/docs/rig
2. But if I come in footer and then click Tutorial , it redirects to https://facebook.github.io/facebook360_dep/docs/calibration.html

Describe what you expected to happen:

I expect when I click on Tutorial in the footer section , it should also redirect to https://facebook.github.io/facebook360_dep/docs/rig like it does in the navbar

Snack, code example, or link to a repository:

Link to website: https://facebook.github.io/facebook360_dep/

Screenshot:

If this issue seems valid , I would like to work on this , so please assign me

Feature: Quick Stitch Option

On set, having quick turnaround times on renders is crucial: it helps uncover setup issues (i.e. flipped cameras). A "quick stitch option" on the UI would skip calibration and background entirely, and directly produce depth estimates and a color equirect at low res.

The current behavior and new feature

Render: To see whether the camera arrangement is proper, we must go through the entire pipeline to produce an equirect. Having a quick stitch allows a quick glance first pass sanity check, which could save hours down the road.

Expected use cases

Will be useful for end users (i.e. producers on set).

Docs: How is RGB-D data packed in result?

I couldn't find any examples or relevant code about how the resulting RGB-D videos and images are packed.

In the past I've seen that facebook uses the lower half of the video to pack 24 bits depth in the RGB channels (or YUV for that matter).

There is a vulnerability in jQuery 1.7.2,upgrade recommended

facebook360_dep/website/static/source/html/jquery.js

Line 2 in 2ec9e65

* jQuery JavaScript Library v1.7.2

CVE-2020-11023 CVE-2020-11022 CVE-2020-7656

Recommended upgrade version：3.5.0.1