jc211 / nerfcapture

An iOS app that collects/streams posed images for NeRFs using ARKit

NeRF Capture

Collecting NeRF datasets is difficult. NeRF Capture is an iOS application that allows any iPhone or iPad to quickly collect or stream posed images to InstantNGP. If your device has a LiDAR, the depth images will be saved/streamed as well. The app has two modes: Offline and Online. In Offline mode, the dataset is saved to the device and can be accessed in the Files App in the NeRFCapture folder. Online mode uses CycloneDDS to publish the posed images on the network. A Python script then collects the images and provides them to InstantNGP.

Online Mode

Use the Reset button to reset the coordinate system to the current position of the camera. This takes a while; wait until the tracking initialized before moving away.

Switch the app to online mode. On the computer running InstantNGP, make sure that CycloneDDS is installed in the same python environment that is running pyngp. OpenCV and Pillow are needed to save and resize images.

pip install cyclonedds

Check that the computer can see the device on your network by running in your terminal:

cyclonedds ps

Instructions found in here

Offline Mode

In Offline mode, clicking start initializes the dataset. Take a few images then click End when you're done. The dataset can be found as a zip file in your Files App in the format that InstantNGP expects. Unzip the dataset and drag and drop it into InstantNGP. We have found it farely difficult to get files transferred from an iOS device to another computer so we recommend running the app in Online mode and collecting the dataset with the nerfcapture2nerf.py script found in InstantNGP.

Citation

If you use this software in your research, please consider citing it.

@misc{
  NeRFCapture,
  url={https://github.com/jc211/NeRFCapture},
  journal={NeRFCapture},
  author={Abou-Chakra, Jad},
  year={2023},
  month={Mar}
} 

nerfcapture's Issues

Error using nerfcapture2nerf.py script

Hello,

Many thanks for your great work!
Unfortunately, I am not able to get any connections on the Nerfcapture app.
I started the nerfcapture2nerf.py script on my computer and then opened the nerfcapture app on my iPhone.
Both the devices are on the same wifi network.

I tried it with 2 different iPhones, still the same issue. Am I missing anything?

Any suggestions, how to resolve it?

app and python script can't connect via dds

I'm running NeRFCapture on iPhone13, but I can't connect with nerfcapture2nerf.py on the PC, but I test the udp connection status through ISH nc on the phone, and it's OK on the wifi network.Does the iPhone need lidar?

I built the nerfCapture project as an ios app, but I have problems with online mode

Hello, I am currently using NerfCapture to build an ios app, but in the online mode, there is no response when I press the send button. Do I need to set the ip or other parameters before building the ios app? Or Is there another step I'm missing?

Pose issues (offline mode)

Hi, I used NeRFCapture to capture a scene with an iPhone 12 Pro. From the instantNGP debug visualization, the camera poses seem to be scaled weirdly (see screenshot). The scene did not converge.

The data zip can be downloaded from here.

TypeError: set_camera_extrinsics(): incompatible function arguments.

Hey Jad,
Really cool project!

I'm trying to run it on an iPhone 13 Pro Max, but I run into the following error when I try to send an image from my phone to instant-ngp when I run the following command $python scripts/nerfcapture2nerf.py --stream.

Do you know what the problem could be?

Cheers!
Christian

Python to give back to IOS

I saw this github while searching for ways to send arkit data to my ubuntu machine.
I think your app / code does just that: sending live streams of RGB-D images and pose.

Since I am working on a project that needs to give the user a feedback like 3D bounding box around objects (I am running a 3D segmentation using machine learning on ubuntu) and show it on Iphone, I wonder if cyclonedds or flask can be used/mixed to fulfill this.

Thank you!

Maximum number of photos with online mode.

When I send images to Instant-NGP in online mode, I noticed I get capped at 10 images. When I take my eleventh image, the 1st image is removed from the GUI. The software will continue the first in, first out pattern as I send more images. Is this expected behavior? Is it so I don't go over my VRAM limit? I am using about 9 GB and I have 24 GB total of VRAM.

Thanks for all of the help!

Tips for Preparing Datasets

Hello, Jad: Thank you for your previous answer. I have encountered some problems when preparing the dataset. I took about 120 images of a large object, but when training the model, I found that the 3D model did not render well. , May I ask if you have some shooting skills and can provide me with some suggestions when collecting dataset? Thank you.

The dataset with depth rendering worse than COLMAP

Hi, I tried to use online mode to capture frames and I saw it got good depth map files.
But when I training the model in instant-ngp, the result always blur and not clear.

I use the same frames and remove depth files, using COLMAP to generate the transforms.json, and training, the result is much better than depth one.

Is there something I doing wrong? How can I get better result with depth files?

NerfCapture

COLMAP

How to merge new dataset into old one?

If I offline captured and got the dataset, later I found I need to add some more frames, how can I merge the later dataset into old one?
Should I shoot the same frames again, or how can I use the photos and depth files to generate transforms.json?

about transform_matrix

Is the transformation matrix in transforms.json in the form of Tcw or Twc?

Thanks！

Images coming in sideways

I tested the app and the images come in rotated 90 degrees. Is the app intended to be ran holding the phone horizontally?

Renders are dusty and have weird color effects

Hi, thanks for your work.

I'm currently experimenting with nerfcapture and instant-ngp. However, for some reason the renders have dust cloud effects as well as weird colorings. There are some small color specks visible, but it looks almost grayscale. Do you have any idea as to why this could be the case? These problems do not occur when working with photos or videos and preprocessing them with the scripts provided by instant-ngp.

Running "python scripts/nerfcapture2nerf.py --stream" crashes the app

Hello.

When I run the command "python scripts/nerfcapture2nerf.py --stream" in instant-ngp with NeRF Caputure running, the app immediately crashes.

Even if I re-launch the app and press "Send" button, it crashes immediately again.

What could be the cause?

Environment
・Ubuntu 22.04
・iPhone 12
・net.core.rmem_max = 2147483647

How to use captured depth images?

Hello,
Thank you for the great work!!

I'm using NeRFCapture app for obtaining RGB-D images and poses.
However, when generating point cloud using these data, It is something weird.

I think it is because of depth image scale.
How to resolve it, and What is the scale of the depth map from the NeRFCapture?

Thank you 🥹

Network Config does not exist

I installed the app on my iPhone 14 Pro and ran cycloneddps. It identifies my iPhone so I am getting a connection. When I run: python scripts/nerfcapture2nerf.py --stream to launch the app with Instant-NGP, I get an runtime error:

16:37:05 SUCCESS  Initialized Vulkan and NGX on device #0: NVIDIA GeForce RTX 3090 Ti
16:37:06 INFO     Loading network config from:
Traceback (most recent call last):
  File "C:\Users\jonat\Instant-NGP-20230404\scripts\nerfcapture2nerf.py", line 241, in <module>
    live_streaming_loop(reader, args.n_frames)
  File "C:\Users\jonat\Instant-NGP-20230404\scripts\nerfcapture2nerf.py", line 82, in live_streaming_loop
    testbed.reload_network_from_file()
RuntimeError: Network config  does not exist.

I am not sure what I am doing wrong on my end.