illixr / illixr Goto Github PK

Requirements:

• System should be able to checkout (possibly new branch) the main module and every submodule to a branch of the same name.
• System should be able to merge the main module and every submodule simultaneously.

Consider:

• Repo

Convert timewarp to use direct mode, using Monado compositor as reference.

A std::shared_mutex permits EITHER n readers OR 1 writer.

• When a thread reads a datastructure, it only needs to acquire std::shared_lock. This permits parallel reads, since they don't conflict, but locks out all writers.
• When a thread writes a datastructure, it needs to acquire std::unique_lock. This locks out all other writers and all readers.

• Should contain timestamp in filename.
• Should dump to metrics/eyes/left/{timestamp}
• Should use timestamps := time since start in ms.

• See if ElasticFusion can be integrated.
• If not, find and use another.

Can probably do this in GitHub actions.

• Same format as EUROC dataset
• Should be able to record our own ZED datasets

• Header functions should be declared static.
• Use #pragma once instead of header guard.
• Either always use _m_field and _p_method or never.

• Standard Camera/IMU calibration format for ILLIXR
• Have runner or runtime parse this file and give as data structure for a slam implementation

Rationale: full VI-SLAM system with loop closure. ORB-SLAM3 has maps that stay persistent across runs, and we will need an efficient mechanism in ILLIXR to store/retrieve them.

• Leave the submodule where it is.
• Symlink relevant parts into a new plugin for decode and a new plugin for encode.
• Adjust plugin line in Makefile.

A workflow needs to run on branches that builds and pushes docker images to ghcr.io and runs the tests in ILLIXR.

• Use logging framework

Unlike EUROC, this would have RGB/depth data.

• In some places we use std::chrono::system_clock; in others we use std::chrono::high_resolution_clock. Which is better?
• Is there a way we could make this work on Windows?

• Decide which camera to get
• Write standalone camera code
• Write plugin

• Use 3.2.1 as base
• Add timing instrumentation
• Exit cleanly

Currently, ILLIXR timesout with 60 seconds. We need a way to configure this time or turn the timeout off. This should probably be set in the config yaml file for runner.py.

• Have a designated way to exit the program (Ctrl+C and handle SIGINT or have a way for a plugin to signal to the runtime "game over").

• Deliberate on the format
• Also standardize naming conventions (always use _m_ for private or never)
• Figure out how to enforce (make part of CI?)

• See if OpenVINS pose integration will suffice.
• Rip out the Newtonian mechanics pose prediction from OpenVINS.
• Expose it through the Phonebook pose_prediction interface.
• It needs some state and some IMU readings. Inside feed_imu_cam, push the last few IMU readings + relevant state into a topic called integrated_state.
• Read async integrated_state in the pose prediction.

• Make the application's rendering (ILLIXR/timewarp_gl | Monado) to wake up right after vsync (known by ILLIXR/timewarp_gl).
• Make xrWaitFrame wait until next vsync period.

• Like ROS bag, but for Switchboard.
• Maybe we could convert EUROC to that. This would obviate the need for offline_imu_cam and groundtruth_slam.

• Figure out how Godot queries pose.
• Confirm that xrLocateViews queries pose in OpenXR apps.
• Figure out how to handle this call in ILLIXR hmd in monado_integration.

• Windows support for offline IMU Camera
• Document Panacea plugin translation for Windows
• Windows support for OpenVINS
• Investigate what would be necessary for MacOS/iOS?

• AR Application: Create one in Godot.
• VR Application: Sponza

Or explain why we shouldn't.
This currently complicates the ZED code (where the Camera thread has to send frames to the IMU thread, which pairs them and sends them on a Switchboard topic).
This currently simplifies OpenVINS (where we need to process the camera after the IMU).

• Possibly OCLint
• Possibly cppclean
• Possibly clang-tidy
• Should detect unused #include

Our record_logger and the CPU timers in #209/#211 are good for computer-readable telemetry. However, we need an infrastructure for data intended to get to a user, such as warnings, error descriptions, and debugging messages.

Perhaps, we should use glog.

• By default, ILLIXR should not print anything except for really critical stuff (why this is considered "good UNIX style")
  • Logs in a loop can use something like glog's LOG_EVERY_N.
• Many of the assert(thing && msg) can be replaced with glog's CHECK(thing) << msg.
  • This permits the message to be a dynamically generated string, like CHECK(thing) << getErrorMessage(msg) which would not work with a raw assert.
  • CHECK is not suppressed by NDEBUG, which has caused us bugs in the past (and pesky [[maybe_unused]] annotations).
    • One can still explicitly reserve a check for debug builds by surrounding the CHECK with #ifndef NDEBUG, as we do already in many places (e.g. common/record_logger.hpp).
• Glog can dump the stacktrace on segfault.
  • We do already have catchsegv. There's not a whole lot of benefit to switching over to glog for that.
• Glog embeds the filename and line number in all logs.
• Perhaps we want to "selectively enable" debug flags for different plugins, as in LLVM. This is the only thing not readily avilable in glog.

• We use three different generators for our three different sub-sites. Google Analytics tracking codes have to be added to each sub-site.

• Google Analytics will allows us to track user demographics (motivate translations), popularity over time (evaluate which marketing works), and which order users visit our pages (help lay out the documentation in a user-friendly way).

• Main website

• Doxygen site

• mkdocs site

For most components, the best we can do is ensure it builds.
However, some components can also have tests make test.
In the past, broken code (non-compiling) has been pushed to master, so automating this has real benefit.

ZED's CMakeLists.txt currently has an additional linkage step which requires us to manually add C++ flags and a fake main(). Need to clean this up.

• This should be a new "compilation profile"
• Address sanitization and undefined behavior sanitization

• If timewarp[i] is on time (earlier than vsync[i]), image should bear the timestamp of vsync[i].
• If timewarp[i] is late, then it should bear the timestamp of vsync[i+1].
  • (However, it may be overwritten by timewarp[i+1], if timewarp[i+1] finishes before vsync[i+1]. The latest timewarp before vsync should be used. This overwriting is how normal file semantics work, so shouldn't complicate the implementation).
• Use pose_predict to get the timestamp of the next (nearest after now()) vsync.
  • This interface is not yet present; It is being developed by @JeffreyZh4ng .

• Install Python3.8 (safely, with pyenv)
• Install Poetry with Python3.8
• Install clang-10
• Make clang-10 default (don't want to export every time).
• Install curl, zip.

Print detailed message listing exactly what will be installed.

We need a logging library for human readable debugging/info. Probably want to use glog for this.

• https://docs.microsoft.com/en-us/cpp/cpp/errors-and-exception-handling-modern-cpp?view=vs-2019
• https://docs.microsoft.com/en-us/cpp/cpp/how-to-design-for-exception-safety?view=vs-2019
• https://docs.microsoft.com/en-us/cpp/cpp/exception-specifications-throw-cpp?view=vs-2019

• How to deliberate over design decisions (RFC)
• How to get help
• How to file a bug
• How to suggest a feature
• How to setup dev environment
• What is the code of conduct?
• How to cite borrowed code?

Recommended reading:

• GitHub Open Source Guides
• TODO group guides
• Governance Models

• default_plugins.md: Remove "TODOs"
• default_plugins.md: Remove NVIDIA GPU info from audio
• writing_your_plugin.md: Doulble-check for correctness with the new build system.

Can we support dynamic languages like Python?
Python is a bit of a special case, because the interpreter supports embedding into a larger process.

• In the logger, redirect some messages to a Switchboard topic.

If user specifies both offline_imu_cam and ZED plugins, the system should fault. Currently, it might assign both feeds to the same switchboard topic, resulting in a spliced feed.

• Translate file I/O
• Create leaf nodes
• Create root node
• Figure out HPVM build process
• Ensure end-to-end run
• Utilize streaming