Capture and develop short clips of openpilot with the openpilot UI (path, lane lines, modes, etc.) included, with the segment ID and seconds marker branded into the clip. Useful for posting clips in the comma.ai Discord's #driving-feedback and/or #openpilot-experience channel, reddit, Facebook, or anywhere else that takes video.
Show the bad and the good of openpilot! Very useful for making outstanding bug reports or posting UI video in interesting situations on Twitter!
Give this project a test with GitHub Codespaces if you haven't already!
- Prior GitHub Codespaces experience is not needed!
- You don't need an existing openpilot development environment setup!
- You don't need to install anything on your computer!
- You just need a free GitHub account.
- The free GitHub account will give you 30 free hours of a 4 CPU machine every month.
- It's preset with a spending limit of $0/mo. There is no risk of charging you anything.
- Setup is fast. You can go from nothing to rendering and developing a clip within two minutes.
- You don't need a comma prime or lite subscription.
- Cleanup is easy, you just delete it from https://github.com/codespaces afterwards.
Demonstration of speed or longitudinal behavior of openpilot with model-based longitudinal is nearly impossible or hard without this clipper:
ae_driving_highway.mp4
Cars can have bugs themselves. Here's my 2020 Corolla Hatchback phantomly braking on metal strips in stop and go traffic probably from the radar. Perhaps a future openpilot that doesn't depend on radar might be the one sanity checking the radar instead of the other way around currently.
phantom-radar-braking.mp4
This is a video of a bug report where openpilot's lateral handling lost the lane:
lost-line.mp4
- This has only been tested on data from a Comma Three. It is unknown if it can work for any other devices.
- The UI replayed is comma.ai's latest stock UI on their master branch; routes from forks that differ alot from stock may not render correctly. Your experience may and will vary. Please make sure to note these replays are from fork data and may not be representative of the stock behavior. The comma team really does not like it if you ask them to debug fork code as "it just takes too much time to be sidetracked by hidden and unclear changes".
- Older routes may not replay correctly or at all on the latest UI in the master branch.
- I strongly recommend you work on this from a desktop, laptop, or at least a tablet.
- You are advised to upload all files of the route to Comma Connect servers before attempting to render a route. If you do not upload all files, the replay will not render past the starting UI.
Ensure your openpilot route's files are fully uploaded on https://connect.comma.ai/. Click Files when viewing a route and select the option to upload all files (Upload ## files). Make sure it says "uploaded".
- This route is not yet fully uploaded:
-
- This route is fully uploaded:
-
- The driver or interior camera is not required to be enabled for recording or uploading for this. It's easier to just hit that "Upload all" button though. Unfortunately there's no only upload all wide camera, forward camera, and logs button.
- If this is news to you about recording or uploading driver video, you should be aware of a toggle in the openpilot UI to not record driver video and thus effectively not allowing upload of the driver video. Unfortunately, there's no record but block upload driver videos option.
- Note: If you do not upload all the forward camera files, the replay will not progress past the starting UI.
- It is possible to upload only a portion of the route and still render a clip, but it's not recommended if you are new to this clipper. You can find those instructions in Advanced Tips > Partial Uploads..
We will be using GitHub Codespaces.
- Setup: 1 minute
- Per Clip: About 5 minutes to capture a 30 second frame with the UI and compress the 30 second clip to ~50MB.
-
Right click on this button below, select
Open in New Tab, and launch a codespace in US West region. It'll be fully loaded when icons appear in the left sidebar for files.
-
In the left sidebar, open scratch_run.sh. You will be editing this file and running it to run the script to generate a clip.
There are options but these are the basic steps. Note, the screenshots below may vary in themes and UI, but the layout is generally the same.
- Find the openpilot route you wish to take a clip from in Comma Connect at https://connect.comma.ai.
- Find the starting seconds value. The route's timeline will have a widget below your cursor that's "segment number, local time". Segments are made every minute and start from 0. So scrub it, and do a little mental arithmetic to get the starting second. I usually do "60 * segment number + offset" as my mental calculation. Edit the starting second in the
scratch_run.shfile to this value.- Sample:
- In this example, the starting second would be at least 60 * 3 = 180 seconds.
- Don't stress on this, if this is your first time, just wing it. You'll get it.
- Sample:
- Get any segment ID of the route from
More Info. The example below would be071ba9916a1da2fa|2022-09-04--11-15-52--1. Edit the route ID in thescratch_run.shfile to this value. -
- Get a JWT Token from https://jwt.comma.ai with the same account type you log into Comma Connect with. It'll be a long string that starts a bit like
eyJ0eXAiOiJKV1QiLCJhb.... Edit the JWT token in thescratch_run.shfile to this value. Keep this token private, do not share it with anyone as it will grant access to your comma connect account for a year. - Change the clip length value in
scratch_run.shto the number of seconds you want to capture. Longer lengths take proportionally longer to capture. - Run the script with
./scratch_run.shin the Terminal.- Sample:
- Sample:
- Wait 3 minutes, and the script should complete.
- Sample:
- Sample:
- After it completes, click "Go Live" in the bottom right corner to start a web server and open the web server in a new tab. Browse to the
sharedfolder- Clicking Go Live:
-
- Web Server View Sample
-
- Clicking Go Live:
- Right click and download
clip.mp4(or any files you've generated) to your computer. You can share or upload this file wherever you want. - If you want to make more clips, continue to edit and run
./scratch_run.sh, and refresh the web server's tab. - Cleanup is easy. Delete the GitHub Codespace here: https://github.com/codespaces. If you forget, the GitHub Codespace will automatically stop after 30 minutes of inactivity and will automatically be completely deleted after 30 days of idle by default.
- It is also possible to restart a pre-existing codespace and continue where you left off if it wasn't deleted.
Maybe you want to run this on your own computer, like if you want to generate many clips, really long clips, have run out of the free Codespace hours, or some other reason.
The requirements may be a bit high.
- 4 vCPUs/hyperthreads
- 2vCPUs/hyperthreads with a lower
--speedhack-ratiovalue that renders slower to maintain stability
- 2vCPUs/hyperthreads with a lower
- A working Docker-Compose setup. Docker for Windows or Docker for Mac will work.
- Intel or AMD processor.
- Emulation of Intel on Apple Silicon with Docker for Mac is too slow to handle the requirements. Please use a suitable Intel or AMD machine.
- 10 GB of disk space. More needed if you're rendering longer clips as intermediates are quite raw.
- 100MB/s disk speed.
- Docker for Windows users should clone the repository to the Linux filesystem to meet the requirement.
- A GPU is not needed. However, one could be used to accelerate the rendering process.
The CPU requirement is due to a number of factors:
- Reliable H.265 hardware decoding is not always available. The high quality forward video is only captured in H.265 and could only be decoded at 0.7 speed on a Ryzen 2800 and at half speed reliabily for the purposes of capture. And there are also two video streams: telescope and wide!
- Reliable OpenGL rendering is not always available. Software OpenGL rendering is used instead to guarantee compatibility.
- Capturing the UI can be quite intensive due to all the software and non-hardware-accelerated rendering and decoding.
- Capturing the UI must be done with everything not mismatching by speed. Otherwise, you get weird rendering issues like the planner's line lagging and not matching the forward video such as in the case of the forward video not decoding fast enough. A generous margin of extra performance is used to ensure that the UI is captured at the same speed as the forward video in case of unexpected system jitters.
Even with these CPU requirements, it was not enough to run the tooling at full speed on the CPU. Some measures have been done to make clip recording possible.
- Relevant processes are speedhack'd with
faketimeto run at 0.3x by default. - Capture is done in real time but undercranked to simulate full speed.
- Install Ubuntu for WSL2: https://ubuntu.com/tutorials/install-ubuntu-on-wsl2-on-windows-10
- Install Docker for Windows: https://docs.docker.com/desktop/install/windows-install/
- Open up the Ubuntu terminal and clone this repository:
git clone https://github.com/nelsonjchen/op-replay-clipper/- For performance reasons, make sure you clone to the Linux filesystem. The default working directory of your home directory in Ubuntu when you open up Ubuntu is adequate.
- Change folders to
cd op-replay-clipper - Continue to Steps.
If you are knowledgeable about Docker, Linux, Docker-Compose and whatnot, I'm sure you can figure it out. Just clone this repo down and go through the Steps.
You may need to chmod the shared folder to be writable by the internal Docker user of the image. Just do 777, it's all temporary anyway.
It's recommended you open the repository as a Dev Container in VS Code:
From there on, follow the Steps as normally used with GitHub Codespaces.
You may want to "Rebuild Container without Cache" to update to a newer Openpilot UI periodically.
Currently only tested with NVIDIA GPUs and on WSL2 in Windows 11. Setup in other environments may be possible, but untested.
See .devcontainer/docker-compose.yml for some lines to uncomment when running this tool inside VSCode's Dev Container. You
will need to "Rebuild Container" from the command palette after uncommenting to enable the GPU. Run nvidia-smi inside the Dev Container. If you see your GPU, you should be able to run this tool with GPU acceleration.
You should be able to run the tool with a higher --speedhack-ratio value (0.5 to 1.5).
Things that are accelerated by passing in a GPU:
- (Auto) UI rendering occurs in hardware and not CPU.
- (Nvidia-only/Auto)
replay's decoding of the forward video if a NVIDIA GPU is provided and CUDA is available. - (Nvidia-only/Manual) Optionally, on NVIDIA GPUs, you can also pass into
clip.shthe option--nv-direct-encodingto encode the captured video directly to a H.264 MP4 via the GPU. Video quality is lower, but it is quick.
Docker for Windows has a terrible memory or handle leak issue. Quit it from the black whale icon in system tray. Additionally, you may want to also shutdown Ubuntu by running wsl --shutdown from a PowerShell or Command Prompt to regain maximum performance.
While Docker for Windows is running, you may also want to click Clean Up while inside it if you want to regain some disk space.
You may want to prune images. Up to you, DIYer!
Best to use the Dev Container.
Here's the result you get when you run the clipper on a computer that's too slow or doesn't meet the requirements after all the performance "modifiers". Nothing lines up and it's slow.
From https://discord.com/channels/469524606043160576/1030241716818681897/1030341590201413664
test.mp4
It is possible to upload only a small portion of a long route for the clipper with Comma Connect's GUI.
You are strongly recommended to select a minute before, and a minute after the incident you want to clip. This will give a buffer before and after the incident to provide a margin for the clipper to render with. Think of the clipper as a fragile film projector; give it some slack and upload the files for these adjacent segments. If you do not, rendering may not start at all.
The video below shows how to do this with Comma Connect's GUI.
Screen.Recording.2022-11-25.at.2.25.15.PM.mov
Just a single shell script that runs an X11 server, and tmux commands to control the replay executable. There is faketime to make it run reliably without modifications to the pre-built openpilot that is in the image. Docker is used to just make it portable, but also easy to cleanup. Docker-Compose is used to make sure the /dev/shm size is correct and to specify the use of already pre-built images for general use or backwards compatibility use.
The real MVP is @deanlee for the replay tool in the openpilot project. The level of effort to develop the replay tool is far beyond this script. The script is just a wrapper around the replay tool to make it easy to use for clipping videos.
https://github.com/commaai/openpilot/blame/master/tools/replay/main.cc
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent