roboflow / video-inference

Example showing how to do inference on a video file with Roboflow Infer

🚨🚨🚨 REPO DEPRECATED 🚨🚨🚨

• For Realtime Video (or webcam): use Roboflow inference; docs for realtime video streaming are available here.
• For Async/Recorded Video: Roboflow now provides a new video inference API, please visit the Hosted Video Inference docs to learn more.

Roboflow Video Inference

Example script to do inference on a video file with Roboflow Infer

An example video (gif) generated with this utility script from a Roboflow model with the following command:

ROBOFLOW_KEY=xxxxxxxx ./infer.sh rf-aquarium-merged/3 IMG_3203.mov fish.gif --fps_in 3 --fps_out 12 --scale 4

Getting Started

Colab Tutorial Here:

Installation

The shell script you need is infer.sh in this repo.

To use it

• copy that file to your local machine,
• add execute permissions chmod +x infer.sh,
• install the requirements (see below)
• and then use it:

# Simple usage with ROBOFLOW_KEY env var set inline
ROBOFLOW_KEY=xxxxx ./infer.sh [YOUR_DATASET]/[YOUR_VERSION] video_in.mp4 video_out.mov

To obtain your ROBOFLOW_KEY, go to your account settings page and copy your API Key (not your Publishable Key). Treat this API Key like a password; it gives access to your Roboflow account and is meant to remain a secret.

The model ID ([YOUR_DATASET]/[YOUR_VERSION] in the command above) is obtained from the Roboflow UI after training. Click "Example Web App" in the "Use Your Trained Model" section and copy it from the pre-filled text field at the top of the page.

Requirements

The script's main requirements are ffmpeg, base64, and curl. On most modern Unix-like operating systems, you will only need to install ffmpeg because the other requirements are preinstalled.

macOS

Already includes base64 and curl.

Use homebrew to install ffmpeg: brew install ffmpeg

Ubuntu

Already includes base64 and curl.

Use apt to install ffmpeg: sudo apt install ffmpeg

Windows

Download ffmpeg from their website and install git for Windows which conveniently includes base64 and curl.

Options

A ROBOFLOW_KEY environment variable with your Roboflow API Key is required.

The infer.sh script accepts the following parameters:

Parameter	Example	Description
--host	--host "http://localhost:9001"	The Roboflow Infer host; set for On-Device Inference (default: https://infer.roboflow.com).
--confidence	--confidence 50	The minimum threshold for the model to output box predictions (default: 50).
--overlap	--overlap 50	The maximum amount two predicted boxes of the same class can intersect before being combined (default: 50).
--stroke	--stroke 5	The thickness of the predicted bounding boxes (default: 5).
--labels	--labels	Enable printing the class names (implicit default: off)
--classes	--classes car	A comma separated (no whitespace) list of classes to predict (default: show all).
--fps_in	--fps_in 6	The sample rate from the input video in frames per second (default: 6).
--fps_out	--fps_out 24	The render speed; setting higher than fps_in gives a timelapse effect (default: 24).
--scale	--scale 2	The amount to shrink the video; eg 2 to make video_out width and height 2x smaller than video_in (default: 1).
--tmp	--tmp .	The tmp directory; must be writable (default: /tmp).
--retries	--retries 3	The number of times to retry a failed inference (default: 3).
--parallel	--parallel 8	The number of concurrent frames to send to the model (default: 8).
model	xx-mymodel--1	The Roboflow model to use for inference (required).
video_in	video_in.mp4	The input video file (required).
video_out	video_out.mp4	The output video file (required). Can be of any format supported by ffmpeg's encoder (eg .mov, .mp4, .gif).

Building

The infer script is built with argbash (a code generator that adds smart parameter parsing for bash scripts) from source.m4.

To build, clone this repo, install argbash, then run the following in the repo's top-level directory:

argbash source.m4 -o infer.sh

You can then use your newly generated infer.sh as described above.

License

This example repo is released under an MIT License. You will need to train a model with Roboflow Train which is goverened by the Roboflow Terms of Service.

**Note: Update from November 2023 - Roboflow now provides a new video inference API, please visit this link to learn more. This repository is archived. **

video-inference's Issues

Repo not updated for new detect.roboflow.com API format

This repo was programmed prior to the change over to detect.roboflow from infer.roboflow and the new URL scheme and api_key parameter name. The inference URL needs to be updated according to the new documentation: https://docs.roboflow.com/inference/hosted-api

Doesn't work for segmentation

Hello, I'm trying to get this running on my segmentation model. I'm making progress, but have now run out of free credits so can't debug any further!

On first attempt to run the script, I got the error No JPEG data found in image (similar to #2).

Checking the docs, I see segmentation models run on a different url: https://detect.roboflow.com

Correcting that, I also noticed that the url returns json data containing b64 encoded image data, so I used jq and base64 to get it into a file.

Finally I noticed the received image data is in png format, so I changed the file suffix to match, to make ffmpeg happy.

At that point the script completes without error and produces a video, however the video is entirely black. I'm not sure if that is due to a remaining bug, or if it is because I am out of credits.

Here's my full diff right now, could probably be cleaned up a bit:

diff --git a/infer.sh b/infer.sh
index 3dbe286..9d54493 100755
--- a/infer.sh
+++ b/infer.sh
@@ -56,7 +56,7 @@ _arg_model=
 _arg_video_in=
 _arg_video_out=
 # THE DEFAULTS INITIALIZATION - OPTIONALS
-_arg_host="https://detect.roboflow.com"
+_arg_host="https://segment.roboflow.com"
 _arg_confidence="50"
 _arg_overlap="50"
 _arg_stroke="5"
@@ -343,7 +343,7 @@ done
     exit 1;
 }
 
-inference_url="$host/$model?api_key=$ROBOFLOW_KEY&format=image&confidence=$confidence&overlap=$overlap&stroke=$stroke"
+inference_url="$host/$model?api_key=$ROBOFLOW_KEY"
 if [ $labels = "on" ]; then
     inference_url="$inference_url&labels=on"
 fi
@@ -393,14 +393,14 @@ do
             if [ ! -z "$verbose" ]; then
                 echo "Running inference on frame $f..."
             fi
-            cat $tmp/roboflow_in/$f | base64 | curl -s -d @- $inference_url > "$tmp/roboflow_out/$f" &
+            cat $tmp/roboflow_in/$f | base64 | curl -s -d @- $inference_url | jq -r .segmentation_mask | base64 --decode > "$tmp/roboflow_out/$f.png"
         fi
     done
 done
 
 rm -f $out
 echo "Rendering final video ($out)."
-ffmpeg -i $tmp/roboflow_out/frame%05d.jpg -vf fps=$fps_out $out
+ffmpeg -i $tmp/roboflow_out/frame%05d.jpg.png -vf fps=$fps_out $out
 
 rm -rf $tmp/roboflow_in
 rm -rf $tmp/roboflow_out

Any chance you could reimburse me some credits in exchange for this bug report? I wanted to work on my project, but ended up spending all my time and credits debugging this script! My roboflow account is associated with this github account.

If the intention is not to support segmentation models with this script, at least it would be nice if that was indicated clearly in the README, such that others do not lose time and credits as I did.

오류입니다.

Simple usage with ROBOFLOW_KEY env var set inline

ROBOFLOW_KEY=xxxxx ./infer.sh [YOUR_DATASET]/[YOUR_VERSION] video_in.mp4 video_out.mov 이 코드를 vscode에서 실행 중인데요, ./infer.sh 이 부분 ./에서 계속 오류가 납니다. 작업파일과 같은 경로에 있는데도 말이지요. 왜 그럴까요?

No JPEG data found in image

Full error

[mjpeg @ 0x7f9713809400] No JPEG data found in image
Error while decoding stream #0:0: Invalid data found when processing input
Cannot determine format of input stream 0:0 after EOF
Error marking filters as finished

If you get this error, you probably tried to infer against a model URL that doesn't exist.

We should surface this and not let the script try to compile the inference video without output frames