RTSP Source adapter: stream from Axis camera doesn't reach the module about savant

[nikcleju]

Even better, I found the problem. The camera was not sending the SPS and PPS information in the stream. Because of this, h264parse was dropping every frame, which was visible when increasing GStreamer log level:

0:00:05.714218719   125 0x55b6db923f00 WARN               h264parse gsth264parse.c:1497:gst_h264_parse_handle_frame:<h264parse0> broken/invalid nal Type: 1 Slice, Size: 15917 will be dropped

I've seen several complaints on the Internet about the missing SPS and PPS in Axis streams, which lead to this dropping.

The solution is to enable the "PS Enabled" option in the Axis camera settings page, as described here (https://lup.lub.lu.se/luur/download?func=downloadFile&recordOId=8995044&fileOId=8995045):

The option PS Enabled is interesting since it places the SPS (Sequence Parameter Set)
and PPS (Picture Parameter Set) in the RTP stream. SPS and PPS contain information on a
sequence of images and a single rendered image respectively. Without the SPS and PPS the
browser does not have su�cient information to decode the video stream. It is also possible
enable this option by adding a �ag to the RTSP URL, videopsenabled=1.

On my camera this setting is in Plain Config -> Image, under H264 and H254.

This fixes it for me.

Many thanks for your support!

[bwsw]

@nikcleju
I do not see any errors. You probably connected parts the wrong way.

The best way to test everything end to end is to use this sample:
https://github.com/insight-platform/Savant/tree/develop/samples/rtsp_cam_compatibility_test

[bwsw]

Also would help if you request stream information with ffprobe <URL>. The NVDEC lacks decoding several color encoding schemes.

[bwsw]

The source of the issue can be related to yuvj420p:
https://forums.developer.nvidia.com/t/compatiblility-of-deepstream/195270

[nikcleju]

Thanks for the quick reply.

I tried with rtsp_cam_compatibility_test sample, with LOGLEVEL=debug, and it's the same problem. The logs look the same as in my first try, the debug messages like "Sending message to ZeroMQ socket" are missing.

ffprobe from the bad stream (I replaced user and pass):

ffprobe rtsp://xxxxx:[email protected]:554/axis-media/media.amp\?camera=1\&streamprofile=NC_use_H265
ffprobe version 4.4.2-0ubuntu0.22.04.1 Copyright (c) 2007-2021 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.2.0-19ubuntu1)
  configuration: --prefix=/usr --extra-version=0ubuntu0.22.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
[rtsp @ 0x5649821a3d80] RTP: dropping old packet received too late
[rtsp @ 0x5649821a3d80] max delay reached. need to consume packet
[rtsp @ 0x5649821a3d80] RTP: missed 29 packets
[rtsp @ 0x5649821a3d80] max delay reached. need to consume packet
[rtsp @ 0x5649821a3d80] RTP: missed 56 packets
[rtsp @ 0x5649821a3d80] max delay reached. need to consume packet
[rtsp @ 0x5649821a3d80] RTP: missed 37 packets
[rtsp @ 0x5649821a3d80] max delay reached. need to consume packet
[rtsp @ 0x5649821a3d80] RTP: missed 29 packets
Input #0, rtsp, from 'rtsp://xxxxx:[email protected]:554/axis-media/media.amp?camera=1&streamprofile=NC_use_H265':
  Metadata:
    title           : Session streamed with GStreamer
    comment         : rtsp-server
  Duration: N/A, start: 0.100011, bitrate: N/A
  Stream #0:0: Video: hevc (Main), yuvj420p(pc, bt709), 1952x2592 [SAR 1:1 DAR 61:81], 20 fps, 20 tbr, 90k tbn, 20 tbc

ffprobe from the good stream (this one has no user and pass):

ffprobe rtsp://192.168.0.15:554/stream2              
ffprobe version 4.4.2-0ubuntu0.22.04.1 Copyright (c) 2007-2021 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.2.0-19ubuntu1)
  configuration: --prefix=/usr --extra-version=0ubuntu0.22.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
Input #0, rtsp, from 'rtsp://192.168.0.15:554/stream2':
  Metadata:
    title           : Session streamed by "nessyMediaServer"
    comment         : h264_2
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: h264 (Main), yuvj420p(pc, bt709, progressive), 1280x720 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 90k tbn, 50 tbc
  Stream #0:1: Audio: pcm_mulaw, 8000 Hz, 1 channels, s16, 64 kb/s
  Stream #0:2: Data: none
Unsupported codec with id 0 for input stream 2

[bwsw]

This looks like a problem:

[rtsp @ 0x5649821a3d80] RTP: dropping old packet received too late
[rtsp @ 0x5649821a3d80] max delay reached. need to consume packet
[rtsp @ 0x5649821a3d80] RTP: missed 29 packets
[rtsp @ 0x5649821a3d80] max delay reached. need to consume packet
[rtsp @ 0x5649821a3d80] RTP: missed 56 packets
[rtsp @ 0x5649821a3d80] max delay reached. need to consume packet
[rtsp @ 0x5649821a3d80] RTP: missed 37 packets
[rtsp @ 0x5649821a3d80] max delay reached. need to consume packet
[rtsp @ 0x5649821a3d80] RTP: missed 29 packets

Probably you have a bad network connection.

[nikcleju]

No, no. I had forgotten to add "-rtsp_transport tcp". With this added, the RTP errors are gone.

ffprobe -rtsp_transport tcp -i rtsp://xxxxx:[email protected]:554/axis-media/media.amp\?camera=1\&streamprofile=NC_use_H265
ffprobe version 4.4.2-0ubuntu0.22.04.1 Copyright (c) 2007-2021 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.2.0-19ubuntu1)
  configuration: --prefix=/usr --extra-version=0ubuntu0.22.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
Input #0, rtsp, from 'rtsp://xxxxx:[email protected]:554/axis-media/media.amp?camera=1&streamprofile=NC_use_H265':
  Metadata:
    title           : Session streamed with GStreamer
    comment         : rtsp-server
  Duration: N/A, start: 0.050022, bitrate: N/A
  Stream #0:0: Video: hevc (Main), yuvj420p(pc, bt709), 1952x2592 [SAR 1:1 DAR 61:81], 20 fps, 20 tbr, 90k tbn, 20 tbc

Additional info. I tried the URI in the test_decode.py script from FFmpeg-Input repository, and I managed to see and save the individual frames (e.g. cv2.imwrite()):

if __name__ == '__main__':
    # s = FFMpegSource("/dev/video0",
    #                  params={"video_size": "1920x1080", "c:v": "v4l2m2m", "input_format": "mjpeg"},
    s = FFMpegSource("rtsp://xxxxx:[email protected]:554/axis-media/media.amp?camera=1", params={"rtsp_transport": "tcp"},
                     queue_len=100,
                     decode=True,
                     ffmpeg_log_level=FFmpegLogLevel.Debug)
    s.log_level = FFmpegLogLevel.Debug
    count = 0
    while True:
        try:
            p = s.video_frame()
            res = p.payload_as_bytes()
            # 1944 2592
            print(p.frame_height, p.frame_width)
            res = np.frombuffer(res, dtype=np.uint8)
            res = np.reshape(res, (p.frame_height, p.frame_width, 3))
            end = time.time()
            print(p.codec, p.pixel_format, p.queue_len, "all_time={}".format(int(end * 1000 - p.frame_received_ts)),
                  "python_time={}".format(int(end * 1000 - p.frame_processed_ts)))
            cv2.imwrite("frame_{}.jpg".format(count), res),
            count = count+1
            # cv2.imshow('Image', res)
            #if cv2.waitKey(1) & 0xFF == ord('q'):
            #    break
        except BrokenPipeError:
            print("EOS")
            break

So I think the RTSP stream is decodable after all, and the ffmpeg_input source is doing its job. Is it possible that the FFMPEG_PARAMS=rtsp_transport=tcp are somehow ignored in the adapter? My adapter docker compose service is like this:

  rtsp-pole5cam1:
    image: ghcr.io/insight-platform/savant-adapters-gstreamer:0.2.10
    entrypoint:
      - /opt/savant/adapters/gst/sources/ffmpeg.sh
    environment:
      FFMPEG_PARAMS: rtsp_transport=tcp,fflags=+genpts
      LOGLEVEL: debug
      SOURCE_ID: pole5cam1
      SYNC_OUTPUT: "true"
      URI: rtsp://xxxxx:[email protected]:554/axis-media/media.amp?camera=1&streamprofile=NC_use_H265
      ZMQ_ENDPOINT: pub+connect:ipc:///tmp/zmq-sockets/input-video.ipc
    networks:
      network: null
    restart: unless-stopped
    volumes:
      - type: volume
        source: zmq_sockets
        target: /tmp/zmq-sockets
        volume: {}
    profiles:
      - pole5cam1

[bwsw]

Yes, but in case of Savant, FFmpeg_input just passes frames to Savant without decoding. Savant decodes them with NVDEC. In your sample uou just decode frames with the FFmpeg CPU-based decoder

[bwsw]

@nikcleju if you can provide an rtsp for us privately we will check it and fix the problem if possible.

[nikcleju]

I'm not sure if I can provide a stream, it's in a private network. I'll try.

In the meantime, I've narrowed it down a little more. I've printed the GStreamer pipeline called inside ffmpeg.sh, and I'm calling directly from a terminal attached to the container.

It seems the data doesn't go through savant_parse_bin.

With the full pipeline, the data doesn't reach the zero_mq element, because I don't see any log message from it:

gst-launch-1.0 --eos-on-shutdown ffmpeg_src uri=rtsp://xxxxx:[email protected]:554/axis-media/media.amp\?camera=1\&streamprofile=NC_use_H265 queue-len=50 loglevel=debug params=rtsp_transport=tcp,fflags=+genpts ! savant_parse_bin ! fps_meter period-frames=1000 output=stdout ! zeromq_sink source-id=pole5cam1 socket=pub+connect:ipc:///tmp/zmq-sockets/input-video.ipc socket-type=DEALER bind=false sync=true ts-offset=0

output is:

 DEBUG insight::savant::ffmpeg_src::ffmpeg_src+ffmpegsrc0    > Received frame with codec hevc, PTS 4329389 and DTS 4329389.
 DEBUG insight::savant::ffmpeg_src::ffmpeg_src+ffmpegsrc0    > 0 frames in queue, 0 frames skipped.
 DEBUG insight::savant::ffmpeg_src::ffmpeg_src+ffmpegsrc0    > Pushing buffer of size 960 with PTS=4329389, DTS=4329389 and duration=18446744073709551615 to src pad.
 DEBUG insight::savant::ffmpeg_src::ffmpeg_src+ffmpegsrc0    > Receiving next frame
[NULL @ 0x7f4b78004dc0] nal_unit_type: 1(TRAIL_R), nuh_layer_id: 0, temporal_id: 0
[NULL @ 0x7f4b78004dc0] nal_unit_type: 1(TRAIL_R), nuh_layer_id: 0, temporal_id: 0
[NULL @ 0x7f4b78004dc0] nal_unit_type: 1(TRAIL_R), nuh_layer_id: 0, temporal_id: 0
[2024-03-22T22:53:09Z DEBUG ffmpeg_input] Frame info: codec_name="hevc", FPS="20/1", AVG_FPS="20/1", width=1952, height=2592, is_key=false, len=963, pts=Some(4333890), dts=Some(4333890), is_corrupt=false, pixel_format=YUVJ420P
 DEBUG insight::savant::ffmpeg_src::ffmpeg_src+ffmpegsrc0    > Received frame with codec hevc, PTS 4333890 and DTS 4333890.
 DEBUG insight::savant::ffmpeg_src::ffmpeg_src+ffmpegsrc0    > 0 frames in queue, 0 frames skipped.
 DEBUG insight::savant::ffmpeg_src::ffmpeg_src+ffmpegsrc0    > Pushing buffer of size 963 with PTS=4333890, DTS=4333890 and duration=18446744073709551615 to src pad.
 DEBUG insight::savant::ffmpeg_src::ffmpeg_src+ffmpegsrc0    > Receiving next frame
[NULL @ 0x7f4b78004dc0] nal_unit_type: 1(TRAIL_R), nuh_layer_id: 0, temporal_id: 0
[NULL @ 0x7f4b78004dc0] nal_unit_type: 1(TRAIL_R), nuh_layer_id: 0, temporal_id: 0
[NULL @ 0x7f4b78004dc0] nal_unit_type: 1(TRAIL_R), nuh_layer_id: 0, temporal_id: 0
[2024-03-22T22:53:09Z DEBUG ffmpeg_input] Frame info: codec_name="hevc", FPS="20/1", AVG_FPS="20/1", width=1952, height=2592, is_key=false, len=861, pts=Some(4338398), dts=Some(4338398), is_corrupt=false, pixel_format=YUVJ420P
 DEBUG insight::savant::ffmpeg_src::ffmpeg_src+ffmpegsrc0    > Received frame with codec hevc, PTS 4338398 and DTS 4338398.
 DEBUG insight::savant::ffmpeg_src::ffmpeg_src+ffmpegsrc0    > 0 frames in queue, 0 frames skipped.
 DEBUG insight::savant::ffmpeg_src::ffmpeg_src+ffmpegsrc0    > Pushing buffer of size 861 with PTS=4338398, DTS=4338398 and duration=18446744073709551615 to src pad.
 DEBUG insight::savant::ffmpeg_src::ffmpeg_src+ffmpegsrc0    > Receiving next frame

If however I remove savant_parse_bin from the pipeline, the data does reach zeromq_sink, I see the log messages:

root@e6f68d4a521b:/opt/savant# gst-launch-1.0 --eos-on-shutdown ffmpeg_src uri=rtsp://xxxxx:[email protected]:554/axis-media/media.amp\?camera=1\&streamprofile=NC_use_H265 queue-len=50 loglevel=debug params=rtsp_transport=tcp,fflags=+genpts ! fps_meter period-frames=1000 output=stdout ! zeromq_sink source-id=pole5cam1 socket=pub+connect:ipc:///tmp/zmq-sockets/input-video.ipc socket-type=DEALER bind=false sync=true ts-offset=0

output is:

DEBUG insight::savant::ffmpeg_src::ffmpeg_src+ffmpegsrc0    > Receiving next frame
 DEBUG insight::savant::ffmpeg_src::ffmpeg_src+ffmpegsrc0    > Received frame with codec hevc, PTS 148513 and DTS 148513.
 DEBUG insight::savant::ffmpeg_src::ffmpeg_src+ffmpegsrc0    > 21 frames in queue, 0 frames skipped.
 DEBUG insight::savant::ffmpeg_src::ffmpeg_src+ffmpegsrc0    > Pushing buffer of size 4691 with PTS=148513, DTS=148513 and duration=18446744073709551615 to src pad.
[NULL @ 0x7f1664004dc0] nal_unit_type: 1(TRAIL_R), nuh_layer_id: 0, temporal_id: 0
[NULL @ 0x7f1664004dc0] nal_unit_type: 1(TRAIL_R), nuh_layer_id: 0, temporal_id: 0
[NULL @ 0x7f1664004dc0] nal_unit_type: 1(TRAIL_R), nuh_layer_id: 0, temporal_id: 0
[2024-03-22T22:58:24Z DEBUG ffmpeg_input] Frame info: codec_name="hevc", FPS="20/1", AVG_FPS="20/1", width=1952, height=2592, is_key=false, len=2648, pts=Some(247527), dts=Some(247527), is_corrupt=false, pixel_format=YUVJ420P
 DEBUG insight::savant::zeromq_sink::zeromq_sink+zeromqsink0 > Processing frame 1650144444 of size 4691
 DEBUG savant_rs::zeromq::writer                             > Sending message to ZeroMQ socket: [112, 111, 108, 101, 53, 99, 97, 109, 49] Message { meta: MessageMeta { lib_version: "0.2.16", routing_labels: [], span_context: PropagatedContext({}), seq_id: 34 }, payload: VideoFrame(VideoFrameProxy { inner: SavantArcRwLock(SavantRwLock(RwLock { data: VideoFrame { previous_frame_seq_id: None, source_id: "pole5cam1", uuid: 196466479191291598917597108577656102689, creation_timestamp_ns: 1711148304728550955, framerate: "20/1", width: 1952, height: 2592, transcoding_method: Copy, codec: Some("hevc"), keyframe: Some(false), time_base: (1, 1000000000), pts: 1650144444, dts: Some(1650144444), duration: None, content: External(ExternalFrame { method: "zeromq", location: None }), transformations: [InitialSize(1952, 2592)], attributes: [], objects: {}, max_object_id: 0 } })) }) }
 DEBUG savant_rs::zeromq::writer                             > Message sent to ZeroMQ socket. Time spent: 0 ms

In this case I'm also seeing the data being received by zeromq on the module side, but afterwards it is not processed through the pipeline (maybe it can't be decoded because h264parse was missing).

Any other suggestions on what I could investigate or try? I'm pretty much out of ideas, and at the limit of my GStreamer/Savant knowledge.

[bwsw]

@nikcleju you did a great job. I will ask a developer to take a look at logs, etc on Monday. Please share access to the stream or give us access to the computer if possible.

[bwsw]

@nikcleju awesome investigation!

[nikcleju]

awesome library, Savant!

[grimen]

Great discovery, this is highly relevant for us since we will run on a wide range of AXIS cameras.