As discussed in #13, connecting to a VStarcam camera and feeding its frames to ffmpeg produced ffmpeg errors. From discussion there:
@LucasZanella wrote:
However, on my app, while the frame producing works, passing retina::codec::VideoFrame::data().borrow()
to the ffmpeg nal units parser sometimes reject, and sometimes parse and send to the decoder, which procues
[h264 @ 0x559374083fc0] non-existing PPS 16 referenced
[h264 @ 0x559374083fc0] Invalid NAL unit 0, skipping.
[h264 @ 0x559374083fc0] Invalid NAL unit 0, skipping.
[h264 @ 0x559374083fc0] Invalid NAL unit 0, skipping.
[h264 @ 0x559374083fc0] Invalid NAL unit 0, skipping.
[h264 @ 0x559374083fc0] no frame!
Here's one VideoFrame:
[2021-07-27T19:52:20Z INFO liborwell::rtsp::retina_client] video frame: VideoFrame { timestamp: 189124 (mod-2^32: 189124), npt 2.061, start_ctx: RtspMessageContext { pos: 42441, received_wall: WallTime(Timespec { sec: 1627415540, nsec: 361449153 }), received: Instant { tv_sec: 421014, tv_nsec: 736674930 } }, end_ctx: RtspMessageContext { pos: 42441, received_wall: WallTime(Timespec { sec: 1627415540, nsec: 361449153 }), received: Instant { tv_sec: 421014, tv_nsec: 736674930 } }, loss: 0, new_parameters: None, is_random_access_point: false, is_disposable: false, data_len: 383 }
I wrote:
I haven't tried feeding video directly from retina to ffmpeg yet, but in principle it should work. The frames should be fine to pass to ffmpeg. How are you setting up the stream with ffmpeg? You'll likely need to pass it the extra_data from VideoParameters.
The log messages from ffmpeg suggest it's not seeing a valid stream—NAL unit types should never be 0, and I think it's rare for the PPS id to be 16 rather than 0. But maybe the problem is just that without the extra data, ffmpeg is expecting a stream in Annex B format, and my code is passing it instead in AVC format. (The former means that NAL units are separated by the bytes 00 00 01, and the latter means that each NAL unit is preceded by its length in bytes as a big-endian number which can be 2, 3, or 4 bytes long. My code uses 4 bytes.) If you prefer to get Annex B data, it'd be possible to add a knob to retina to tell it that. Or conversion isn't terribly difficult: you can scan through NAL units and change the prefix to be 00 00 01.
I suppose I could add a retina example that decodes with ffmpeg into raw images or something. What ffmpeg crate are you using?
When you don't get the packet follows marked packet with same timestamp error, have you tried saving a .mp4 and playing it back in your favorite video player? Does it work?
@LucasZanella wrote:
For ffmpeg I'm using https://github.com/lucaszanella/rust-ffmpeg-1 which uses https://github.com/lucaszanella/rust-ffmpeg-sys-1 (this one is not needed, I just added some vdpau linking stuff, the original could be used). I had to modify the rust-ffmpeg-1
to add support for ffmpeg's av_parser_parse2
which parses the individual nal units. The original project doe snot have this and he doesn't want to maintain. My patch is very experimental.
I haven't tried feeding video directly from retina to ffmpeg yet, but in principle it should work. The frames should be fine to pass to ffmpeg. How are you setting up the stream with ffmpeg? You'll likely need to pass it the extra_data from VideoParameters.
I've never needed to pass additional parameters to ffmpeg, just the nal units. I extracted the h264 bitstream from a big buck bunny .mp4 file and passed to ffmpeg calling av_parser_parse2
to break into individual nal units and then passed those units using avcodec_send_packet
and it works. The same process is not working for retina. When my code used to be all C++, I used to pass the output of ZLMediaKit to ffmpeg in this way also and it worked.
Even though av_parser_parse2
has the option to pass pts, dts, etc, I never used but I'll read more about these parameters.
VideoParameters
debug:
Some(Video(VideoParameters { rfc6381_codec: "avc1.4D002A", pixel_dimensions: (1920, 1080), pixel_aspect_ratio: None, frame_rate: Some((2, 15)), extra_data: Length: 41 (0x29) bytes
0000: 01 4d 00 2a ff e1 00 1a 67 4d 00 2a 9d a8 1e 00 .M.*....gM.*....
0010: 89 f9 66 e0 20 20 28 00 00 03 00 08 00 00 03 00 ..f. (.........
0020: 7c 20 01 00 04 68 ee 3c 80 | ...h.<. }))
I've sent you a dump of the camera via email.
If you prefer to get Annex B data, it'd be possible to add a knob to retina to tell it that. Or conversion isn't terribly difficult: you can scan through NAL units and change the prefix to be 00 00 01.
do you have experience in which types the rtsp clients out there do these things? I've never took a deep look on how ZLMediaKit does, I simply used it and now I'm getting deeper into RTSP/RTP/h264/etc because rust had no rtsp clients so I had to make one.
This is how I extracted the big buck bunny to make it work:
ffmpeg -i BigBuckBunny_512kb.mp4 -vbsf h264_mp4toannexb -vcodec copy -an big_buck_bunny_1280_720.h264
as you see by h264_mp4toannexb
, it's as you supposed.
May I know why you use the AVC format in your code? Isn't the Annex B proper for streaming?