MJPEG recording locks up camera about picamera

[waveform80]

This is looking more like a firmware issue - in the lockup cases the encoder callbacks just stop and MMAL hangs when disabling the port ... weird!

[waveform80]

Reasonably confident this is a firmware issue now - when recording at 1080p the encoder callback gets called for about a dozen frames and then suddenly doesn't get called anymore. Attempts to disable the port don't return and a reboot is required to recover. Marking as an upstream issue for now.

[waveform80]

This looks awfully similar to #62...

[waveform80]

This shouldn't be closed as it's still an issue. Need to track down the exact resolution it occurs at and report it upstream. After updating the tests for #65 it appears even 720p locks up the MJPEG encoder.

[rsacchettini]

Same issue while trying to do a still image capture when in video capture mode (works like perfect in h264):

def captureToFile(camera, file):
    #camera.resolution = (2592, 1944)
    camera.capture(file,format="jpeg", use_video_port=True)

def streamVideo(camera, connection):
    print "start recording"
    camera.resolution = (640, 480)
    camera.framerate = 25
    camera.start_recording(connection, format='mjpeg')
    #

    #camera.stop_recording()
    #time.sleep(2)

    print "recording stopped"

if __name__ == "__main__":

    server_socket = socket.socket()
    server_socket.bind(('0.0.0.0', 8000))
    server_socket.listen(0)

    try:
        with picamera.PiCamera() as camera:
            camera.resolution = (640, 480)
            camera.framerate = 24

            # Start a preview and let the camera warm up for 2 seconds
            print "preview"
            #camera.start_preview()
            time.sleep(2)

            # Accept a single connection and make a file-like object out of it
            connection = server_socket.accept()[0].makefile('wb')
            print "connection made"
            captureStream = io.BytesIO()

            streamVideo(camera, connection)
            for i in range(0, 3):
                camera.wait_recording(8)
                t0 = time.clock()
                filename = "sample-%d_%s.jpeg" % (i, time.strftime("%H:%M:%S"))
                print "filename: {}", filename
                captureToFile(camera, captureStream)
                t1 = time.clock()
                diff = t1 - t0

                print ("time end of streaming => end of capture = %s" % (str(diff)))

    except KeyboardInterrupt:
        # quit
        sys.exit()

    finally:
        connection.close()
        server_socket.close()

[rsacchettini]

A simpler code to reproduce the issue:

import picamera

with picamera.PiCamera() as camera:
    camera.resolution = (800, 600)
    camera.start_preview()
    camera.start_recording('foo.mjpeg', format='mjpeg')
    camera.wait_recording(10)
    camera.capture('foo.jpg', use_video_port=True)
    print " recorded foo.jpg"
    camera.wait_recording(10)
    camera.stop_recording()
    print "stopped recording"

[waveform80]

The script posted by @rsacchettini above produces the following output in sudo vcdbg log msg (I suspect only the last line is pertinent):

...
003240.919: TV service:host side not connected, dropping notification 0x00000002, 0x00000002, 0x0000001c
145303.210: camsubs: Looking for camera 0: i2c_port = 0, led gpio = 5, power enable gpio = 21
145304.832: camsubs: Camera found OK
145306.878: gpioman: gpioman_get_pin_num: pin CAMERA_LED not defined
146048.103: mmal: mmal_vll_load: could not load VLL 'videnc.vll': 
191142.236: mmal: fail_destroy: Timed out waiting to destroy ril.video_encode

After this message is produced, killing the python process leaves the camera operational and a reboot is required to restore functionality. Interestingly, reducing the resolution to 640x480 permits the script to work. MJPEG generally seems to have issues when the resolution is beyond VGA. Currently all picamera tests with MJPEG are xfailed where the vertical resolution exceeds 480; however all tests at VGA res or below succeed (including capture-while-recording, split-recording, etc). Whether the issue is the actual resolution or the bandwidth required by high resolutions, I'm not sure.

[nikoder]

For what it's worth, I am having no problems recording 1080p MJPEG through the v4l2 interface (using python-v4l2capture) with the Pi Camera, although I am having the same problems described here with Picamera. I'm not sure if this is helpful or not, because I am not sure if they rely on the firmware in the same way.

Also, it might be worth noting that I did have to make sure that only minimal buffers were allocated by v4l2. Allocating more than 2 frames of buffer at 1080 resolution caused memory issues with my "original" model B Pi. I'm not sure if the allocation of buffers is within the control of Picamera?

Any news from upstream? I'm not sure where to look for a potential upstream issue.

If anyone happens to be aware of a firmware/picamera version combination where this works, I would be grateful to know!

[waveform80]

Interesting - I'm beginning to wonder if this has something to do with picamera's usage of a splitter after reading 6by9's fascinating comments in this thread. Stripping out the splitter isn't an option (too much functionality relies on it) but there's a possibility I might be able to make it optional (obviously in cases where it wasn't used things like multi-res recording and capture-while-recording wouldn't work).

I'll mark this for 1.9 to remind me to look into it.

[waveform80]

More information in this response too.

[mparkachov]

I came across the same issue. Currently I have model A + camera 1.3. For me even 640x480 doesn't work. The biggest resolution, which I can get working is 512x384. But when it is working, it seems to be stable. I used the same code as @rsacchettini to reproduce issue with file set to '/dev/null' to avoid interference with SD.

I don't know if this is relevant or not, but I can set camera.resolution to max values without problem, but I have to set then resize=(512,384) in start_recording() call.

[andyg24]

This is still an issue on a B+ board with latest firmware, although recording at 640x480 does work reliably.

Quick question -- how does MJPEG recording differ from continuous_capture(use_video_port=True) in JPEG format? My understanding is that MJPEG is just a concatenation of JPEG images with a few bytes of extra headers on each frame. continous_capture() seems to work reliably even at high resolutions. Would it be possible to emulate start_recording(format='mjpeg') using capture_continuous() under the hood? Would there be a performance penalty in this approach?

[andyg24]

A couple more observations:

1. I downloaded the latest version of userland, modified RaspiVid.c to output in MJPEG format instead of H264 and ran some tests. Exactly the same thing happens there. Video encodes just fine at 640 x 480. If I change resolution to 800 x 600, the camera hangs and a reboot is required. So I doubt that eliminating the splitter in picamera will change anything -- the problem appears more fundamental than that.

2. The problem is not resolution but the amount of data processed by the encoder. If I turn off the light in my room so that jpeg data compresses very well, 800 x 600 mode works just fine. As soon as I turn on the light, encoding fails.

[waveform80]

First of all, many thanks to @andyg24 for testing a modified RaspiVid.c - that's been on my todo list for weeks (as a means of determining whether removing the splitter will make a difference, and in order to have something to submit upstream). That's also a very interesting observation on whether compressibility is a factor in the crash; I wonder if @nikoder's success at 1080p with the V4L2 driver had anything to do with that?

I've also just realized I never responded to the comment above about whether picamera can control buffers, sorry! Via MMAL it can control the number and size of buffers allocated at each stage of the pipeline. At the moment it does what raspistill/raspivid do, which is to set them to the minimum or recommended size (the reason for minimum/recommended is there's a minor upstream bug: it's mentioned at the end of #73). Still, if circumstances demanded (i.e. if buffer allocation made a difference) I'd be happy to add a special case for MJPEG output.

@andyg24 - sounds like you've got plenty for an upstream bug report there, if you fancy opening one? That said, given 6by9's comments on the state of the MJPEG codec I wouldn't hold my breath waiting for a fix!

[andyg24]

Sure, I can file a bug with the userland project on github (is that the best way to report this problem upstream?)

I will share the steps to reproduce the issue here anyway.

1. Download userland from github:

wget https://github.com/raspberrypi/userland/archive/master.zip

2. Unzip the archive and go to userland-master/host_applications/linux/apps/raspicam

3. Edit RaspiVid.c:

• line 1384: change

encoder_output->format->encoding = MMAL_ENCODING_H264;
to
encoder_output->format->encoding = MMAL_ENCODING_MJPEG;

4. Comment out setting of H264 video profile (lines 1465 - 1477)

5. Compile. On Raspbian, this should work:

gcc -O3 -o mjpeg_test RaspiVid.c RaspiCamControl.c RaspiCLI.c RaspiPreview.c -I/opt/vc/include/ -I/opt/vc/include/interface/vcos/pthreads -I/opt/vc/include/interface/vmcs_host/linux -L/opt/vc/lib/ -lmmal -lmmal_components -lmmal_core -lmmal_util -lrt -lbcm_host -lvcos -lpthread

6. Now, this works:

./mjpeg_test -w 640 -h 480 -fps 15 -o test.mjpeg

7. This hangs the camera

./mjpeg_test -w 800 -h 600 -fps 15 -o /dev/null

[andyg24]

I've opened issue #208 in userland project:

raspberrypi/userland#208

There is not much activity in that project, and many bug reports haven't been acknowledged for months. I wouldn't hold my breath for this to be fixed quickly. Additionally, I don't see any MJPEG-related code in userland. Perhaps this functionality is part of the closed-sourced Broadcom driver?

[andyg24]

In the upstream discussion, a user suggested setting encoder buffer_size to a higher value. This seems to work: if I change line 465 in encoders.py to

self.output_port[0].buffer_size = 500000;

then MJPEG encoding seems to work at 1024 x 768 without wedging the camera. Don't know if this fix is reliable or if there are any downsides to a large buffer, but it's definitely an improvement.

Another suggestion in the upstream thread was to stop capture and wait for a few milliseconds before closing the encoder. I haven't tried this yet.

[andyg24]

#179 seems to do the trick. No lockups since I started testing this patch three days ago.

[waveform80]

PR #179 merged. Note to self: run the results through the test suite to see how far we can push it with the bigger buffer and then we can close this one!

[nikoder]

That's also a very interesting observation on whether compressibility is a factor in the crash; I wonder if @nikoder's success at 1080p with the V4L2 driver had anything to do with that?

No, I don't think compressibility is the cause of my success with V4L2: although I don't have the best performance (since I'm streaming through the network and the default MJPG quality level does not give enough compression for that), it does work without locking up.

But I guess this is not so relevant, now that we know that the buffer size helps (sorry for my late response :S ).

I have hacked the buffer fix, which was merged, into my Picamera installation and that allowed me to get similar results as with the v4l2 driver (including at 1080p). I have only tested it once at that resolution though. It worked many times at 720p.

There were still some problems but they might be unrelated:

• occasionally, it did not release the camera properly after recording
• when I came back to do more tests with 1080p, it failed to start recording with any resolution (red light came on and then no data arrived, but killing the python process freed it up) -> probably something else going really wrong, but unfortunately it stopped me from doing further testing

[jonasoh]

We are seeing this exact issue, using camera V2 at high resolutions. Lowering resolution and framerate seems to help. However, getting the maximum resolution would be a large benefit to us. Manually enlarging the buffer, as indicated in #179, seems to not help. Any suggestions?

[jonasoh]

Updating with our experiences. With the V2 camera, we seem to be able to avoid hangs by setting a resolution of 2592x1944 and a fixed framerate of 10 fps. Any use of framerate_range at this resolution causes the lockup on stop_recording().

It may be possible to go higher with either framerate or resolution, although we settled for these values. Maximum resolution seems to trigger the hang even when using fixed framerates.