kbarni / lccv

LibCamera wrapper library for OpenCV

Researcher in image processing at the IMS Bordeaux Laboratory, my main research topic is proximal sensing in agricultural applications.

This is the place for the open-source projects I build, mostly in my free time. These range from simple exercices to more complex libraries.

Some of the tools that I use:

• C++/Python for programming
• Qt/QML for graphical user interfaces and mobile development
• OpenCV library
• Deep neural networks
• GPU programming with CUDA
• Arduino and microcontrollers
• Embedded systems for conception of new cameras

lccv's Issues

Cannot compile with newer libcamera version

This is the commit that broke it I think:
raspberrypi/libcamera@26a4b83

I solved it locally by replacing using StreamRoles = libcamera::StreamRoles; with using StreamRoles = std::vector<libcamera::StreamRole>;.

imshow size

Dear @kbarni,

Thanks for the nice wrapper! It is just what we needed. It requires less CPU load compared to GStreamer.
If it is ok with you, I like to use your wrapper on my GitHub page as the default camera interface in my deep learning apps for the Raspberry Pi. Of course, I will give a credit link to your GitHub repo.

On little remark.
I had to remove this line cv::namedWindow("Image",cv::WINDOW_NORMAL); in the takevideo.cpp.
Otherwise, the imshow didn't resize to the given dimensions but stuck to a small initial square.

libopencv_core.so.405, needed by /usr/local/lib/arm-linux-gnueabihf/liblccv.so

Hello!

Apologies for raising this user/operator error as an issue but I need your indulgence, please.

I am using Code::Blocks with your sample takephoto with resolution as the only modification. The code compiles without any errors or warnings (i.e. all include files are pulled in correctly). Unfortunately, the build fails with the following message:

||warning: libopencv_core.so.405, needed by /usr/local/lib/arm-linux-gnueabihf/liblccv.so, may conflict with libopencv_core.so.4.5|

The linker options include declarations for library files as follows:

/usr/lib/arm-linux-gnueabihf/libcamera.so /usr/local/lib/arm-linux-gnueabihf/liblccv.so /usr/local/lib/libopencv_core.so

LCCV was built using your CMake instructions. Unfortunately, a modified CMake (excluding takevideo and correctly referencing the OpenCV and libcamera files) wouldn't build the example as a standalone executable because it couldn't find the OpenCV core library.

The platform is Raspberry Pi Zero under Bullseye. I can use the camera with GStreamer but as you can understand a lighter wrapper would be best for RPi0 work. I've read the Q-Engineering tutorial using your library but I guess I got myself into knots trying to adapt your CMakeLists.txt for equivalent use in Code::Blocks. The only extra statement in the CMakeLists.txt was the reference to the OpenCV library (and deleting the takevideo statement).

As you can imagine, I'm very naive about CMake/CCMake and Code::Blocks. I know that your repository should not be the first resort for 3rd party guidance but by adapting to your instructions for my simple exercise I should be able to get going with my use of the Raspberry Pi Camera under Bullseye (and avoid the heavy lifting from GStreamer). Getting LCCV to work under Code::Blocks will be a very big step for me. Thanks for your understanding.

Kind regards.

P.S.

The full error message at the Build step is:

||warning: libopencv_core.so.405, needed by /usr/local/lib/arm-linux-gnueabihf/liblccv.so, may conflict with libopencv_core.so.4.5|

I am not designating any version suffix for shared libraries in Code::Blocks build options.

Memcpy should be avoided.

At : https://github.com/kbarni/LCCV/blob/main/src/lccv.cpp#L190, there is a memcpy of the captured image into a new buffer.

Can OpenCV be given the buffer directly? memcpy on image buffers should be avoided, and libcamera aims to support zero-copy usecases.

cam.captureFrame(image) allocates memory on each call, out of memory error

Example project: takephoto.cpp

Add loop:

for (int i=0;i<100;i++) {
if(!cam.captureFrame(image)){
std::cout<<"Camera error"<<std::endl;
}
}

Crash with out of mem error:
ERROR RPI raspberrypi.cpp:817 Failed to allocate buffers

libcamera_app.cpp fd.fd() change to fd.get()

Thank you for making this. I've spent a while trying to get NV12 format out of libcamera. Ended up trying to memcpy the two planes together but that didnt do it.

Anyway, when I build this the FrameBuffer::Plane::SharedFD didnt want to build with .fd.fd().

My shared_fd.h has .fd() member under private, method get() is public.

Compiled and ran after changing this.

make fails

Trying to perform make and get the following error:
In file included from /home/pi/cpp-projects/LCCV/include/lccv.hpp:9,
from /home/pi/cpp-projects/LCCV/src/lccv.cpp:1:
/home/pi/cpp-projects/LCCV/include/libcamera_app.hpp:53:33: error: ‘StreamRoles’ in namespace ‘libcamera’ does not name a type; did you mean ‘StreamRole’?
53 | using StreamRoles = libcamera::StreamRoles;
| ^~~~~~~~~~~
| StreamRole
compilation terminated due to -Wfatal-errors.
make[2]: *** [CMakeFiles/liblccv.dir/build.make:82: CMakeFiles/liblccv.dir/src/lccv.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:95: CMakeFiles/liblccv.dir/all] Error 2

Did I miss doing something in prep?

Debian 12(bookworm), gcc-12, atomic and memory_order

Trying to compile on a system running bookworm with the default g++ (g++ (Raspbian 12.2.0-14+rpi1) 12.2.0). I get the error
In file included from /usr/include/c++/12/atomic:41,
from /home/cdd/projects/LCCV/include/lccv.hpp:5,
from /home/cdd/projects/LCCV/src/lccv.cpp:1:
In member function ‘std::__atomic_base<_IntTp>::__int_type std::__atomic_base<_IntTp>::load(std::memory_order) const [with _ITp = bool]’,
inlined from ‘bool std::atomic::load(std::memory_order) const’ at /usr/include/c++/12/atomic:112:26,
inlined from ‘bool lccv::PiCamera::startVideo()’ at /home/cdd/projects/LCCV/src/lccv.cpp:102:20:
/usr/include/c++/12/bits/atomic_base.h:488:31: error: invalid memory model ‘memory_order_release’ for ‘unsigned char __atomic_load_1(const volatile void*, int)’ [-Werror=invalid-memory-model]
488 | return __atomic_load_n(&_M_i, int(__m));
| ~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~

From what I've read "memory_order" should be one of std::memory_order_relaxed, std::memory_order_consume, std::memory_order_acquire and std::memory_order_seq_cst in an atomic::load.

I've changed line 102 of lccv.cpp from
if(running.load(std::memory_order_release)){
to
if(running.load(std::memory_order_consume)){
lccv now compiles but I have no idea if std::memory_order_release is the correct value to use.

running examples terminates with camera not available

Hello,
Here's the output when I try to run takephoto...

0
Opening camera...
[186:44:12.125952427] [14582] INFO Camera camera_manager.cpp:293 libcamera v0.0.0+3415-76530215
[186:44:12.154250783] [14591] WARN CameraSensorProperties camera_sensor_properties.cpp:141 No static properties available for 'imx477'
[186:44:12.154330226] [14591] WARN CameraSensorProperties camera_sensor_properties.cpp:143 Please consider updating the camera sensor properties database
[186:44:12.154495149] [14591] ERROR CameraSensor camera_sensor.cpp:551 'imx477 10-001a': Camera sensor does not support test pattern modes.
[186:44:12.321544655] [14591] INFO RPI raspberrypi.cpp:1317 Registered camera /base/soc/i2c0mux/i2c@1/imx477@1a to Unicam device /dev/media2 and ISP device /dev/media0
terminate called after throwing an instance of 'std::runtime_error'
what(): selected camera is not available
Aborted

"libcamera-hello -t 0" has similar startup INFO/WARN/ERROR/RPI messages, but then proceeds to work. "takevideo" has the same problem. Any suggestions as to what's happening? Thanks!

BTW; the cam.options->width and height for 4056, 3040 were fixed to photo_width and photo_height, as optimal for imx477, with the 2028, 1520 lines commented out.

plane.fd.fd() has changed to plane.fd.get()

Hi
depending on the version of libcamera you use the usage of plane() has changed.
My libcamera version is:
pi@campi:~/LCCV/build $ cmake ..
-- Platform: armhf
-- libcamera library found:
-- version: 1.0
-- libraries: /usr/local/lib/arm-linux-gnueabihf/libcamera.so;/usr/local/lib/arm-linux-gnueabihf/libcamera-base.so
-- include path: /usr/local/include/libcamera
-- Configuring done
-- Generating done
-- Build files have been written to: /home/pi/LCCV/build

With this I go a compilation error:

/home/pi/LCCV/src/libcamera_app.cpp: In member function ‘void LibcameraApp::setupCapture()’:
/home/pi/LCCV/src/libcamera_app.cpp:556:54: error: ‘const class libcamera::SharedFD’ has no member named ‘fd’; did you mean ‘std::shared_ptrlibcamera::SharedFD::Descriptor libcamera::SharedFD::fd_’? (not accessible from this context)
556 | if (i == buffer->planes().size() - 1 || plane.fd.fd() != buffer->planes()[i + 1].fd.fd())
| ^~
compilation terminated due to -Wfatal-errors.
make[2]: *** [CMakeFiles/liblccv.dir/build.make:95: CMakeFiles/liblccv.dir/src/libcamera_app.cpp.o] Fehler 1
make[1]: *** [CMakeFiles/Makefile2:95: CMakeFiles/liblccv.dir/all] Fehler 2
make: *** [Makefile:171: all] Fehler 2

So changing following lines in libcamera_apps.cpp from plane().fd.fd() to plane().fd.get() is now working for me:

void LibcameraApp::setupCapture()
...
if (i == buffer->planes().size() - 1 || plane.fd.get() != buffer->planes()[i + 1].fd.get())
{
void *memory = mmap(NULL, buffer_size, PROT_READ | PROT_WRITE, MAP_SHARED, plane.fd.get(), 0);
...

Best Regards and thank's a lot for you work!

Discussion: Fast RAW image acquisition

Hi,

I am making a simple ROS wrapper for the libcamera to grab images from the Raspberry Pi HQ camera and publish them as in a ROS topic.
I attempted to do as suggested in the wiki, but capturePhoto(cv::Mat &) is slow due to the continuous start & stop of the camera.

I wonder if you have any suggestions on how to start the camera and continuously capture images. I did a quick attempt to just comment the Teardown and Close calls inside capturePhoto but it crashes due to "Multiple IPAManager objects not allowed"

Thanks

get highest resolution?

how to set pi camera's highest resolution automatically?
Thank you.

Compilation breaks on kernel version 5.15.84

After running rpi-update, installing dependencies for LCCV and compiling it fails with this:

In file included from /home/pi/LCCV/include/lccv.hpp:9,
                 from /home/pi/LCCV/src/lccv.cpp:1:
/home/pi/LCCV/include/libcamera_app.hpp: In constructor ‘FrameInfo::FrameInfo(libcamera::ControlList&)’:
/home/pi/LCCV/include/libcamera_app.hpp:194:20: error: invalid type argument of unary ‘*’ (have ‘int’)
  194 |    exposure_time = *exp;
      |                    ^~~~
compilation terminated due to -Wfatal-errors.
make[2]: *** [CMakeFiles/liblccv.dir/build.make:82: CMakeFiles/liblccv.dir/src/lccv.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:95: CMakeFiles/liblccv.dir/all] Error 2
make: *** [Makefile:171: all] Error 2

After reading old issues it seemed like this also was an issue back when libcamera had an update, but this was seemingly fixed. It seems odd then, after keeping my system up to date, that this appears again.

Kernel version: 5.15.84-v7l+
GCC version: 10.2.1
CMake version: 3.18.4
libcamera version (from apt show libcamera-dev): 0~git20221130+0684c373-1

Any idea what causes this? Should I reinstall Bullseye?

Implementation of Manual Focus Control for Raspberry V3 Cameras

Hi,

First of all, thank you for the great work you have done with this project.
I am wondering if there are any plans to implement the option for manual focus control on Raspberry V3 cameras. Currently, the autofocus functionality is excellent, but having the ability to adjust it manually would be very useful for some specific projects.
Is there any update or future plan for this feature?

Thank you in advance for your attention and any information you can provide.

libcamera capitalisation throughout

Really trivial: libcamera is always supposed to be lower case.

question

hi,

does it also work for raspberry pi zero 2 w?

compilation fails at libcamera_app.cpp lines 451 and 453

After the last libcamera updates for raspberry bullseye "src/libcamera_app.cpp" gives a compiler error at lines 451 and 453.

Just replace in both lines fd.fd() with fd.get().

Reference: #1

640x480 cropped

When taking a video you can set the video width and height.

    cam.options->video_width=3280;
    cam.options->video_height=2464;

Only in the 640 x 480 mode, the image is cropped to the centre of the screen, while all other resolutions have the same scene.

It would be very convenient if 640 x 480 also covered the whole scene. Now I have to stream a larger format and resize it by OpenCV before processing further. Better to let the hardware work than precious computing power.

How to use to read a video .avi file?

I would like to read a video .avi file instead of getting frames from a camera stream. How would I do that with lccv; or do I simply use opencv's normal method to read a video file.

Unable to compile after update of libcamera

Hello,
everything was working really well until today in the morning. I updated libcamera (with development packages). Then I tried to compile lccv, but with no success. The error I've got when I run make is:

pi@campi:~/LCCV/build $ make
[ 25%] Building CXX object CMakeFiles/liblccv.dir/src/lccv.cpp.o
In file included from /home/pi/LCCV/include/lccv.hpp:9,
from /home/pi/LCCV/src/lccv.cpp:1:
/home/pi/LCCV/include/libcamera_app.hpp: In constructor ‘FrameInfo::FrameInfo(libcamera::ControlList&)’:
/home/pi/LCCV/include/libcamera_app.hpp:192:43: error: cannot convert ‘const libcamera::Control’ to ‘unsigned int’
192 | if (ctrls.contains(libcamera::controls::ExposureTime))
| ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~
| |
| const libcamera::Control
compilation terminated due to -Wfatal-errors.
make[2]: *** [CMakeFiles/liblccv.dir/build.make:82: CMakeFiles/liblccv.dir/src/lccv.cpp.o] Fehler 1
make[1]: *** [CMakeFiles/Makefile2:95: CMakeFiles/liblccv.dir/all] Fehler 2
make: *** [Makefile:171: all] Fehler 2

Please could you take a look how this could probably be solved?

Many thanks and BR.

Error trying to run sample takevideo

I'm trying to run the sample. I've compiled and linked it after building the liblccv.so but I get error:

./takevideo: error while loading shared libraries: liblccv.so: cannot open shared object file: No such file or directory

Forcing raw video stream

It might be out of scope for this project but enabling raw frames would indeed be helpful for experimenting with different ways to handle pixel data. Would forcing the encoder to send YUV420 bayer frames be equivalent to this, or at the least close?

Discussion points

A couple more questions, as there's no direct discussions form here:

• libcamera bindings to OpenCV are not specific to Raspberry Pi. This will work on any libcamera supported platform, including the Intel IPU3, Rockchip RK3399, and very soon a range of Mediatek and NXP platforms (And UVC) as well as the Raspberry Pi.

• Can this work be done directly within the OpenCV project ? An issue was opened in their repository at opencv/opencv#21653. Adding libcamera support directly to OpenCV (with review and support from the OpenCV developers) will provide a better integration.

Unable to compile

I am unable to compile, My OS is below------------------------------------------------------------------
secam@secam:~/work/LCCV/build $ cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 12 (bookworm)"
NAME="Raspbian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

Error is below ------------------------------------------------------------------------------------------------------------
secam@secam:/work/LCCV/build $ make -j4
[ 75%] Building CXX object CMakeFiles/liblccv.dir/src/libcamera_app.cpp.o
[ 75%] Building CXX object CMakeFiles/liblccv.dir/src/lccv.cpp.o
[ 75%] Building CXX object CMakeFiles/liblccv.dir/src/libcamera_app_options.cpp.o
/home/secam/work/LCCV/src/libcamera_app.cpp: In member function ‘void LibcameraApp::ConfigureStill(unsigned int)’:
/home/secam/work/LCCV/src/libcamera_app.cpp:104:21: error: ‘class libcamera::CameraConfiguration’ has no member named ‘transform’
104 | configuration_->transform = options_->transform;
| ^~~~~~~~~
compilation terminated due to -Wfatal-errors.
make[2]: *** [CMakeFiles/liblccv.dir/build.make:90: CMakeFiles/liblccv.dir/src/libcamera_app.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
In file included from /usr/include/c++/12/atomic:41,
from /home/secam/work/LCCV/include/lccv.hpp:5,
from /home/secam/work/LCCV/src/lccv.cpp:1:
In member function ‘std::__atomic_base<_IntTp>::__int_type std::__atomic_base<_IntTp>::load(std::memory_order) const [with _ITp = bool]’,
inlined from ‘bool std::atomic::load(std::memory_order) const’ at /usr/include/c++/12/atomic:112:26,
inlined from ‘bool lccv::PiCamera::startVideo()’ at /home/secam/work/LCCV/src/lccv.cpp:102:20:
/usr/include/c++/12/bits/atomic_base.h:488:31: error: invalid memory model ‘memory_order_release’ for ‘unsigned char __atomic_load_1(const volatile void*, int)’ [-Werror=invalid-memory-model]
488 | return __atomic_load_n(&_M_i, int(__m));
| ~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~
compilation terminated due to -Wfatal-errors.
cc1plus: all warnings being treated as errors
make[2]: *** [CMakeFiles/liblccv.dir/build.make:76: CMakeFiles/liblccv.dir/src/lccv.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/liblccv.dir/all] Error 2
make: *** [Makefile:156: all] Error 2
secam@secam:/work/LCCV/build $

Can any one guide me?

FrameBufferAllocator undefined symbol

Hello,
I am getting weird error when i try to run LCCV in my project.

undefined symbol: _ZNK9libcamera11FrameBuffer6planesEv

I think i fails after camera streams configuration done. In libcamera_app.cpp line 421.

Any ideas?

Cannot find the dynamic library

Hi, after compiling and installing the library, I cannot execute any app that links LCCV, not even the examples. It says:

error while loading shared libraries: liblccv.so: cannot open shared object file: No such file or directory

Not sure what's wrong. I tried searching for the LCCV using ldconfig and it is there in /usr/local/lib/arm-linux-gnueabihf. I also tried explicitly adding it to LD_LIBRARY_PATH but that did not change anything either.

Sorry if this is a dumb question. A linux noob here...