Refresh `kivy.core.camera.camera_android` to use a recent Android API about kivy

[Samael-TLB]

Recently, for an app I have built a camera provider of android based on camerax as Camera2 needs a lot of device specific edge cases to be dealt with. Now, I'm experimenting with the video use case of camerax as that is still experimental.

So, I would like to work on this as I already have the code ready just need a few discussions and experimentations for video use case.

[RobertFlatt]

My experience is that a pure Java camera provider works well https://github.com/Android-for-Python/camerax_provider
(the cameras video api has been updated since this was written, but that will not change the experience).

Aside from performance, some of the newer Android apis implement callbacks as a reference to an instance of a "provider" Java class. It is not possible to pass a reference to PythonJavaClass in this case as this is Python not Java. The workarounds compound, it is much simpler to implement the full provider in Java or Kotlin (there are examples!). The provider probably belongs as a recipe. Try it here https://pypi.org/project/camera4kivy/ , Here is a simple test case https://github.com/Android-for-Python/c4k_photo_example

The camera widget api must be extended as file capture is performed by the provider not the widget (except screenshot). In addition users expect an image analysis api - they want a camera as part of some other behavior. Also the widget must handle various aspect ratio, orientation, pause/resume , pinch-spread zoom, and tap focus.

Personally I view Kivy camera widget as great for taking screenshots from a web cam and not much else, very 2000. I choose to ignore Kivy camera widget and implement my own widget, my thinking at the time was backwards compatibility would be hard and unfortunately might be expected. For example Camerax is not usable (including connect) till one time-step after Kivy on_start(), whereas Kivy camera assumes connect at build(). This and the threaded implementation in Camerax means the api must be at least partially event driven.

Finally for performance a major consideration is the size of any Kivy frame buffer, the hardware camera and screen lead to user expectations of high resolution, too large a Kivy frame buffer may lead to preview latency issues. The Kivy camera bypasses this by defaulting to 640x480, this would not make a mobile user happy. As far as I remember it worked up HD buffer, maybe my Kivy buffer implementation could have been optimized I didn't look at it.

Have fun with your project. This comment is just a drive by, I do not typically visit here.

[Samael-TLB]

Thanks for the assignment. In this regards, I would like to express a few of my findings in the implementation. So, if we go for a python implementation through pyjnius there's no signficant performance penalty even with preview+imagecapture+imageanalysis+video (note video is not supported in all devices along with imagecapture or imageanalysis) in working while there's slight extra latency in startup during initialization. So for any normal work this extra latency is allowable as its typically under 1-2 seconds, measured from getting the camera provider upto preview texture viewing in ui. This is inflated as I'm considering the total time to first start to see the preview from the initial configurations. While the individual components like preview texture binding with camera source is typically within 500-800ms, while other parts are also low latency. While pure java implementation is on average faster in startup by about 300-500ms in total, while there's no significant difference in performance afterwards between python and java implementations.

In this regard, one thing to mention is that camerax also allows to capture photos in memory and only for the callback of this process we need to subclass an abstract class. In short, if we want to support this in memory capturing of photos then we need to have a java file inclusion.

In view of the above, I would like to understand what the core developers would think regarding the implementation should we have a java implementation and use it through pyjnius or a completely python pyjnius side implementation with just one callback being defined within java due to being an abstract class (only if we want to support that).

[Samael-TLB]

Thanks for your suggestion @RobertFlatt. First of all sorry for the delayed response. Was away from work due to health issues. I took inspiration from your works too.

Aside from performance, some of the newer Android apis implement callbacks as a reference to an instance of a "provider" Java class. It is not possible to pass a reference to PythonJavaClass in this case as this is Python not Java. The workarounds compound, it is much simpler to implement the full provider in Java or Kotlin (there are examples!). The provider probably belongs as a recipe. Try it here https://pypi.org/project/camera4kivy/ , Here is a simple test case https://github.com/Android-for-Python/c4k_photo_example

I agree, but as I mentioned this is the case for just one callback currently that is for in memory photo capture, which has an alternative of photo capture via save to disk directly. But I tend to see a shift in the android apis from normal interface classes to abstract classes which necessitate the use of Java or Kotlin.

The camera widget api must be extended as file capture is performed by the provider not the widget (except screenshot). In addition users expect an image analysis api - they want a camera as part of some other behavior. Also the widget must handle various aspect ratio, orientation, pause/resume , pinch-spread zoom, and tap focus.

The current goal for 3.0 is complete rewrite of the api so as to enable incorporation of future technologies easily, image analysis and gesture handling are very important and are gonna be included by default in the api.

Personally I view Kivy camera widget as great for taking screenshots from a web cam and not much else, very 2000. I choose to ignore Kivy camera widget and implement my own widget, my thinking at the time was backwards compatibility would be hard and unfortunately might be expected. For example Camerax is not usable (including connect) till one time-step after Kivy on_start(), whereas Kivy camera assumes connect at build(). This and the threaded implementation in Camerax means the api must be at least partially event driven.

Yes, the current implementation is not very feature rich but the new proposed one is targeting to be much more useful to the end users. Using CameraX we are safe till Android 5.0 and above which is what officially supported by Kivy and suggested by Android. Though some features if used would raise the minimum supported version to 6 or higher. I am going forward with a lazy permission based approach, where you need permissions first explicitly only when you use the guarded apis, before that you are free to do whatever android allows you to do without permissions, following android guidelines of asking permissions. So this allows for at least no jank in my tests.

Finally for performance a major consideration is the size of any Kivy frame buffer, the hardware camera and screen lead to user expectations of high resolution, too large a Kivy frame buffer may lead to preview latency issues. The Kivy camera bypasses this by defaulting to 640x480, this would not make a mobile user happy. As far as I remember it worked up HD buffer, maybe my Kivy buffer implementation could have been optimized I didn't look at it.

As said above, I have implemented the api and tested it from Android 8-13, with no potential drawback or performance issues or significant difference noticed between pure python and java implementations. I didn't have any issues with the Kivy buffer which supported upto the highest size available on the device during my tests. Though there might be some issues for Android 5-7 (as untested by me).

[misl6]

Hello everyone!

Let's start by saying that as @Samael-TLB pointed out, 3.0.0 is the right moment for us to rewrite what needs to be rewritten from scratch. Camera API is one of these components. It was faulty and buggy since the beginning. Nope. Tech just evolved, we now have camera phones that can compete with professional-grade cameras, and people (and apps) are using cameras more than ever.

I will work out to have a proposal (initially just drawings on paper) for #8582, based on my experience and suggestions taken from this issue, GitHub issues in general, discussions (both on Discord and IRL), etc ... (so please keep an eye on #8582, and feel free to make suggestions as soon as you see something).

Regarding Android:

• Do we need to consider any specific feature to be included in #8582 ?
• Latest Android NDK removed support for API 19 and 20 (Android 4.4), and I do not exclude they are going to remove 21 and 22 soon (Android 5). So, if keeping support for Android 5 does not add any extra effort, keep it, otherwise you have my +1 to target Android 6.
• If Android 6 is also too old, and by dropping support for it we can have more features / keep code simpler / performs better, we can talk about setting Android 7 as the minimum supported version.
• Based on my experience, let's avoid using pyjnius for everything. Instead, pyjnius is super-great to interface a Java object that handles all the hard tasks with our abstraction. Our users are Pythonistas, but we do not necessarily have to rely on Python for everything under the hood, if things can be more performant, less verbose and better to mantain.
• I agree on using CameraX. Since CameraX is a JetPack library, we will need to make the proper changes on python-for-android to automatically include this dependency (unless the user decides to not have it). I was thinking about creating an automated way for packages to announce python-for-android that a non-native library is needed, but ATM we can just add it forcefully for testing purposes.
• Java files location: You're not alone. We will need to take this decision for other features (and even other platforms):
  • How it sounds a kivy/android-support repo?
    • Package name org.kivy.android.support ?
    • Then we can have org.kivy.android.support.camera and org.kivy.android.support.whatever under the same package?
    • We keep track of the needed release in kivy/kivy, but code is not strictly tied with kivy/kivy.
    • We can make an Android library out of it
    • During development would be easier to debug / re-build, without re-building kivy from scratch.
    • kivy-garden packages can make use and implement new features in this "library" without touching kivy/kivy source code.

A quick eagle eye on what I expect to have in #8582:

• Multiple cameras at the same time with the same provider
• High-performance preview box. If possible any CPU-intensive task should be avoided.
• Multi-camera support:
  • List the available cameras and their capabilities for each provider
  • Can I have "wide" camera? The provider checks if any available camera have the "wide" capability and returns a set of compatible cameras
  • Continuous zoom feature: The camera provider uses all the available lenses to have a smooth zoom, without the user / developer having to switch between lenses.
• AF/MF
• Attachable image analysis (barcode / face / anything available): Google MLKit, iOS implementation
• Take photo (selectable resolution/compression/etc) and save it (no screenshot)
• Record a video (selectable resolution/compression/etc) and save it