Infinite-ISP is a full-stack ISP development platform - from algorithm development to RTL design, FPGA/ASIC implementation, and associated firmware, tools, etc. It offers a unified platform that empowers ISP developers to accelerate ISP innovation. It includes a complete collection of camera pipeline modules written in Python, an FPGA bitstream & the associated Firmware for the implementation of the pipeline on the Kria KV260 development board, and lastly, a stand-alone Python-based Tuning Tool application for the pipeline. The main components of the Infinite-ISP project are listed below:
Repository name | Description |
---|---|
Infinite-ISP_AlgorithmDesign | Python based model of the Infinite-ISP pipeline for algorithm development |
Infinite-ISP_ReferenceModel ⚓ | Python based fixed-point model of the Infinite-ISP pipeline for hardware implementation |
Infinite-ISP_FPGABinaries | FPGA binaries (bitstream + firmware executable) for the Xilinx® Kria KV260’s XCK26 Zynq UltraScale+ MPSoCA |
Infinite-ISP_Firmware | Firmware for the Kria kV260’s embedded Arm® Cortex®A53 processor |
Infinite-ISP_TuningTool | Collection of calibration and analysis tools for the Infinite-ISP |
Infinite-ISP Reference Model: A Python-based Model for RTL Implementation of Camera Pipeline Modules
Infinite-ISP Reference Model is a Python based fixed-point implementation of Infinite-ISP pipeline. It is a comprehensive collection of camera pipeline modules, designed to convert input RAW images from sensors to output RGB images. The model provides a reference implementation that aids in generating RTL code enabling rigorous testing, verification, and validation of the pipeline's functionality and behavior.
This model uses lookup tables for complex functions like Gaussian and Sigmoid, applies fixed-point numbers or custom approximations for divisions and square roots, optimizing for minimum loss in image quality.
In its current state, the model implements simple algorithm per module, with plans to incorporate RTL-friendly complex algorithms in future versions.
ISP pipeline for InfiniteISP_ReferenceModel v1.0
-
RTL-friendly Code: The model provides a Python implementation of camera pipeline modules that can be directly translated to RTL, with optimizations such as lookup tables, custom approximations, and integer calculations.
-
Test Vector Generation: The project includes automated scripts for generating test vectors for individual as well as multiple modules, simplifying the testing process.
-
Dataset Processing: The model facilitates execution for multiple images with different or same configuration files.
-
Video Processing: The model also features a video processing script that allows for sequential frame processing, with operational 3A Statistics data flowing between frames.
The primary objective of Infinite-ISP_ReferenceModel is to create a open source Python-based Model that streamlines the process of implementing camera pipeline modules into RTL-friendly implementations. This enables seamless integration with hardware designs and simplifies the development of efficient image processing systems. By providing optimized algorithms, test vector generation, and video processing capabilities, the model aims to be an invaluable tool for developers working on image processing projects and RTL implementations.
The table below provides a feature list of the model. The version 1.0
of the model implements hardware-friendly and simple algorithms for each module.
Modules | Infinite-ISP_ReferenceModel |
---|---|
Crop | Crops image with Bayer pattern safe cropping |
Dead Pixel Correction | Modified Yongji's et al, Dynamic Defective Pixel Correction for Image Sensor |
Black Level Correction | Calibration / sensor dependent - Applies parameters from the config file tunable using the tuning tool |
Optical Electronic Transfer Function (OECF) | Calibration / sensor dependent - Implements a LUT from config |
Digital Gain | Gains from config file - In auto mode AE feedback is incorporated for digital gain selection |
Bayer Noise Reduction | Green Channel Guiding Denoising by Tan et al - Implements Chroma and Spatial filters through LUTs |
Auto White Balance | Enhanced Gray World - AWB stats calculations within an optimal threshold |
White Balance | WB gains multiplication - Parameters from the config file tunable using the tuning tool |
Demosaic | Malwar He Cutler’s demosaicing algorithm |
Color Correction Matrix | Calibration / sensor dependent - 3x3 CCM from the config file tunable using the tuning tool |
Gamma Correction | Implements a LUT from config |
Auto Exposure | Auto Exposure - AE stats calculations based on skewness |
Color Space Conversion | YCbCr digital - BT 601 - Bt 709 |
Noise Reduction | Non-local means filter - Implements intensity level difference through a LUT |
RGB Conversion | Converts YCbCr digital image to RGB |
Invalid Region Crop | Crops image to a fixed size |
Scale | Nearest Neighbor - Integer Scaling |
YUV Format | YUV - 444 - 422 |
The project is compatible with Python_3.9.12
The dependencies are listed in the requirements.txt file.
The project assumes pip package manager as a pre-requisite.
Follow the following steps to run the pipeline
- Clone the repo using
git clone https://github.com/10xEngineersTech/Infinite-ISP_ReferenceModel
- Install all the requirements from the requirements file by running
pip install -r requirements.txt
- Run isp_pipeline.py
python isp_pipeline.py
There are a few sample images with tuned configurations already added to the project at in_frames/normal folder. In order to run any of these, just replace the config file name with any one of the sample configurations provided. For example to run the pipeline on Indoor1_2592x1536_12bit_RGGB.raw
simply replace the config file name and data path in isp_pipeline.py
CONFIG_PATH = './config/Indoor1_2592x1536_12bit_RGGB-configs.yml'
RAW_DATA = './in_frames/normal/data'
There are two scripts that run Infinite-ISP on multiple images:
-
isp_pipeline_dataset.py
Execute multiple images. Raw image should have its own config file with name<filename>-configs.yml
where<filename>
is raw filename otherwise the default configuration file configs.yml is used. -
video_processing.py
Each image in the dataset is considered as video frame in sequence. All images use the same configuration parameters from configs.yml and 3A Stats calculated on a frame are applied to the next frame.
After cloning the repository and installing all the dependencies follow the following steps:
- Set
DATASET_PATH
to dataset folder. For example if images are in in in_frames/normal/data folder
DATASET_PATH = './in_frames/normal/data'
- If your dataset is present on another git repository you can use it as a submodule by using the following commands in the root directory. In the command,
<url>
is the address of git repository such ashttps://github.com/<user>/<repository_name
and<path>
is the location in your repository where you want to add the submodule and for Infinite ISP<path>
should be./in_frames/normal/<dataset_name>
. Please keep in mind that your<dataset_name>
should not bedata
because directory in_frames/normal/data already exists.
git submodule add <url> <path>
git submodule update –-init –-recursive
- After adding git repository as a submodule update
DATASET_PATH
variable in isp_pipeline_dataset.py to./in_frames/normal/<dataset_name>
. Git does not allow to import a repository’s subfolder using a submodule. You can only add an entire repository and then access the folder. If you want to use images from a subfolder of a submodule modify theDATASET_PATH
variable in isp_pipeline_dataset.py or video_processing.py accordingly.
DATASET_PATH = './in_frames/normal/<dataset_name>'
- Run
isp_pipeline_dataset.py
orvideo_processing.py
- The processed images are saved in out_frames folder.
Please refer to the provided instructions for generating test vectors for multiple images, considering individual or multiple modules as the Device Under Test (DUT).
Please read the Contribution Guidelines before making a Pull Request
Here are the results of this pipeline compared with a market competitive ISP. The outputs of our ISP are displayed on the right, with the underlying ground truths on the left.
A comparison of the above results based on PSNR and SSIM image quality metrics
Images | PSNR | SSIM |
---|---|---|
Indoor1 | 22.5788 | 0.8923 |
Outdoor1 | 19.1544 | 0.9048 |
Outdoor2 | 18.8681 | 0.8071 |
Outdoor3 | 17.2825 | 0.7304 |
Outdoor4 | 19.9814 | 0.8198 |
For more comprehensive algorithms documentation and to understand how to use the python model, please visit the User Guide.
This project is licensed under Apache 2.0 (see LICENSE file).
For any inquiries or feedback, feel free to reach out.
Email: [email protected]
Website: http://www.10xengineers.ai