We should have a central class for all of the OpenCV code to be added to rather than keeping each script independent as they have a lot of the same functionality.

Note that this issue is dependent on the completion of #1 and can likely be completed by the same person.

Motivation

After identifying the ELISA plate's rectangle in the image, we have to continue to process the image. To start doing this, the best course of action is to normalize the semi-rectangular image to a rectangle to be able to best obtain accurate color readings. We want this to line up with the plate and nothing but the plate, excluding external information to make the images easier to process while eliminating variables like differences in perspective. This is akin to a perspective crop in Photoshop or similar.

Task

Utilizing OpenCV, take the image box recognized by #1 upon the press of the camera button, and convert the subset of the image selected by the image box to an image that can be displayed or saved as an image file.

Logistics

This is another script that can easily be integrated with the code to find the bounding box of an image as well as the application as a whole at a later date. It looks like a lot of this work has been done before:

• sample perspective crop with ios objective c
• perspective transform without crop
• perspective transform and crop from gameboy
• perspective transform on a piece of paper

The most difficult part of this will be to ensure the bounding box is relatively accurate and to ensure a relatively high quality image is produced without losing much color or camera information from the transformation.

There might be some trouble identifying the bounding box on surfaces of certain colors; part of the assignment will be determining what the best background would be for the user to use. If something like this is necessary, checking the background color should also be integrated into the application (we can do something as simple as (if background not in range of colors we allow, throw error)).

Motivation

If a user is scanning in their ELISA plate, we need to ensure that the image isn't blurry so that colors aren't blending together and we can more easily identify the vials in the image. We'd like to warn the end user of blurry images and their inability to provide proper data to the end user before they take the image.

Task

Utilizing OpenCV, write a Python script to read in an image file and determine whether the image is blurry. This can be done in multiple ways; this post suggests doing some cool math, while this link seems to have an implementation close to what we're looking for (though we probably want our algorithm to have a lower tolerance).

Logistics

Essentially, the code can be some function that accepts an image, or the file name associated with an image, and outputs true or false - whether the image is blurry or not for some definition of blurriness.

Motivation

The central motivation for this application is to read accurate absolute and relative color values from each of the vials on the ELISA plate. As lighting conditions can vary substantially from location to location and can be dependent on variables such as time of day, we need a way to normalize the colors of the image to ensure their accuracy.

Task

Utilizing OpenCV, work with existing sample ELISA plate images to normalize their colors to some ground truth values of white and black, ensuring that the colors of the image are independent of the color of lighting in the room and attempting to minimize the impact of the device camera's color profile.

Logistics

This task is going to be pretty difficult; it's not a good task to start with, as our metrics for absolute color are not well-defined as of yet and we need some standard to hold this to. OpenCV has the ability to normalize the color profiles of images, but doing so in a way which respects the lighting in the room and perhaps specific devices is difficult (attempting to cater to many specific devices will be much later down the road).
This task also likely requires a research component. How do we standardize color values? What exactly do we standardize them to? What background would be best to provide some truth for compensating? How should this perform in variable lighting?

Some links that might be of help:

• normalizing in another lang
• segmenting an image into color spaces

There's a lot that can be done here, and this is where we might want to start introducing some machine learning library able to compensate for shadows, lighting with perspective and the like. This task will become more concrete and can be split into different tasks once we have a better understanding of the challenge here.

Note that this issue may be dependent on the completion of #6.

Motivation

Our goal with this application is to obtain color values from each vial. Here is where we do so.

Task

Utilizing OpenCV, identify the color of each vial pinpointed through the previous work and display a label on some input image indicating the color's R,G,B value as well as the absolute color value desired by the researcher (to be determined) atop the corresponding vial.

This task also involves some way to store and track the numbers associated with each vial. We probably want to be writing each value atop its corresponding vial and saving the image (our researcher said he wanted to produce images with the color values written on them as proof that the readings are directly associated with the plate) as well as saving each value in a two-dimensional coordinate grid that we can reference to get the value of a specific vial, but the final implementation is up to you.

The form of the output data should be easy enough to work with so that we can display some interesting graphs or information pertaining to the gradient from control group to high concentration, but exactly how we'd like to display this will be determined when we know we have the time to play with the data and extend the application beyond the minimum viable product.

Logistics

It's likely that we'll want to pinpoint the center of each vial as selected by previous work, then either take a sampling of readings of points, average readings from a sample or the full circle, or somehow evaluate local pixels to distill some absolute color value from each individual vial. This data should also be tracked and stored as corresponding to the vial somehow; that'll require some data design for this person to choose the best data structure and implementation for their use case.

Motivation

For our application to work, we need to create a simple camera application with Kivy to interface with a device's camera, view images that can be sent to other programs or take pictures and save them to device storage. We'd also like to have some way of interfacing with the display of the camera to display markings on top of images with OpenCV.

Task

Utilizing the Kivy framework, write a simple application functional on either Android or iOS (it should be easily compatible on either) that's able to display the current output of the camera, take photos and save them.

Logistics

Here are some helpful links I've found that will likely cover a lot of the assignment for you.

• Camera API Documentation
• taking a selfie with Kivy
  -example camera app

The most difficult part of this will likely be interfacing with OpenCV and allowing the image feed to be modified in real time by OpenCV code. Whether this is possible I'm not sure, and it might not even require additional work. Here is some work done with OpenCV and Kivy; it would be ideal to integrate an OpenCV webcam into Kivy rather than using a Kivy webcam just like this. We can work out what would be better; it would likely be best to get the OpenCV camera working but we can likely display post-processed images back through Kivy as well.

Note that this issue may be dependent on the completion of #3 .

Motivation

To integrate with the OpenCV code, the Kivy application should be able to display various forms of output through OpenCV so that we can decorate the camera view. We need some way of allowing opencv to modify the images so that our app can come together, basically.

Task (+ Logistics)

Utilizing Kivy and OpenCV, convert the output from the Kivy camera module to a view or series of images that can be input into OpenCV, then either convert this OpenCV code back into Kivy-compatible images or display the view through OpenCV's .imshow() function or similar.

Bonus points - figuring out how to save the images taken with opencv things applied to such images would be golden. This should be totally possible with Kivy's image API as well as editing frame by frame with OpenCV.

Logistics

Some useful links:

• kivy camera docs
• kivy's example of integration with opencv
• external example of kivy opencv integration

One way I found to get opencv information through Kivy's image module:

self.img1=Image()
self.capture = cv2.VideoCapture(0)        
ret, frame = self.capture.read()
self.frame = frame

// do cv2 conversions on the frame like so
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

Motivation

We'd like this technology to be available to more researchers, and to be available to researchers regardless of smartphone. Further, it's possible that build dependencies and configuration issues obstruct the construction and publishing of the cross-platform application. Hosting a website with a backend that runs the same code our application would run on the front-end would be super convenient for researchers to use. Basically, we want to provide an alternative way to access our technology.

Task

Utilizing either Flask or Django (or some other Python web framework of choice, for ease of compatibility with the current code written in OpenCV), create an API able to accept an image (presumably of an ELISA plate) via a POST request or some other method of transmitting the image and produce a response providing some information:

• Whether the image was too blurry to use
• If not, a 2D grid of the color readings from the plate.
• Potentially a downloadable image of the plate reading with the color values written on top of each of the corresponding plates.

This application would also provide a simple UI for the application, allowing a user to choose a file to upload, press some 'submit' button, then receive a displayed output of their result.

Ideas:
It would be super neat to somehow store all the readings made on a server with accounts so researchers can log into the web app and view all previous readings, but that's something we can add far in the future.

Logistics

Really up to us; this is super flexible. Depending on experience of the assignee with web development, the hours expended putting this together could vary. Ideally the server imports functions from the opencv folder or the application's source folder and exposes them via an API.

Flask is probably the easiest way to do this but Django afaik is more robust and more applicable to larger-scale projects in industry.

Motivation

We currently don't have a way to detect the bounds of an image with a rectangle. It would be helpful to be able to identify a clear rectangle from the ELISA plate, similar to a banking app that allows for the scanning of checks or a scanning app that shows the user what is being identified as the page.

Task

Utilizing OpenCV, write a Python script to read in an image file, identify the ELISA plate present in the image file, then draw a bounding box around it to match the border of the plate. Display this with cv2.imshow() or similar with the bounding box drawn on the image to verify that the box has been drawn correctly.

Logistics

This is a script we can easily integrate into a camera module when constructing the application. There should be quite a few tutorials on something similar on OpenCV, but the general process is the following:

• convert the image to black and white
• obtain the contours of the image
• draw lines on important contours
• connect these lines to form a rectangle
• display such a rectangle on the image

Here are some tutorials to get started:

• detecting edges, lines and shapes
• identifying and detecting different shapes
• OpenCV rectangle detection on Android - logic should be easy to translate to python given that they're the same library
• finding bounding boxes and perspective warping on a game boy

There might be some trouble identifying the bounding box on surfaces of certain colors; part of the assignment will be determining what the best background would be for the user to use. If something like this is necessary, checking the background color should also be integrated into the application (we can do something as simple as (if background not in range of colors we allow, throw error)).

Note that this issue may be dependent on the completion of #1 and #4, depending on approach.

Motivation

To obtain color values from each of the vials on the ELISA plate, we should start by identifying the locations of each individual vial. This will allow us to pinpoint the color of each vial in the image and record these colors, both relative and absolute.

Task

Utilizing OpenCV, identify each vial on the ELISA plate in the image. Track each vial by highlighting them in a visual output from the image and storing some information about the location of each image in a fitting data structure mapping the ELISA coordinate system to the vial (i.e. vial A-1 is at posn (x,y) of the image). We can then either extend this script to track color values or pass this information on to another system that will do so.

Logistics

We know we have a fixed number of vials in a fixed arrangement; if we utilize the normalized image, we may simply be able to align the image with some grid of locations and pinpoint the locations of each vial by lining things up properly. However, identifying the circles atop the vials is more extensible, and will allow us to error upon not finding the proper number of vials in the image (to avoid misidentification).

Many of the resources from #1 are useful here as well, they just have to be translated to locating a multitude of circles rather than a single rectangle in the image. If our algorithm properly bounds the box and normalizes the image, we know we have x circles in the image, and we can use this to ensure that we find all of them. Locating a gradient between fluid in a vial and either table or fluid seen through multiple layers of glass seems like the way to go here to find each circle.

Extension! Many of the vials have cross-contamination from other vials, perhaps due to uneven lighting conditions or other problems with the process of taking the picture and processing the image. As such, lighting from test tubes can be cast onto neighboring test tubes, slightly changing their color and appearance depending on the color and direction of the light. Our algorithm can attempt to compensate for this by excluding parts of the circle that don't match some color profile of the entire circle or stand out in some way (how they stand out is to be determined). This is a task all its own and may require its own issue.