plantnet / malpolon

Deep-SDM framework

PlantNet

Developed by Tom August

last built 2019-07-23

The R-package interfaces with the PlantNet image classification API. This API is designed to receive images and return species classifications. You can find out more about the PlantNet project here: https://plantnet.org. To use the API you need to have registered an account and generated an API key. All this can be done here: https://my.plantnet.org/.

Install the package

To install the development version of this package from Github use this code.

# Install using the devtools package
devtools::install_github(repo = 'BiologicalRecordsCentre/plantnet')

library(plantnet)

Using the package to classify images

The images that you want to classify need to have URLs. If the images you have are not online you will need to put them online and then copy all of the URLs. The other information you need is your API key. You can create this after registering here: https://my.plantnet.org/.

Single image

With these we can do a single image classification like this. Here we use this photo of a lavender

# Get your key from https://my.plantnet.org/
key <- "YOUR_SUPER_SECRET_KEY"

# Get the URL for your image
imageURL <- 'https://upload.wikimedia.org/wikipedia/commons/thumb/6/60/Single_lavendar_flower02.jpg/800px-Single_lavendar_flower02.jpg'

classifications <- identify(key, imageURL)
classifications

##       score     latin_name                 common_name         
##  [1,] 35.92154  "Lavandula dentata"        "French lavender"   
##  [2,] 31.78688  "Lavandula angustifolia"   "Lavender"          
##  [3,] 18.06659  "Lavandula stoechas"       "Topped lavender"   
##  [4,] 6.717049  "Lavandula latifolia"      "Broadleaf lavender"
##  [5,] 1.665378  "Perovskia atriplicifolia" "Russian-sage"      
##  [6,] 1.022022  "Lavandula pinnata"        NA                  
##  [7,] 0.5533172 "Vitex agnus-castus"       "Chasteberry"       
##  [8,] 0.5320209 "Salvia farinacea"         "Mealy sage"        
##  [9,] 0.485067  "Lavandula multifida"      "Fern-leaf lavender"
## [10,] 0.3812005 "Lavandula canariensis"    NA                  
## [11,] 0.2864826 "Nepeta tuberosa"          NA                  
## [12,] 0.1719182 "Lavandula minutolii"      NA                  
## [13,] 0.1427725 "Perovskia abrotanoides"   "Russian Sage"

You can see in the the table returned there are three columns returned. The first gives you a score, this is the likelihood that the species in this row is the correct classification for the image you provided. As you can see in this instance there is not much between the top two species so we could not be confident which species it is. The second column gives the Latin names as binomials, and the final column gives the most commonly used common name for the species.

If you want more information, such as a list of all of the common names for each classification, or the full Latin name including authors, you can access all of this information by using simplify = FALSE.

classifications <- identify(key, imageURL, simplify = FALSE)
str(classifications,1)

## List of 4
##  $ query              :List of 3
##  $ language           : chr "en"
##  $ preferedReferential: chr "florefrance"
##  $ results            :List of 13

The top elements give some information about the call we made to the API, but the results contains what we are usually after. Results has one entry for each species classification, or each row in the table we saw above. Let's look at one.

classifications$results[[1]]

## $score
## [1] 35.92154
## 
## $species
## $species$scientificNameWithoutAuthor
## [1] "Lavandula dentata"
## 
## $species$scientificNameAuthorship
## [1] "L."
## 
## $species$genus
## $species$genus$scientificNameWithoutAuthor
## [1] "Lavandula"
## 
## $species$genus$scientificNameAuthorship
## [1] "L."
## 
## 
## $species$family
## $species$family$scientificNameWithoutAuthor
## [1] "Lamiaceae"
## 
## $species$family$scientificNameAuthorship
## [1] ""
## 
## 
## $species$commonNames
## $species$commonNames[[1]]
## [1] "French lavender"
## 
## $species$commonNames[[2]]
## [1] "Spanish Lavender"

Here we have information on the Latin name, authors, and at the end, a list of the common names.

Multiple images

You can get a better identification if you provide more than one image of the plant, and of multiple organs of the plant. The organs that PlantNet considers are: leaf, flower, fruit, bark. You can also take images classed as habit (the overall form of the plant), or other, but you can only have an image labelled as one of these if you also have an image labelled as one of the primary organs (i.e. leaf, flower, fruit, bark).

In this example we are going to use three images of Quercus robur from the Encyclopedia of Life.

# We can search using up to five images
# Here are three picture of Quercus robur
imageURL1 <- 'https://content.eol.org/data/media/55/2c/a8/509.1003460.jpg'
imageURL2 <- 'https://content.eol.org/data/media/89/88/4c/549.BI-image-16054.jpg'
imageURL3 <- 'https://content.eol.org/data/media/8a/77/9b/549.BI-image-76488.jpg'

identify(key, imageURL = c(imageURL1, imageURL2, imageURL3))

##      score    latin_name           common_name      
## [1,] 68.28053 "Quercus robur"      "Pedunculate oak"
## [2,] 28.3852  "Terminalia catappa" "Indian-almond"  
## [3,] 19.02534 "Quercus petraea"    "Sessile oak"

In this case all three images have been used to arrive at one classification. Here we get a few species suggested but the top species is correct, and has a significantly higher score than the second species. In this example we have not told the API what the organs are. The API can use this information to help give a better classification.

# This time I specify the organs in each image
identify(key,
         imageURL = c(imageURL1, imageURL2, imageURL3),
         organs = c('habit','bark','fruit'))

##      score    latin_name           common_name      
## [1,] 68.28053 "Quercus robur"      "Pedunculate oak"
## [2,] 19.02534 "Quercus petraea"    "Sessile oak"    
## [3,] 18.92347 "Terminalia catappa" "Indian-almond"

Notice that now the API knows which organs each image is of, we get slightly different results, the top species is the same but we can have higher confidence that this is the correct answer because the distance between the score of the first and second image is larger than before.

malpolon's Issues

Enrich PatchExtractor with a geographic coordinates system defined area

• Context

Multi-modal datasets like GeoLifeCLEF's comprise numerous bioclimatic and pedologic variables that can be aggregated to construct rasters, scaling up to countries or continents.

These rasters are then divided into smaller patches that are being fed into the deep learning module by calling the private method environmental_raster.Raster._extract_patch through the PatchExtractor class constructor, by indexing an instance of said class.

During the process, the python module rasterio is used to process input data (Tiff, gTiff...) into a custom, useable dataset object.

• Problem

Currently, the size of the wanted patch is passed as parameter in the PatchExtractor class constructor (which is then passed on to the Raster class constructor) and is only interpreted as pixel size.

For example : PatchExtractor("./my_rasters", 256) will instanciate a patch of size 256x256 pixels where each pixel maps a certain geographical area in the raster (depending on the resolution of the raster).

However, in order to work on or querry a specific patch, one would need to know the mapping between pixel coordinates and geographic coordinates, which is not trivial.

• Solution

Introduce a new way of registering patch sizes in the malpolon.data.environmental_raster.Raster class so that users can choose the unit they want to work with.

Maybe add a new parameter size_type: str = "px" with values taken in ["px", "km", "m", "arcseg"] (or similar) and define a coordinates conversion method that would return the corresponding pixel coordinates mapping.

Generalize GeoLifeCLEF raster loader

• Context
  The GeoLifeCLEF challenge was a main subject to adress with the Malpolon framework and custom raster & patch loaders were implemented in malpolon > data > datasets > geolifeclef.py

• Problem
  These specific loaders cannot be reused or called as they are to load other data. On the other hand, implementing new custom raster and patch loaders for each new dataset would most probably result in code redundancy.

• Solution
  A generic raster and patch loader should be written, allowing easy import and inheritance of common data structures and parameters.

Accuracy fails on torchmetrics 0.11

The functional torchmetrics.functional.Accuracy requires a third parameter "task" since version 0.11.

Solution proposition : instantiate the accuracy class on row 137 file standard_prediction_systems.py
Accuracy(task="multiclass", num_classes=4)
drawback : you need to specify the number of classes

Allow advanced users to choose custom transformation functions on raw prediction scores

• Context
  Malpolon is built for both beginners and advanced users in machine & deep learning. In such, the framework allows (or will allow) selecting a task among several available (binary classification, multi-label classification, abundance prediction [regression]...) with default parameters and transparent (but parametrable) use of loss functions.

• Problem
  Depending on the use case, one advanced user might want to work with the raw scores of the model before passing them on to the last activation function, which is sometimes bound with the loss function.

Currently this is not possible and we would need to find an elegant way to implement this possibility without adding layers of complexity for beginner users.

• Solution
  Most likely, model_builder.py and standard_prediction_systems.py will have to be re-structured to allow this new parametrization.
  This new parameter could very well be read from the .yaml config files but seeing that this new feature would only be used by advanced users, it remains to be decided weither to complexify files directly manipulated by beginners or simply embed the code with the possibility of extracting the raw scores layer.

[Enhancement] Add parametrable checkpoint loading in training phase

Is your feature request related to a problem? Please describe.
There isn't any configuration key to trigger checkpoint loading in training phase. In fact, the main scripts provided in the examples do not offer the choice of loading a custom trained model, whether from a checkpoint or a saved model .pth, this needs to be added by hand.

Describe the solution you'd like
Add the possibility of loading custom trained models to perform transfer learning or resume a stopped/paused training. This should be done through a simple boolean key-value set in the examples' config files.

Describe alternatives you've considered
It would be best to offer the choice of loading model weights both through PytorchLightning and PyTorch (similar to what's being done in inference).

Additional context

[Enhancement]

Is your feature request related to a problem? Please describe.
There is no easy support for model inference as users have to manually load checkpoints and run model.predict().

Describe the solution you'd like
User should be able to switch from training to inference using their configuration files.

Describe alternatives you've considered
We could provide users with examples files entirely dedicated to inference, base on training main files. This way, there are 2 main files and the config file doesn't have to change.

Additional context
Users may want to alter the data as they test their models to assess its generalization power.
Users need to be able to output metrics on test data.
Users may want to all 3 training steps (train, val, test) in a single command.

Raster filter based on prefix/suffix

Is your feature request related to a problem? Please describe.
Currently, when using GLC23 raster datasets it is possible to filter loaded rasters based on their entire name, or their channel in the case of satellite images. But it is not possible to finely select files with a regex-like filter on the files' name.

Describe the solution you'd like
Either a similar solution to RasterTorchGeoDataset where both a glob and a regex expression are provided; or additional arguments prefix and suffix which would take a strings as input.

Describe alternatives you've considered

Additional context

Better support of multi-modalities in input data transformations and augmentation

It is currently not clear how to adapt the input data transformation and augmentation system to multi-modal data when the modalities can not be put together in a single tensor (e.g., when the patches are of different sizes, etc.).
This case happens quite often in practice and should be handled properly.

If there are some randomized transforms (typically with data augmentation), the modalities can not be processed totally separately otherwise different transforms will be applied to each of them.

To solve this, we might need to use functional transforms to handle the seed of the randomizers manually.

Error management for out-of-bounds patches

• Context

Multi-modal datasets like GeoLifeCLEF's comprise numerous bioclimatic and pedologic variables that can be aggregated to construct rasters, scaling up to countries or continents.

These rasters are then divided into smaller patches that are being fed into the deep learning module by calling the private method environmental_raster.Raster._extract_patch through the PatchExtractor class constructor, by indexing an instance of said class.

During the process, the python module rasterio is used to process input data (Tiff, gTiff...) into a custom, useable dataset object.

• Problem

Let us consider patch_extractor = PatchExtractor("./my_rasters", size=256) where size is the patch size.

Since a patch is generated from its center pixel deduced from the $[x, y]$ input geographic coordinates, one could trigger an error exception by indexing $[x, y]$ on patch_extractor if these coordinates were not far enough from the edge of the raster.

Indeed, for a patch to be correctly built and returned, there needs to enough space betwen the $[x,y]$ input geographic coordinates and the horizontal & vertical borders of a raster to query the $|\frac{size}{2}|$ values of the patch (in the pixel-coordinate system).

Otherwise, out-of-bounds values (relatively to the raster) will be queried to construct a patch.

• Solution
  It is unclear about which solution is best suited for every user, but there are several possibilitis :

1. Force the user to index valid input coordinates so that the patch window fits in the raster.
2. Fill with nan values. Downside : no distinction between ocean and out-of-bounds pixels.
3. Apply a padding around the raster data (either mirror or repeat pattern)
4. Leave the choice to the user