tripalcultivate / tripalcultivate-phenotypes

Provides generic support for large scale phenotypic data and traits with importers, content pages and visualizations.

Tripal Cultivate: Phenotypes

Developed by the University of Saskatchewan, Pulse Crop Bioinformatics team.

NOTE: This package will replace the following Tripal v3 modules: Raw Phenotypes, AnalyzedPhenotypes.

• Creates genus-specific Tripal Content Types for Trait pages to provide a landing page for all information about a specific trait. These are specific to the genus to ensure that all data summarized is relevant and to respect that traits to vary between genus in their expression and specific definition.

• Supports using genus-specific ontologies to ensure you capture each trait fully and mapping of these genus-specific terms to domain and system specific ontologies to enable comparison and data sharing.

• Focuses on the Trait - Method - Unit formula for describing phenotypic data.

  • This supports collecting all data for a specific trait (e.g. Plant Height) into a single page while still fully describing methodology and units for accurate analysis.

  • For the Plant Height trait, you would have data available for multiple experiments, measurement methodology (e.g highest canopy point, average canopy height in a plot, drone captured height based on NDVI) and units on the same page but they would not be combined across experiment, method or units.

• A holding space for raw phenotypic data / measurements right after collection which is private by default and sharable with individual accounts. These data are kept outside the main schema for your biological data since they are raw, unpublished results. There is an easy means to backup data, validate and import by season.

Citation

If you use this module in your Tripal site, please use this citation to reference our work any place where you described your resulting Tripal site. For example, if you publish your site in a journal then this citation should be in the reference section and anywhere functionality provided by this module is discussed in the above text should reference it.

Lacey-Anne Sanderson and Reynold Tan (2023). TripalCultivate Phenotypes: Large-scale trait and phenotypic data integration for Tripal. Development Version. University of Saskatchewan, Pulse Crop Research Group, Saskatoon, SK, Canada.

Install

Using composer, add this package to your Drupal site by using the following command in the root of your Drupal site:

composer require tripalcultivate/phenotypes

This will download the most recent release in the modules directory. You can see more information in the Drupal Docs.

Then you can install it using Drush or the Extensions page on your Drupal site.

drush en trpcultivate_phenotypes

Technology Stack

See specific version compatibility in the automated testing section below.

• Drupal
• Tripal 4.x
• PostgreSQL
• PHP
• Apache2

Automated Testing

This package is dedicated to a high standard of automated testing. We use PHPUnit for testing and CodeClimate to ensure good test coverage and maintainability. There are more details on our CodeClimate project page describing our specific maintainability issues and test coverage.

The following compatibility is proven via automated testing workflows.

Drupal	10.0.x	10.1.x
PHP 8.1
PHP 8.2

tripalcultivate-phenotypes's Issues

[Bug] Homepage WSOD Chado Schema error on Drupal 10.0.x and 8.1

Drupal Version

10.0

Tripal Version

4.x dev (most recent commit)

Module Version

4.x dev (most recent commit)

Site Operating System

Debian Linux

Installation Method

Docker image built from this module

Browsers

Firefox, Safari

Observations

The docker was able to build with no error, however, when I load the homepage it is a WSOD with the following error message:

Expected Behaviour

I would expect to see the home page with no errors. Tripal Core works on this Drupal and PHP version and this is the only additional module installed. Furthermore, this docker image is based off Tripal Docker with the only real addition being using testchado instead of chado and enabling these modules.

Steps to Reproduce

1. Clone this repository

git clone https://github.com/TripalCultivate/TripalCultivate-Phenotypes.git trpcultpheno-g0.31-wsod

2. Build the docker image as follows:

docker build --tag=tripalcultivate-phenotype:local --build-arg drupalversion="10.0.x-dev" ./

3. Create a container from the docker image:

docker run --publish=80:80 --name=phenog0.31 -tid --volume=$(pwd):/var/www/drupal9/web/modules/contrib/TripalCultivate-Phenotypes tripalcultivate-phenotype:local

4. Start the database:

docker exec phenog0.31 service postgresql restart

5. Load up http://localhost in the browser... you will see the following:

Drupal\tripal\TripalDBX\Exceptions\SchemaException: Invalid or unsupported Chado schema version ''. in Drupal\tripal_chado\Database\ChadoSchema->getSchemaDef() (line 56 of modules/contrib/tripal/tripal_chado/src/Database/ChadoSchema.php).

Relevant log output

Drupal\tripal\TripalDBX\Exceptions\SchemaException: Invalid or unsupported Chado schema version ''. in Drupal\tripal_chado\Database\ChadoSchema->getSchemaDef() (line 56 of modules/contrib/tripal/tripal_chado/src/Database/ChadoSchema.php).

Drupal\tripal\TripalDBX\TripalDbxSchema->getTableDef('analysis', Array) (Line: 110)
Drupal\tripal_chado\Plugin\Field\FieldType\ChadoStringTypeItem::tripalTypes(Object) (Line: 301)
Drupal\tripal\TripalField\TripalFieldItemBase::schema(Object) (Line: 461)
Drupal\field\Entity\FieldStorageConfig->getSchema() (Line: 498)
Drupal\field\Entity\FieldStorageConfig->getColumns() (Line: 322)
views_field_default_views_data(Object) (Line: 195)
views_views_data() (Line: 236)
Drupal\views\ViewsData->Drupal\views\{closure}(Object, 'views') (Line: 388)
Drupal\Core\Extension\ModuleHandler->invokeAllWith('views_data', Object) (Line: 244)
Drupal\views\ViewsData->getData() (Line: 154)
Drupal\views\ViewsData->get('node_field_data') (Line: 85)
Drupal\views\Plugin\ViewsHandlerManager->getHandler(Array, NULL) (Line: 898)
Drupal\views\Plugin\views\display\DisplayPluginBase->getHandlers('sort') (Line: 1055)
Drupal\views\ViewExecutable->_initHandler('sort', Array) (Line: 913)
Drupal\views\ViewExecutable->initHandlers() (Line: 2328)
Drupal\views\Plugin\views\display\DisplayPluginBase->preExecute() (Line: 1707)
Drupal\views\ViewExecutable->preExecute(Array) (Line: 1642)
Drupal\views\ViewExecutable->executeDisplay('page_1', Array) (Line: 81)
Drupal\views\Element\View::preRenderViewElement(Array)
call_user_func_array(Array, Array) (Line: 101)
Drupal\Core\Render\Renderer->doTrustedCallback(Array, Array, 'Render #pre_render callbacks must be methods of a class that implements \Drupal\Core\Security\TrustedCallbackInterface or be an anonymous function. The callback was %s. See https://www.drupal.org/node/2966725', 'exception', 'Drupal\Core\Render\Element\RenderCallbackInterface') (Line: 788)
Drupal\Core\Render\Renderer->doCallback('#pre_render', Array, Array) (Line: 374)
Drupal\Core\Render\Renderer->doRender(Array, ) (Line: 204)
Drupal\Core\Render\Renderer->render(Array, ) (Line: 238)
Drupal\Core\Render\MainContent\HtmlRenderer->Drupal\Core\Render\MainContent\{closure}() (Line: 580)
Drupal\Core\Render\Renderer->executeInRenderContext(Object, Object) (Line: 239)
Drupal\Core\Render\MainContent\HtmlRenderer->prepare(Array, Object, Object) (Line: 128)
Drupal\Core\Render\MainContent\HtmlRenderer->renderResponse(Array, Object, Object) (Line: 90)
Drupal\Core\EventSubscriber\MainContentViewSubscriber->onViewRenderArray(Object, 'kernel.view', Object)
call_user_func(Array, Object, 'kernel.view', Object) (Line: 111)
Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher->dispatch(Object, 'kernel.view') (Line: 168)
Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object, 1) (Line: 74)
Symfony\Component\HttpKernel\HttpKernel->handle(Object, 1, 1) (Line: 58)
Drupal\Core\StackMiddleware\Session->handle(Object, 1, 1) (Line: 48)
Drupal\Core\StackMiddleware\KernelPreHandle->handle(Object, 1, 1) (Line: 191)
Drupal\page_cache\StackMiddleware\PageCache->fetch(Object, 1, 1) (Line: 128)
Drupal\page_cache\StackMiddleware\PageCache->lookup(Object, 1, 1) (Line: 82)
Drupal\page_cache\StackMiddleware\PageCache->handle(Object, 1, 1) (Line: 48)
Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle(Object, 1, 1) (Line: 51)
Drupal\Core\StackMiddleware\NegotiationMiddleware->handle(Object, 1, 1) (Line: 51)
Drupal\Core\StackMiddleware\StackedHttpKernel->handle(Object, 1, 1) (Line: 692)
Drupal\Core\DrupalKernel->handle(Object) (Line: 19)

G2.10 Data Importer Importer + Validator API

Create importer + validator api for phenotypes collect and phenotypes share.

G5.45 Update Docker paths

Branch

g5.45-updateDockerPaths

Groups

Group 5 | Automated Testing | Performance

Describe

Thanks to the end of life for Drupal 9 and Tripal PR #1681, the paths in our dockers and automated testing need to change.

Specifically, we have changed from \var\www\html\drupal9 to \var\www\html\drupal. This will insulate us against changes needed to paths in the future but will break all automated testing once the Tripal PR is merged.

Design

Make sure to update all places drupal9 exists in your code and also bump the version of tripal/test-tripal-action from 1.1 to 1.2 in your workflows.

G5.26 Update testing environment + Docker to use `testchado` for the Chado schema name

Branch

g5.26-useTestChado

Groups

Group 5 | Automated Testing | Performance

Describe

Currently our dockerfile and testing use the default tripaldocker images. Unfortunately these images use chado as the name of the schema that Chado is installed in. This does not help us to confirm we are not making any assumptions of what the schema name is... this is an important assumption to test for since Tripal 4 allows the admin to name the schema anything and also supports multiple Chado instances. Testing on a site with no schema named chado makes it more likely we are compatible with multiple Chado instances.

Design

We will switch from using the already built image as a base for our dockerfile and instead explore building our docker based on the same process as the tripaldocker so we can also pass in the parameters to name our chado install. This may also need changes in Tripal core to ensure an image is built without Chado installed... We will explore the how as part of development.

Additionally, we will update our testing workflows to build our own dockerfile rather than using an already built image... This will be a bit slower but we need to test this assumption and it does ensure our development + test environments are even more similar.

G4.47 Data File Validator

Branch

g4.47-Data-file-validator

Groups

Group 4 - API | Services | Plugins

Describe

Create a validator plugin that validates Data File for

1. Is not empty/0 file id.
2. Has Drupal File Id assigned/created.
3. File Id can loaded using Drupal file system API.
4. A valid TSV or TXT file extension and file MIME type.
5. File size is greater than 0.
6. File can be opened/read.

Design

Similar to other validator plugin, this method will implement validator plugin with the following method().

public function validate() {
   Check file id is > 0
   
   Load file id
   Check if file can be loaded through Drupal File API
   Use MIME information of the file to validate file type.
   Use the File size information to validate file size is > 0
  
   Open/Read File 
   Check if file can be opened/read
}

G2.39 - Data Collection Template File Generator

Branch

g2.02-Template-Generator

Groups

Group 2 - Data Importing

Describe

Create a data collection template file in tab-separated format (tsv). The headers encoded in the template file are the headers defined by the headers property in the importer instance.

Template file can be downloaded by clicking on the the link Download a template file located just below the list of headers of the importer.

Design

Defined a route that will become the value of the href attribute of the download link mentioned in the Describe section.

trpcultivate_phenotypes.template_generator:
  path: '/tripalcultivate-phenotypes/download/template-file'
  defaults:
    _title: 'Download: Data Collection Template File'
    _controller: '\Drupal\trpcultivate_phenotypes\Controller\TripalCultivatePhenotypesTemplateFileController::generateFile'
  requirements:
    _permission: 'administer tripal'

The controller that handles the file will have the following components/logic:

1. Retrieve the configuration value relating to directory for storing template files.
2. Access headers defined in the importer. This set of headers will be transformed into tsv string (tab separated).
3. Create a file in the specified directory.
4. Write the headers into the file created in the previous step.
5. Start a download action for user to download the template file.

For discussion:
Passing the headers from the importer to the controller.
Options:

1. Using Routing parameter - query string can be hacked

example.user:
  path: '/example/{user}'

3. Drupal Session Storage - is client-side and can be susceptible to XSS attack.

$session = \Drupal::service('session')->set('column_headers', $headers);

4. Drupal State Storage - for information about system state.

\Drupal::state()->set('column_headers', $headers);

5. Drupal Tempstore - Temporary buffer to save data.

\Drupal::service('tempstore.private')->set('column_headers', $headers);

6. Other

G0.06 Configuration Form for Ontology and Terms

This configuration will define vocabulary terms and controlled vocabulary terms used to house traits and by many processes in the phenotypes module.

G4.12 Terms Service for creating default terms on install

This service is the upgraded version of Analyzed Phenotypes CV API with a focus on the new D10 functionality.

Service Name: TripalCultivatePhenotypesTermsService
Found here: trpcultivate_phenotypes/src/Service/TripalCultivatePhenotypesTermsService.php

Associated Config:

We will use a YAML file to define the terms rather than the old nested array structure. This has been done in Tripal Core here:

• Tripal defines the schema
• Service to create the content types defined in the YML
• Tripal Chado implementing that schema

Methods:

Note:

• current defineTerms() is replaced by the associated config above.
• current mapDefaultTermToConfig() will no longer be needed as we will have a term_key (i.e. experiment_year, experiment_location).
• current getTermConfigValue() becomes getTermId() below.
• current getTerm() will go away. We think we don't need it yet due to the autocomplete which was contributed to core. If we do need something like this, we need to design such a service in Tripal Core and then just use it in this module.

/**
 * Insert cvterms (+ cvs as needed) into Chado based on YAML file 
 * and set term configuration variable defaults.
 * 
 * @return boolean
 *   True if all terms were inserted successfully and false otherwise.
 */
public function loadTerms();

/**
 * Retrieves the ID of the term configured for a specific role.
 * 
 * It is expected that the administrator can configure these terms. As such
 * this method will pull the value from configuration rather then look it up
 * in the database.
 * 
 * @param string $term_key
 *   The unique identifier for the term of interest. This should be one of:
 *   data_collector, genus, experiment_location, collection_method, experiment_container,
 *   unit_to_method_relationship_type, method_to_trait_relationship_type, 
 *   experiment_replicate, experiment_year.
 * @return integer
 *   The chado cvterm_id for the term associated with that key.
 */
public static function getTermId(string $term_key);

g0-issue2-Configuration: R Convertion Rules

Trait names can be converted to R compatible version for R data processing support. This configuration page will allow phenotypes module to set conversion rules.

G4.41 - Genus Validator

Branch

g4.41-Genus-Validator

Groups

Group 4 - API | Services | Plugins

Describe

Create a validator plugin that validates Genus value for

1. No value supplied - Empty value
2. Is a value that exits in chado.organism table, genus field.
3. Is an active genus in that it had been configured.
4. If it is used in combination with Project/Experiment field, that it is the Genus set for the project.

Design

Identical to the Project validator plugin, this method will implement validator plugin with the following method().

public function validate() {
  Check if user left genus field empty.
  If it has a value, check the following:
    Check if it is an active genus - This check will cover both checks to ensure
    that genus is active and that it exits in chado.organism
  
  Final check, but first see it genus was used in combination with project, in this case 
  check to see if genus-project pair matched.
}

Update compatibility chart and automated testing to match Tripal core.

We need to update the versions we are testing on as follows:

[Bug] Unable to find configuration for genus added after terms installed + form loaded once before

Drupal Version

10.0

Tripal Version

4.x dev (most recent commit)

Module Version

4.x dev (most recent commit)

Site Operating System

Debian Linux

Installation Method

Docker image built from this module

Browsers

Firefox

Observations

I receive the following error (indicating that $this->service_genusontology->getGenusOntologyConfigValues($genus); returned NULL when trying to configure multiple genus. This seems to be dependant on the process by which the organisms were added.

Expected Behaviour

I expect to be able to configure all my genus' ;-p

Steps to Reproduce

1. Create a docker image and container from this module. This was first observed in PR #32 but that one will soon be merged.

Execute the following from within a cloned version of this repository:

docker build --tag=trpcultivate-pheno:g0.06 ./
docker run --publish=80:80 -tid --name=phenog0.06 --volume=`pwd`:/var/www/drupal9/web/modules/contrib/TripalCultivate-Phenotypes trpcultivate-pheno:g0.06
docker exec phenog0.06 service postgresql restart

2. Go to the configuration form: http://localhost/admin/tripal/extension/tripal-cultivate/phenotypes/ontology
3. Create an organism (http://localhost/tripal/add/organism)
4. Go back to the configuration form. You will be asked to load the terms for this module by executing the tripal jobs -do so using docker exec phenog0.06 drush trp-run-jobs --username="drupaladmin"
5. Once the terms are loaded, go back to the configuration form -you should see a row in the configuration form for the genus you added ✅ this is good :-)
6. Now go create a new organism with a different genus that the original one.
7. Return to the configuration form and you should get an error that it could not find configuration for the second genus you created.

Relevant log output

No response

G4.02 Validation API

Branch

g4.02-Validation-API

Groups

Group 4 - API | Services | Plugins

Describe

This is a Drupal Plugin-based validation API that will be used to validate imported data file in all levels: PROJECT, GENUS, FILE, TRAITS and VALUES.

Design

In contrast to Tripal 3 and Drupal 7 version where the validation API is hook-based, this version will implement a Drupal 9 Plugin for each validation levels. Components are as follows:

1. Plugin manager and plugin manager class.
2. A service for the plugin manager
3. A validator plugin base class with methods to:
   a. loadAssets - this will initialize class properties pertaining to project, genus and file id. All validator plugin instance will have access to these values.
   b. getValidatorName - will return the name of the validator plugin as set in the plugin annotation definition.
   c. getValidatorScope - will return the scope/level the validation is required. Scope can be PROJECT, GENUS, DATA_FILE, TRAITS or VALUES.
4. Plugins titled with the level it is required to validate (ie. Project for PROJECT validator) will be a class that extends the validator base class and will have a validate() method that contains the validation logic required.

Plugin factory will be in the formValidate() of the Importer.
This is an example of executing Projects validator plugin.

$scope = 'PROJECT'; 
$manager = \Drupal::service('plugin.manager.trpcultivate_validator');
$plugins = $manager->getDefinitions();
$plugin_definitions = array_values($plugins);

// Get Project plugin validator definitions.
$plugin_key = array_search($scope, array_column($plugin_definitions, 'validator_scope'));
$validator = $plugin_definitions[ $plugin_key ]['id'];
$instance = $manager->createInstance($validator);

$instance->loadAssets($project, $genus, $file);
$validation_result[ $scope ] = $instance->validate();
...

G4.01 Service to associate genus with project

This service is the upgraded version of Analyzed Phenotypes Genus API with a focus on the new D10 functionality.

Service will handle setting of genus to project and resolving genus of a project. Mapping of genus is done using projectprop table where type_id is the configuration value of the term genus.

Service Name: TripalCultivatePhenotypesGenusProjectService
Found: trpcultivate_phenotypes/src/Service/TripalCultivatePhenotypesGenusProjectService.php

Associated Config:

trpcultivate.phenotypes.ontology.terms.genus

/**
 * Assign a genus to an experiment/project.
 * 
 * @param int $project
 *   Project (project id number) the parameter $genus will be assigned to.
 * @param string $genus
 *   Genus name/title.
 * @param boolean $replace
 *   True to replace existing genus of a project with a different genus.
 *   Default to False.
 *
 * @return boolean
 *   True, genus was set successfully or false on error/fail.
 */
function setGenusToProject($project, $genus, $replace = FALSE)

/**
 * Get genus of an experiment/project.
 * 
 * @param int $project
 *   Project (project_id number) to search.
 *
 * @return array
 *   Keys is genus/organism id number and value is the genus name/title.
 *    
 */
public function getGenusOfProject($project)

/**
 * Get all genus that have been configured (traits, unit, method, database and crop ontology).
 *
 * @return array
 *   An array of genus names, sorted alphabetically.
 */
public function getActiveGenus() 

g0-issue3-Configuration: Watermark Charts

Charts generated by Tripal Cultivate Phenotypes module may contain personalized water-makings. This configuration page will allow users to upload image and define watermark scheme.

Update testing grid to include Drupal 10.0 and 10.1 + PHP 8.2

## Compatibility

![Target Drupal Version 10.0.x-dev](https://img.shields.io/badge/Target%20Drupal%20Version-10.0.x-informational)

![PostgreSQL 13](https://img.shields.io/badge/PostreSQL-13-success)

| Drupal      | 9.4.x           | 9.5.x           | 10.0.x          | 10.1.x          |
|-------------|-----------------|-----------------|-----------------|-----------------|
| **PHP 8.0** | ![Grid1A-Badge] | ![Grid1B-Badge] |                 |                 |
| **PHP 8.1** | ![Grid2A-Badge] | ![Grid2B-Badge] | ![Grid2C-Badge] | ![Grid2D-Badge] |
| **PHP 8.2** |                 |                 | ![Grid3C-Badge] | ![Grid3D-Badge] |

[Grid1A-Badge]: https://github.com/tripal/tripal/actions/workflows/MAIN-phpunit-php8.0_D9_4x.yml/badge.svg
[Grid1B-Badge]: https://github.com/tripal/tripal/actions/workflows/MAIN-phpunit-php8.0_D9_5x.yml/badge.svg

[Grid2A-Badge]: https://github.com/tripal/tripal/actions/workflows/MAIN-phpunit-php8.1_D9_4x.yml/badge.svg
[Grid2B-Badge]: https://github.com/tripal/tripal/actions/workflows/MAIN-phpunit-php8.1_D9_5x.yml/badge.svg
[Grid2C-Badge]: https://github.com/tripal/tripal/actions/workflows/MAIN-phpunit-php8.1_D10_0x.yml/badge.svg
[Grid2D-Badge]: https://github.com/tripal/tripal/actions/workflows/MAIN-phpunit-php8.1_D10_1x.yml/badge.svg

[Grid3C-Badge]: https://github.com/tripal/tripal/actions/workflows/MAIN-phpunit-php8.2_D10_0x.yml/badge.svg
[Grid3D-Badge]: https://github.com/tripal/tripal/actions/workflows/MAIN-phpunit-php8.2_D10_1x.yml/badge.svg

G4.14 (G0.03.2) Service for managing CV and Cvterm configuration values

This service is the upgraded version of Analyzed Phenotypes Ontology API with a focus on the new D10 functionality.

Service Name: TripalCultivatePhenotypesGenusOntologyService
Found: trpcultivate_phenotypes/src/Service/TripalCultivatePhenotypesOntologyService.php

Associated Config:

Schema:

Genus:
-trait // Trait Vocabulary
-method // Method Vocabulary
-unit // Unit Vocabulary
-database // Database id
-crop_ontology // Crop Ontology Vocabulary

allownew // True or False, Allow new traits to be added during upload

Access to configuration values will be based on the genus.

Methods

/**
 * Fetch all genus from chado.organism in the host site and construct a genus ontology
 * configuration values described above. Each genus will contain a configuration value 
 * for trait+unit+method, database and crop ontology.
 *
 * @return array
 *    Associative array where each element is keyed by genus and configuration values array
 *    as the value.
 */
public function defineGenusOntology()

/**
 * Register a configuration entry and set each genus ontology configuration values to
 * a default value of 0 (not set).
 *
 * @return boolean
 *   True all genus ontology configuration created and set a default value, False on error.
 */
public function loadGenusOntology()

/**
 * Get genus ontology configuration values.
 *
 * @param string $genus
 *    Genus
 *
 * @return array
 *    Associated configuration values trait, unit, method, database and crop ontology.
 */

public function getGenusOntologyConfigValue($genus)

/**
 * Prepare genus text to be used as variable name by removing any formatting and converting spaces to underscore.
 *
 * @param string $genus
 *   Genus
 *
 * @return string
 *    Lowercase version and all spaces replaced by underscore.
 */

public function formatGenus($genus)

Typing errors

Drupal Version

9.4

Tripal Version

4.0.0-alpha1

Module Version

4.x dev (most recent commit)

Site Operating System

Debian Linux

Installation Method

TripalDocker docker

Browsers

No response

Observations

Rectify typographical errors in comments section.

Expected Behaviour

n/a

Steps to Reproduce

n/a

Relevant log output

No response