Proposal for a WMO Metadata quality KPI Scoring
The following issue summarizes a first draft proposal for providing a scoring algorithm for qualifying each individual metadata records (or information provided regarding the product information content).
This scoring mechanism is greatly inspired by the NOAA rubric (https://data.noaa.gov/metaview/page?xml=NOAA/NESDIS/NCDC/Geoportal//iso/xml/C00589.xml&view=rubricv2/recordHTML&header=none) and the Qualys SSL Labs "SSL Certificate" evaluation scoring explained here: https://github.com/ssllabs/research/wiki/SSL-Server-Rating-Guide
Results of an evaluation page can be checked here:
https://www.ssllabs.com/ssltest/index.html
KPI Types
Within the discussion and creation of KPIs and looking at other scoring engines for metadata quality 3 categories of KPIs are emerging:
-
Mandatory: It regroups all the rules that are necessary to comply to a WMO Core Profile 1.3 (compliance to Metadata schema and compliance to additional rules)
-
Required: That category regroups all the additional KPIs (the ones currently defined by TT-WMD) that have been judged necessary to create a meaningful metadata record
-
Recommended: this category is an optional category that is not required but makes a metadata record standing out of the rest of the record
Overall score mechanism
Following the Qualys labs, a two levels strategy can be adopted to evaluate a metadata record.
The overall score can be between 0 and 100 and mapped to a letter grade (A, B, C ,D) with the following translation table.
Letter grade translation
Numerical Score |
Grade |
score >= 80 |
A |
score >= 65 |
B |
score >= 50 |
C |
score >= 35 |
D |
score >= 20 |
E |
score < 20 |
F |
This score will be calculated only if the metadata is compliant to WMO Core profile 1.3. If not compliant a special Letter (U like uncompliant) should be provided.
Score computation
To compute the overall score over 100, the different KPIs are split in categories:
Here is an initial proposal of the categories:
Inside each category, there will be required and recommended KPIs. A 0 score for a required KPI brings a 0 in the complete category.
For instance, a 0 in the Title KPI would give the IdentificationInfo category a 0.
Then each category will receive a weighted value. For instance (numbers are to be defined).
Category |
Score |
Example of score for metadata |
C1 Identification Info |
20% |
85 over 100 |
C2 Coverage |
25% |
75 over 100 |
C3 Distribution Information |
25% |
55 over 100 |
C4 Data Policy |
15% |
45 over 100 |
C5 Enhancements |
15% |
65 over 100 |
Formula pending the overall scoring contribution is over 100 and the individual category scores are over 100:
(C1x0.2) + (C2x0.25)+(C3x0.25)+(C4x0.15)+(C5x0.15) = 66 % that would be B.
Individual Categorie Scoring
A category should contain multiple KPIs tests and should return a score over 100.
There are different strategies that can be applied: a ponderation similar applied to make the overall score:
Category A:
-
KPI 1 50%
-
KPI 2 25%
-
KPI3 25%
Or in the case that some of the KPIs results should be used or prevented it could impact the result.
For instance in the data policy, let’s imagine that only data policies with anchors are authorized but non anchor data policy are authorized for a transition period, and the metadata record contains both.
To define the total score, you can take the best way of encoding (anchor DP), add the non anchor DP score and divide by 2.
Non Compliance information
For each category and KPI, all individual errors should be presented: error title, show what XML element is evaluated and its value and then the value expected.
The NOAA rubric can be copied for that.
Example of the Qualys SSL Test web site
![screencapture-ssllabs-ssltest-analyze-html-2020-01-23-13_47_51](https://user-images.githubusercontent.com/1482/72986591-cd0b5380-3de8-11ea-9a7e-5607b7139d9b.png)