resostandards / web-api-commander

Client tools and libraries to work with and test RESO Web APIs.

RESO Commander

The RESO Commander is an OData client Java library and command-line client, as well as an automated RESO Certification testing tool.

Getting Started

To begin using the RESO Commander, choose one of the following topics:

RESO Certification

One of the Commander's primary uses is as an automated testing tool. This section explains how to run the following tests:

• Data Dictionary
• Web API Core

Command-Line OData Web API Tools

The RESO Commander contains command line tools for working with OData Web APIs. This section covers how to validate metadata, make requests, and generate sample requests for testing.

Web API Client Library

The RESO Commander offers a convenient wrapper around the Apache Olingo Java client for OData, with added support for OAuth2 Bearer Tokens and Client Credentials.

The client library can be used to fetch and validate metadata, fetch data, and perform other OData client tasks.

Docker

For those who prefer Docker, both the Command Line and Automated Testing Tools can be run in a Docker container.

Codegen

The RESO Commander may also be used to generate reference XML Metadata, DDL, database seeds, automated acceptance tests, reference server models, and to convert XML Metadata to Open API 3 format.

web-api-commander's Issues

Add RESOScript parameter validation to the Commander

Certain cases need to be validated, such as valid URLs being passed as well as format parameters.

Also, need to replace the submitted RESOScript requests for the given endorsement with the reference ones to ensure the correct requests are used. Have been doing this by hand, and it can be automated.

Convert Web API Server tests to Core, and remove Date Part comparisons.

This item is to track the conversion of Web API Server testing from Gold/Platinum to Core, which means removing any non-Core tests from the testing tool.

Items to be removed are:

• REQ-WA103-QO18
• REQ-WA103-QO19
• REQ-WA103-QO20
• REQ-WA103-QO21
• REQ-WA103-QO22
• REQ-WA103-QO23
• REQ-WA103-QO24

Removal of the testing rules for the DataSystem endpoint has also been recommended by the Certification subgroup.

Extend precision for fractionalseconds comparisons

Currently, fractionalseconds are only being compared to two places past the decimal, but some implementations offer more precision. This issue is to increase the precision of the comparisons.

Error in getMetadata

I'm trying to use test-commander to start to validate our RESO API. When I run the jar to do a "getMetadata", I get an error file generated with the following message:

ERROR: request failed!
URI: https://localhost:5001/odata/$metadata
Exception: org.apache.olingo.client.api.http.HttpClientException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Can you help me figure out why this is?

Finish $expand @platinum acceptance tests

This issue covers creating the glue code for the following $expand acceptance tests:

• REQ-WA103-QO29.2
• REQ-WA103-QO29.3
• REQ-WA103-QO29.4

See here for test descriptions.

Incorrect behavior with case-sensitivity in some tests

Consider making these tests case-insensitive:
REQ-WA103-QO13
REQ-WA103-QO14
REQ-WA103-QO15

These tests are likely noops in some databases:
REQ-WA103-QO16 (tolower)
REQ-WA103-QO17 (toupper)

Should these queries return the same count?
https://host/OData/DDv1.7/Property/$count?$filter=StreetName eq 'MAIN'
https://host/OData/DDv1.7/Property/$count?$filter=StreetName eq 'main'

Or these queries?
https://host/OData/DDv1.7/Property/$count?$filter=contains(StreetName, 'MAIN')
https://host/OData/DDv1.7/Property/$count?$filter=contains(StreetName, 'main')

The answer depends on the collation settings used by the underlying database. The OData and RESO specifications don't seem to offer an explicit opinion on the case-sensitivity of string values. The tests should be agnostic to casing to accommodate a given implementation's unspecified database collation settings.

Finish @geospatial @platinum acceptance tests

Previously, there were issues with the queries being used for the Geo-coordinate queries shown in the Cucumber .feature file here.

• REQ-WA103-QM5.1
• REQ-WA103-QM5.2

These queries should be fixed at this point, and have had the SRID=X; parameter added, so their tests can be finished at this point. Ideally, this could be done against a reference server who has already implemented this feature, and a recording could be captured for future regression testing in the Commander, but a mock could be manually created as well.

OData-Version response header false negative @REQ-WA103-RC11

This test is failing due to an apparent missing OData-Version header, however, I can confirm that our server is including "odata-version: 4.0" in the response headers for this exact request (see excerpt from logs at bottom). I'm guessing the check for that response header just needs to be made case insensitive.

@REQ-WA103-RC11 @core @2.5.2 @core-endorsement
Scenario: REQ-WA103-RC11 - 501 Not Implemented Request # src/main/java/org/reso/certification/features/web-api/web-api-server-1.0.2.feature:297
Using RESOScript: generic.gold.resoscript
Given a RESOScript file was provided # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:73)
RESOScript loaded successfully!
And Client Settings and Parameters were read from the file # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:80)
Bearer token loaded... first 4 characters:
Service root is: https://api.flexmls.com/Reso/OData
And a test container was successfully created from the given RESOScript # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:88)
Authentication Type: authorization_code
And the test container uses an authorization_code or client_credentials for authentication # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:95)
Request ID: REQ-WA103-RC11
Request URI: https://api.flexmls.com/Reso/OData/Property?$search=red%20OR%20blue
When a GET request is made to the resolved Url in "REQ-WA103-RC11" # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:311)
Asserted Response Code: 501, Server Response Code: 501
Then the server responds with a status code of 501 # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:316)
And the server has an OData-Version header value of "4.0" or "4.01" # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:1019)
java.lang.AssertionError: ERROR: must specify an 'OData-Version' in the response header!
See: http://docs.oasis-open.org/odata/odata/v4.0/errata03/os/complete/part1-protocol/odata-v4.0-errata03-os-part1-protocol-complete.html#_Toc453752225
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertNotNull(Assert.java:713)
at org.reso.certification.stepdefs.WebAPIServer_1_0_2.lambda$new$65(WebAPIServer_1_0_2.java:1023)
at ?.the server has an OData-Version header value of "4.0" or "4.01"(file:web-api-commander-master/src/main/java/org/reso/certification/features/web-api/web-api-server-1.0.2.feature:300)

[2020-04-06 13:35:40.572] [DEBUG] [headers] - http-outgoing-30 << HTTP/1.1 501 Not Implemented
[2020-04-06 13:35:40.572] [DEBUG] [headers] - http-outgoing-30 << Server: nginx
[2020-04-06 13:35:40.572] [DEBUG] [headers] - http-outgoing-30 << Date: Mon, 06 Apr 2020 18:35:42 GMT
[2020-04-06 13:35:40.572] [DEBUG] [headers] - http-outgoing-30 << Content-Type: application/json;odata.metadata=minimal
[2020-04-06 13:35:40.572] [DEBUG] [headers] - http-outgoing-30 << Connection: keep-alive
[2020-04-06 13:35:40.572] [DEBUG] [headers] - http-outgoing-30 << cache-control: no-cache

[2020-04-06 13:35:40.572] [DEBUG] [headers] - http-outgoing-30 << x-xss-protection: 1; mode=block
[2020-04-06 13:35:40.572] [DEBUG] [headers] - http-outgoing-30 << x-runtime: 0.034894
[2020-04-06 13:35:40.572] [DEBUG] [headers] - http-outgoing-30 << x-content-type-options: nosniff
[2020-04-06 13:35:40.572] [DEBUG] [headers] - http-outgoing-30 << odata-version: 4.0
[2020-04-06 13:35:40.572] [DEBUG] [headers] - http-outgoing-30 << x-powered-by: Phusion Passenger 5.0.6
[2020-04-06 13:35:40.572] [DEBUG] [headers] - http-outgoing-30 << status: 501 Not Implemented

DDL refinements

Improve Jenkins Builds in the Commander

There are certain changes needed to make the Commander more Jenkins friendly:

• Use scenario.write to log information about what was found rather than logging to standard error.
• When Data Dictionary tests fail, the code to remove the skipped tests to make the reports pretty for Jenkins and Cucumber doesn't run and those using Jenkins see both skipped tests and failed tests in this case. Since the number of tests in the DD is large, skipped tests need to be removed in the error case as well.

Rename Default Branch

The default branch has been renamed to main.

Please update your references accordingly:

git branch -m master main
git fetch origin
git branch -u origin/main main
git remote set-head origin -a```

Update Web API Testing Code with Web API Core changes

This issue is to track work on the following changes to the Web API Server Core 1.0.2 testing rules from the workgroup meetings:

• Remove DataSystem endpoint test (REQ-WA10x-END2). Vendors MAY still use this endpoint, but RESO won't test for it in Web API Server Core. An additional endorsement could be created in the future if there's interest.
• Remove OData data type testing rules (REQ-WA10x-QM1) from Web API Server Core.
• Remove OData query parameter case sensitivity testing rules (REQ-WA1xx-Q1) from Web API Server Core.
• Remove Year, Month, Day part tests in favor of ISO-8601 date and datetime queries.
• Add testing rules for 'eq' and 'ne' operators on OData Edm.EnumType enumerations.
• Add testing rules for OData any() and all() queries.

Prepare reporting output for Cert API

Once the Payloads sampling tool is done, the results will need to be posted to the RESO Cert API. This issue covers creating the certification data to be used for reporting.

Create Common Schema DDL Generator

The Commander currently supports generating XML Metadata as well as Cucumber BDD tests from a RESO Data Dictionary spreadsheet.

The goal of this task is to create a generator for DDL so that a reference database may be generated, including all relevant relationships and keys, etc., which are currently provided through Standard Relationships.

Improve Web API Summary Report Generator

When providers are certified, there's a JSON report with parameters that's generated from the config file used by the applicant.

We need to update the current generator to be inline with what the cert site accepts.

Add better error handling in the fetch-by-key test

Currently, the fetch-by-key Web API Core query will fail if an array of values is returned from the server, as it expects a single value.

If a values array is provided instead, the tests fail but the error messages aren't very informative. Add an error message that tests for an array of values in this case and informs the user to pass a singleton instead.

Clean up generic.resoscript input references

There is at least one XML entity in the generic resoscript file that makes a reference to a single quoted value, whereas it's actually looking for an unquoted value. 

web-api-commander/generic.resoscript

Line 125 in 648c263

<Parameter Name="EndpointResource" Value="REQUIRED: YOUR RESOURCE GOES HERE, FOR EXAMPLE 'Property'" />

Further down we see slightly different notation for what should be the same format :
https://github.com/RESOStandards/web-api-commander/blob/master/generic.resoscript#L174
https://github.com/RESOStandards/web-api-commander/blob/master/generic.resoscript#L179

Super minor overall, but would be good to clean these up and make them uniform. 

Add $count test to Web API Core

As was discussed in workgroup meetings, we omitted the $count=true option from the Web API Core tests and need to add it.

Add Scoring Functionality

Once #56 is complete, results need to be scored. This step will count and warn when differences in fields between access levels are found. For example, if an IDX field is present for one set of data but not another. This consists of running the same matching algorithm twice and therefore only one set of code is produced.

TODO:

• Add specific Payload handling since it was moved from the sampling phase - IDX scoring needs to handle that part.
• Finish Difference Reports
• Gradle Task
• README

Add support for empty collections with the all() operator

• all() may return either the exact value requested OR the empty list. All test needs to be adjusted to account for this case.

$expand Refinements

While working on the $expand tests, I had to make some tweaks. This issue captures the reasoning behind the changes that are in the related PR.

1. Line 895 reads like it belongs inside the if (clientProperty.getName().equals(expandField)) { statement (https://github.com/RESOStandards/web-api-commander/blob/master/src/main/java/org/reso/certification/stepdefs/WebAPIServer_1_0_2.java#L897). I haven't been able to get the type check to pass even with valid Navigation Property metadata. Maybe it should be removed entirely?

2. If the expand field's value is null, the test doesn't fail since there is no guarantee that every record will have a valid reference to other records.

Migrate Gradle/Groovy Script to Kotlin

Groovy causes a number of issues for JDK 14 and above.

While the Commander is usually run in a Docker container with JDK 10, many users have newer JDK versions installed. In order to maximize compatibility with these versions, the Groovy code should be migrated to Kotlin.

Issue with filter-coll-enum-all.json and filter-coll-enum-any.json

I've got tests failing based on the default queries for filter-coll-enum-all and filter-coll-enum-any. The filter clause for those queries are defined like this:
AccessibilityFeatures/all(enum: enum eq org.reso.metadata.enums.AccessibilityFeatures'AccessibleApproachWithRamp')

However, because I need to persist my enums as classes in my EntityModel, the enums are a property of the containing class. That would mean the filter query would need to be something like:

AccessibilityFeatures/all(enum: enum/MyEnumProperty eq org.reso.metadata.enums.AccessibilityFeatures'AccessibleApproachWithRamp')

where MyEnumProperty is the enum-based property within my AccessibilityFeatures lookup class that contains the lookup value.

Unless I'm missing something, this test doesn't seem to be canonical oData for an "all" query.

Thanks in advance!

"@orderby-timestamp-asc-filter-int-gt" and "@orderby-timestamp-asc" fail when processing null values

This appears to impact both of the tests noted in the issue title. Here's a copy/paste of test execution for one of them.

@orderby-timestamp-asc-filter-int-gt @2.4.4
Scenario: orderby-timestamp-asc-filter-int-gt - Query Support: $orderby timestamp asc # src/main/java/org/reso/certification/features/web-api/web-api-server.core.1.0.2.feature:110
Given a RESOScript file was provided # org.reso.certification.stepdefs.WebAPIServer.runBackground(WebAPIServer.java:1089)
And Client Settings and Parameters were read from the file # org.reso.certification.stepdefs.WebAPIServer.runBackground(WebAPIServer.java:1097)
And a test container was successfully created from the given RESOScript # org.reso.certification.stepdefs.WebAPIServer.runBackground(WebAPIServer.java:1105)
And the test container uses an authorization_code or client_credentials for authentication # org.reso.certification.stepdefs.WebAPIServer.runBackground(WebAPIServer.java:1119)
Given valid metadata have been retrieved # org.reso.certification.stepdefs.WebAPIServer.(WebAPIServer.java:1000)
Request ID: orderby-timestamp-asc-filter-int-gt
Request URI: https://api.flexmls.com/Reso/OData/Property?$top=20&$select=ListingKey,BedroomsTotal,OriginalEntryTimestamp&$orderby=OriginalEntryTimestamp%20asc&$filter=BedroomsTotal%20gt%201
Request succeeded...4354 bytes received.
When a GET request is made to the resolved Url in "orderby-timestamp-asc-filter-int-gt" # org.reso.certification.stepdefs.WebAPIServer.(WebAPIServer.java:251)
Asserted Response Code: 200, Server Response Code: 200
Then the server responds with a status code of 200 # org.reso.certification.stepdefs.WebAPIServer.(WebAPIServer.java:266)
Reported OData-Version header value: '4.0'
And the server has an OData-Version header value of "4.0" or "4.01" # org.reso.certification.stepdefs.WebAPIServer.(WebAPIServer.java:982)
Response is valid JSON!
And the response is valid JSON # org.reso.certification.stepdefs.WebAPIServer.(WebAPIServer.java:314)
Results count is: 20
And the response has results # org.reso.certification.stepdefs.WebAPIServer.(WebAPIServer.java:345)
Searching metadata for fields in given select list: [ListingKey, BedroomsTotal, OriginalEntryTimestamp]
And resource metadata for "Parameter_EndpointResource" contains the fields in the given select list # org.reso.certification.stepdefs.WebAPIServer.(WebAPIServer.java:737)
select list is: [ListingKey, BedroomsTotal, OriginalEntryTimestamp]
Number of Results: 20
Number of Fields: 3
Fields with Data: 58
Percent Fill: 96.7%
And data are present for fields contained within the given select list # org.reso.certification.stepdefs.WebAPIServer.(WebAPIServer.java:140)
And DateTimeOffset data in "Parameter_TimestampField" is sorted in "asc" order # org.reso.certification.stepdefs.WebAPIServer.(WebAPIServer.java:611)
java.lang.AssertionError: ERROR: java.lang.NullPointerException

The problem only seems to occur when a null value is encountered in the Parameter_TimestampField. Since an ascending sort is applied by the test we have no way around this one; we order nulls first.

Fix incomplete Client Settings tags in generic and generic.gold.resoscripts

It was recently pointed out that we have some incomplete client settings tags in the generic.resoscript and generic.gold.resoscript files. This issue has been created to fix them.

Create Data Generator Generators

The goal of this issue is to:

• build a cache of generator expressions from the Data Dictionary generator spreadsheet
• create Faker to read the DD sheet values and make the appropriate decisions about how to generate data for each data type

Validate metadata separately from web API cert tests

@darnjo, opened this one up per our email discussion.

When server metadata doesn't 100% pass validation a number of totally unrelated web API cert tests fail. Here's one example:

And resource metadata for "Parameter_EndpointResource" contains the fields in the given select list
java.lang.AssertionError: ERROR: Field name 'ListingKey' is not present in server metadata!

I can verify with 100% certainty that the ListingKey and BedsTotal fields are both in the metadata (although for the purpose of this test should that even matter?), so I'm wondering if it's choking on another aspect of the validation and throwing a false negative.

@REQ-WA103-QO4 @core @2.4.4 @filterability-endorsement
Scenario: REQ-WA103-QO4 - Query Support: $filter - Integer Comparison: gt # src/main/java/org/reso/certification/features/web-api/web-api-server-1.0.2.feature:141
Using RESOScript: generic.gold.resoscript
Given a RESOScript file was provided # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:73)
RESOScript loaded successfully!
And Client Settings and Parameters were read from the file # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:80)
Service root is: https://api.flexmls.com/Reso/OData
And a test container was successfully created from the given RESOScript # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:88)
Authentication Type: authorization_code
And the test container uses an authorization_code or client_credentials for authentication # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:95)
Request ID: REQ-WA103-QO4
Request URI: https://api.flexmls.com/Reso/OData/Property?$top=5&$select=ListingKey,BedroomsTotal&$filter=BedroomsTotal%20gt%209
Request succeeded...1018 bytes received.
When a GET request is made to the resolved Url in "REQ-WA103-QO4" # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:311)
Asserted Response Code: 200, Server Response Code: 200
Then the server responds with a status code of 200 # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:316)
Reported OData-Version header value: '4.0'
And the server has an OData-Version header value of "4.0" or "4.01" # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:1019)
Response is valid JSON!
Response: {
"@odata.context" : "https://api.flexmls.com/Reso/OData/$metadata#Property",
"@odata.nextLink" : "https://api.flexmls.com/Reso/OData/Property?%24filter=BedroomsTotal+gt+9&%24select=ListingKey%2CBedroomsTotal&%24skiptoken=20140916145247358266000000&%24top=5",
"value" : [ {
"ListingKey" : "20140916144954333968000000",
"BedroomsTotal" : 12,
"@odata.id" : "https://api.flexmls.com/Reso/OData/Property('20140916144954333968000000')"
}, {
"ListingKey" : "20140916145028952092000000",
"BedroomsTotal" : 14,
"@odata.id" : "https://api.flexmls.com/Reso/OData/Property('20140916145028952092000000')"
}, {
"ListingKey" : "20140916145039632072000000",
"BedroomsTotal" : 10,
"@odata.id" : "https://api.flexmls.com/Reso/OData/Property('20140916145039632072000000')"
}, {
"ListingKey" : "20140916145146671730000000",
"BedroomsTotal" : 10,
"@odata.id" : "https://api.flexmls.com/Reso/OData/Property('20140916145146671730000000')"
}, {
"ListingKey" : "20140916145247358266000000",
"BedroomsTotal" : 11,
"@odata.id" : "https://api.flexmls.com/Reso/OData/Property('20140916145247358266000000')"
} ]
}
And the response is valid JSON # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:353)
Results count is: 3
And the response has results # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:421)
Searching metadata for fields in given select list: [ListingKey, BedroomsTotal]
And resource metadata for "Parameter_EndpointResource" contains the fields in the given select list # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:771)
java.lang.AssertionError: ERROR: Field name 'ListingKey' is not present in server metadata!
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertNotNull(Assert.java:713)
at org.reso.certification.stepdefs.WebAPIServer_1_0_2.lambda$null$44(WebAPIServer_1_0_2.java:779)
at ?.resource metadata for "Parameter_EndpointResource" contains the fields in the given select list(file:web-api-commander-master/src/main/java/org/reso/certification/features/web-api/web-api-server-1.0.2.feature:147)

If the metadata is the blocker on these tests it may even be better to outright refuse to fully execute the suite until a set of metadata tests pass, thus turning it into a two step process:

1. Run metadata tests and verify all is well
2. Run web API cert

That feels quite a bit more performant compared to looping through metadata tests as a part of each individually web API cert scenario and makes the suite more resilient overall - eg. you can be sure that a failure in a test is related to the actual functionality you're trying to test.

Add documentation and README for Payloads testing tool

This work covers the documentation phase of the Payloads testing tool work.

DataSystem TimeZoneOffset

Error message during testing:

$.value[0].Resources[3].TimeZoneOffset: string found, integer expected

The value of TimeZoneOffset that's causing the error is "GMT+0".

According to the WebAPI 1.0.2 Spec, this is the correct format: https://members.reso.org/display/API2/2.6.1+DataSystem+Resource

TimeZoneOffset | The TimeZone Offset provided in standard TimeZone notation of GMT[+|-]X, where X is the number of hours in the offset.

Improve README and configs

The goal of this issue is to reorganize the README and formalize the tasks for the Payloads testing tool.

Create JSON Schema files for DD Schema Validation in Payloads Testing

The Commander currently supports generating XML Metadata as well as Cucumber BDD tests from a RESO Data Dictionary spreadsheet.

The goal of this task is to create a generator for JSON Schema so that schema validation may be performed on data returned during Payloads sampling to ensure it matches the RESO Common Schema and type information in cases where standard items exist.

Allow for certification of lightweight APIs

There is a business need to support certification of APIs that may not include a set of specific DD fields - for example a lightweight API that includes a small subset of fields; none of which are Integer or Decimal. With that in mind we need the capability to skip tests on such fields without failing certification.

Another important consideration here is that ModificationTimestamp is not always going to be filterable, primarily to enforce replication restrictions in favor of live calls.

From Josh Darnell:

The Commander used to ignore any test that had an empty request URL in it, but as I've been moving in the direction of loading a fixed set of reference queries that can't be skipped, might no longer be a good option and I'd removed that code anyway. Perhaps a better solution would be to only test on known requirement IDs that are in the config, and if they're not present, skip the tests that rely on them. In other words, given requirementID is present in the RESOScript, run test.

This sounds like a great path forward to me.

Add additional matching logic to DD testing tool

Additional matching was included in the DD specification that wasn't part of the MVP release, but should be added at some point. The DD matching code is like a sieve in the sense that we will continue to add matching techniques to catch cases that the previous matching heuristic didn't, which is aimed at reducing false positives from people using variations of standard items but not being flagged for them.

Currently the tool is using Levenshtein distance (allows variations for up to 25% of the word length), as well as case-insensitive fragment matching, both of which have a low error (false negative) rate when combined at around 5% (n=400). The Commander has a mechanism for dealing with false negatives by adding them to an ignored list, in which case they will never be offered as suggestions again.

Additional heuristics possible:

• Check against known variations, which we have from previous certification data.
• Substring Matching (catches cases like Bay in BayWindows). This is fairly noisy and would have to be used selectively, and perhaps for recommendations only and not failures.
• N-gram matching (should help catch more complex cases than Edit Distance alone).
• Use of a custom dictionary (Hunspell).

Add Data Dictionary Testing to Commander

This issue is for adding Data Dictionary testing to the RESO Commander.

See the Data Dictionary Test specification for more information.

In general, this means validation of the following:

• Metadata (either remotely or locally)
• String Fields
• Integer Fields
• Date Fields (ISO-8601 'yyyy-mm-dd' format)
• Decimal Fields
• Boolean Fields
• Single Enumerations (Edm.EnumType with IsFlags="false")
• Multiple Enumerations (either Edm.EnumType with IsFlags="true" or Collection(Edm.EnumType))
• Timestamp Fields (ISO-8601 format)

Additionally:

• Reporting
• Warnings for SHOULD items like length and precision
• Lookups tests
• Synonym Checking
• Similar Name Matching
• Added strict mode so that passing --Dstrict=true to automated tests will cause tests to fail on invalid metadata
• Update acceptance tests
• Update automated gradle build on GitHub
• Create all-in-one test/reporting task for DD testing
• Update README
• ----- MVP RELEASE -----

Outstanding Issues:

• Determine whether Skipped Scenarios can not be counted as Failed in the reports.

Bump Java version to at least 10

Not a huge priority, but it would be good to bump the Java version on the project. At the time of writing, the default for many platforms was still 1.8, but it would seem the community has now made it to 10+.

This means the following tasks:

• Change Gradle build and any local IDEs (local on your own machine)
• Verify Commander Client functionality (getting records, validating metadata).
• Ensure Cucumber tests still work.
• Update Docker build for Commander JAR.
• Update Docker build for Automated testing.
• Update README

XSLT possibly non-functional

Generating OAS doc using current build fails to create valid JSON doc, possibly due to https://github.com/oasis-tcs/odata-openapi/blob/master/tools/V4-CSDL-to-OpenAPI.xsl

Initial Payloads Sampling Tool

This issue covers the work on the initial sampling code that can iterate through a given resource and pull k records at a time. This work will come in two phases, single- and multi-threaded since in practice the sampling process will take too long to run on a single thread if there are multiple resources to sample.

• Single-threaded sampling
• Add multi-threading

Set encoding to UTF-8 and Fix Comparison Tests

There were a couple of items caught in pre-review of the Commander for the recent conference demo. Thanks to Trung for finding both of these issues.

• When running Gradle from the Windows command line, but not on the command line in OSX or Linux, and not in IntelliJ in Windows, OSX, or Linux, the response and file streams are not defaulting to UTF-8 and are therefore causing false negatives in validation.

• The comparisons for Integer ne aren't taking the possibility of null into account. Nulls were previously being filtered by the queries, but the filtering was removed so we could test more of a representative sample of data. The comparisons need to account for this.

Remove or clean up reference to Parameter_FilterHasField and Parameter_FilterHasValueLookupValue

web-api-commander/generic.resoscript

Line 258 in 648c263

<Parameter Name="SelectList"

These parameters have no equivalent "Required Fields and Values" section params so they come through null and cause a handful of tests to fail.

One test in particular is REQ-WA103-QO8

REQ-WA103-QO12 failing with NullPointerException

gradlew testWebAPIServer_1_0_2_Platinum -DshowResponses=true -DpathToRESOScript="..." -Dcucumber.filter.tags="@REQ-WA103-QO12"

@REQ-WA103-QO12 @platinum @2.4.4 @filterability-endorsement @filterability-endorsement
Scenario: REQ-WA103-QO12 - Query Support: $filter - Grouping: filter (ge, le) and (gt, lt) and expect (gt, lt) # src/main/java/org/reso/certification/features/web-api/web-api-server-1.0.2.feature:687
Using RESOScript:
...
And the OData client response has client entity set data # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:918)
java.lang.NullPointerException
at org.reso.certification.stepdefs.WebAPIServer_1_0_2.lambda$new$59(WebAPIServer_1_0_2.java:920)
at ?.the OData client response has client entity set data(file:///C:/Users/e5001999/web-api-commander/src/main/java/org/reso/certification/features/web-api/web-api-server-1.0.2.feature:692)
...

The error comes from here:

web-api-commander/src/main/java/org/reso/certification/stepdefs/WebAPIServer_1_0_2.java

Line 920 in 59c409d

assertTrue("ERROR: no results returned!", getTestContainer().getClientEntitySet().getCount() > 0);

Counting the returned entities seems to be the intent of the code. Consider this as a possible fix:
assertTrue("ERROR: no results returned!", getTestContainer().getClientEntitySet().getEntities().size() > 0);

Change metadata-validation to metadata-request

web-api-commander/web-api-server.core.1.0.2.resoscript

Line 227 in c9eefe4

RequestId="metadata-validation"

No output when setting up --outputFile=C:\metadata.XML in --getMetadata

Hi Team,
A faced with the issue when try to retrieve metadata into output file.
Steps to reproduce:
Preconditions: Windows 10, Java version "1.8.0_251".

1. Execute command according to Getting Metadata
   java -jar web-api-commander.jar --getMetadata --serviceRoot=https://myApi.com/api/v1/OData/endpoint--bearerToken=myToken --outputFile=C:\metadata.xml

Actual result: the execution of data is shown in web-api-commander\build\libs\commander.log file. The metadata is not written to C:\metadata.xml file.

Expected result: the execution of data is shown in web-api-commander\build\libs\commander.log file. The metadata is written to C:\metadata.xml file.

Create Data Dictionary Certification Walkthrough

• Create DD Certification Walkthrough on GitHub Wiki

Remove 403 and 501 Response Code Tests

These tests were determined to not be very effective for testing server functionality, and it's often hard to produce a 403 or 501 error consistently without some additional effort. Often the 403 test was returning 404 as written, and the 501 test was generating 400s or 500s.

Parameter case sensitivity @REQ-WA103-QO1.1

This question may need to be routed through the workgroup instead, but I'm starting here. This test is looking specifically for a 400 error when a parameter value like "$SeLeCt" is used, however, our server simply ignores the parameter and doesn't process it as a proper $select as opposed to throwing a 400. We believe this is just as valid of a solution and still adheres to case sensitivity. An equivalent scenario is throwing a garbage parameter name such as "$testing" at a server which, as far as we understand, is within the spec to outright ignore as well.

@REQ-WA103-QO1.1 @core @2.4.4 @core-endorsement @OData-4.0
Scenario: REQ-WA103-QO1.1 - Query Support: $select case-sensitivity for OData 4.0 # src/main/java/org/reso/certification/features/web-api/web-api-server-1.0.2.feature:91
Using RESOScript: generic.gold.resoscript
Given a RESOScript file was provided # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:73)
RESOScript loaded successfully!
And Client Settings and Parameters were read from the file # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:80)
Bearer token loaded... first 4 characters:
Service root is: https://api.flexmls.com/Reso/OData
And a test container was successfully created from the given RESOScript # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:88)
Authentication Type: authorization_code
And the test container uses an authorization_code or client_credentials for authentication # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:95)
Request ID: REQ-WA103-QO1.1
Request URI: https://api.flexmls.com/Reso/OData/Property?$top=5&$SeLeCt=ListingKey,BedroomsTotal&$filter=BedroomsTotal%20eq%209
Request succeeded...91370 bytes received.
When a GET request is made to the resolved Url in "REQ-WA103-QO1.1" # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:311)
Reported OData-Version header value: '4.0'
And the server has an OData-Version header value of "4.0" or "4.01" # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:1019)
Asserted OData Version: 4.0, Server Version: 4.0
Asserted Response Code: 400, Response code: 200
Then the server responds with a status code of 400 if the server reports OData-Version "4.0" # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:370)
java.lang.AssertionError: ERROR: asserted response code (400) does not match the one returned from the server (200) !
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.reso.certification.stepdefs.WebAPIServer_1_0_2.lambda$new$17(WebAPIServer_1_0_2.java:379)
at ?.the server responds with a status code of 400 if the server reports OData-Version "4.0"(file:web-api-commander-master/src/main/java/org/reso/certification/features/web-api/web-api-server-1.0.2.feature:94)

Migrate Platinum Tests to Separate Endorsements

In Issue #30, language around metallic certifications was removed and the Commander was made Web API Server Core compliant, however those other tests are still useful and need to be migrated to their own separate endorsements. They can be found at this commit.

Most likely this would look something like:

• $expand endorsement
• Geospatial endorsement
• String Filtering endorsement
• Collections Filtering endorsement

Some additional work on endorsements should also be done in order to advertise these features, most likely as capabilities on a given field as that's how OData was designed to work already and we can just expand on that ability.

Allow Edm.String and Collection(Edm.String) for String List, Single and Multi

As outlined in RESO RCP-031 as well as the Data Dictionary 1.7 Testing Specification, there is an established set of data type mappings RESO uses for each Data Dictionary object.

Previously, only OData Edm.EnumType enumerations were allowed for String List, Single items and OData.EnumType with IsFlags="true" OR Collection(Edm.EnumType) for String List, Multi enumerations.

However, RESO is moving towards the adoption of allowing strings and collections of strings for single- and multi-valued lookups once the Lookup resource proposal has been adopted and the type mappings have been approved for those cases now.

Lookups will be addressed in a later specification such as Data Dictionary 1.8 but their values aren't being tested in Data Dictionary 1.7, just their data type mappings.

Metadata Validation Fails on Unknown OData Types

Given Edm.DateTime rather than Edm.DateTimeOffset, the metadata validation portion (from the command line) will say the metadata has passed, but it could still contain invalid types.

• Try to fix XSDs (and submit to OASIS) in order to catch the type differences.
• Fix Commander metadata validation to catch type errors and handle them gracefully.

Parameter_ExpandField included in $select in a number of tests

web-api-commander/generic.resoscript

Line 731 in cccd671

Url="*ClientSettings_WebAPIURI*/*Parameter_EndpointResource*?$top=*Parameter_TopCount*&$select=*Parameter_SelectList*,*Parameter_ExpandField*&$expand=*Parameter_ExpandField*"

Parameter_ExpandField included in the $select param fails on our server since we're only allowing fields included in the $metadata to be used within that param. Including it in the $expand parameter should be enough to fulfill the obligations of the test, no?

Create Model Generators for Web API Reference Server

This issue tracks the work on the model generator for the Web API Reference Server project.