resostandards / web-api-commander Goto Github PK
View Code? Open in Web Editor NEWClient tools and libraries to work with and test RESO Web APIs.
Home Page: https://commander.reso.org
License: Other
Client tools and libraries to work with and test RESO Web APIs.
Home Page: https://commander.reso.org
License: Other
gradlew testWebAPIServer_1_0_2_Platinum -DshowResponses=true -DpathToRESOScript="..." -Dcucumber.filter.tags="@REQ-WA103-QO12"
@REQ-WA103-QO12 @platinum @2.4.4 @filterability-endorsement @filterability-endorsement
Scenario: REQ-WA103-QO12 - Query Support: $filter - Grouping: filter (ge, le) and (gt, lt) and expect (gt, lt) # src/main/java/org/reso/certification/features/web-api/web-api-server-1.0.2.feature:687
Using RESOScript:
...
And the OData client response has client entity set data # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:918)
java.lang.NullPointerException
at org.reso.certification.stepdefs.WebAPIServer_1_0_2.lambda$new$59(WebAPIServer_1_0_2.java:920)
at ?.the OData client response has client entity set data(file:///C:/Users/e5001999/web-api-commander/src/main/java/org/reso/certification/features/web-api/web-api-server-1.0.2.feature:692)
...
The error comes from here:
Counting the returned entities seems to be the intent of the code. Consider this as a possible fix:
assertTrue("ERROR: no results returned!", getTestContainer().getClientEntitySet().getEntities().size() > 0);
Groovy causes a number of issues for JDK 14 and above.
While the Commander is usually run in a Docker container with JDK 10, many users have newer JDK versions installed. In order to maximize compatibility with these versions, the Groovy code should be migrated to Kotlin.
Not a huge priority, but it would be good to bump the Java version on the project. At the time of writing, the default for many platforms was still 1.8, but it would seem the community has now made it to 10+.
This means the following tasks:
README
Previously, there were issues with the queries being used for the Geo-coordinate queries shown in the Cucumber .feature
file here.
These queries should be fixed at this point, and have had the SRID=X;
parameter added, so their tests can be finished at this point. Ideally, this could be done against a reference server who has already implemented this feature, and a recording could be captured for future regression testing in the Commander, but a mock could be manually created as well.
In Issue #30, language around metallic certifications was removed and the Commander was made Web API Server Core compliant, however those other tests are still useful and need to be migrated to their own separate endorsements. They can be found at this commit.
Most likely this would look something like:
$expand
endorsementSome additional work on endorsements should also be done in order to advertise these features, most likely as capabilities on a given field as that's how OData was designed to work already and we can just expand on that ability.
Once #56 is complete, results need to be scored. This step will count and warn when differences in fields between access levels are found. For example, if an IDX field is present for one set of data but not another. This consists of running the same matching algorithm twice and therefore only one set of code is produced.
TODO:
This appears to impact both of the tests noted in the issue title. Here's a copy/paste of test execution for one of them.
@orderby-timestamp-asc-filter-int-gt @2.4.4
Scenario: orderby-timestamp-asc-filter-int-gt - Query Support: $orderby timestamp asc # src/main/java/org/reso/certification/features/web-api/web-api-server.core.1.0.2.feature:110
Given a RESOScript file was provided # org.reso.certification.stepdefs.WebAPIServer.runBackground(WebAPIServer.java:1089)
And Client Settings and Parameters were read from the file # org.reso.certification.stepdefs.WebAPIServer.runBackground(WebAPIServer.java:1097)
And a test container was successfully created from the given RESOScript # org.reso.certification.stepdefs.WebAPIServer.runBackground(WebAPIServer.java:1105)
And the test container uses an authorization_code or client_credentials for authentication # org.reso.certification.stepdefs.WebAPIServer.runBackground(WebAPIServer.java:1119)
Given valid metadata have been retrieved # org.reso.certification.stepdefs.WebAPIServer.(WebAPIServer.java:1000)
Request ID: orderby-timestamp-asc-filter-int-gt
Request URI: https://api.flexmls.com/Reso/OData/Property?$top=20&$select=ListingKey,BedroomsTotal,OriginalEntryTimestamp&$orderby=OriginalEntryTimestamp%20asc&$filter=BedroomsTotal%20gt%201
Request succeeded...4354 bytes received.
When a GET request is made to the resolved Url in "orderby-timestamp-asc-filter-int-gt" # org.reso.certification.stepdefs.WebAPIServer.(WebAPIServer.java:251)
Asserted Response Code: 200, Server Response Code: 200
Then the server responds with a status code of 200 # org.reso.certification.stepdefs.WebAPIServer.(WebAPIServer.java:266)
Reported OData-Version header value: '4.0'
And the server has an OData-Version header value of "4.0" or "4.01" # org.reso.certification.stepdefs.WebAPIServer.(WebAPIServer.java:982)
Response is valid JSON!
And the response is valid JSON # org.reso.certification.stepdefs.WebAPIServer.(WebAPIServer.java:314)
Results count is: 20
And the response has results # org.reso.certification.stepdefs.WebAPIServer.(WebAPIServer.java:345)
Searching metadata for fields in given select list: [ListingKey, BedroomsTotal, OriginalEntryTimestamp]
And resource metadata for "Parameter_EndpointResource" contains the fields in the given select list # org.reso.certification.stepdefs.WebAPIServer.(WebAPIServer.java:737)
select list is: [ListingKey, BedroomsTotal, OriginalEntryTimestamp]
Number of Results: 20
Number of Fields: 3
Fields with Data: 58
Percent Fill: 96.7%
And data are present for fields contained within the given select list # org.reso.certification.stepdefs.WebAPIServer.(WebAPIServer.java:140)
And DateTimeOffset data in "Parameter_TimestampField" is sorted in "asc" order # org.reso.certification.stepdefs.WebAPIServer.(WebAPIServer.java:611)
java.lang.AssertionError: ERROR: java.lang.NullPointerException
The problem only seems to occur when a null value is encountered in the Parameter_TimestampField. Since an ascending sort is applied by the test we have no way around this one; we order nulls first.
This test is failing due to an apparent missing OData-Version header, however, I can confirm that our server is including "odata-version: 4.0" in the response headers for this exact request (see excerpt from logs at bottom). I'm guessing the check for that response header just needs to be made case insensitive.
@REQ-WA103-RC11 @core @2.5.2 @core-endorsement
Scenario: REQ-WA103-RC11 - 501 Not Implemented Request # src/main/java/org/reso/certification/features/web-api/web-api-server-1.0.2.feature:297
Using RESOScript: generic.gold.resoscript
Given a RESOScript file was provided # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:73)
RESOScript loaded successfully!
And Client Settings and Parameters were read from the file # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:80)
Bearer token loaded... first 4 characters:
Service root is: https://api.flexmls.com/Reso/OData
And a test container was successfully created from the given RESOScript # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:88)
Authentication Type: authorization_code
And the test container uses an authorization_code or client_credentials for authentication # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:95)
Request ID: REQ-WA103-RC11
Request URI: https://api.flexmls.com/Reso/OData/Property?$search=red%20OR%20blue
When a GET request is made to the resolved Url in "REQ-WA103-RC11" # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:311)
Asserted Response Code: 501, Server Response Code: 501
Then the server responds with a status code of 501 # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:316)
And the server has an OData-Version header value of "4.0" or "4.01" # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:1019)
java.lang.AssertionError: ERROR: must specify an 'OData-Version' in the response header!
See: http://docs.oasis-open.org/odata/odata/v4.0/errata03/os/complete/part1-protocol/odata-v4.0-errata03-os-part1-protocol-complete.html#_Toc453752225
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertNotNull(Assert.java:713)
at org.reso.certification.stepdefs.WebAPIServer_1_0_2.lambda$new$65(WebAPIServer_1_0_2.java:1023)
at ?.the server has an OData-Version header value of "4.0" or "4.01"(file:web-api-commander-master/src/main/java/org/reso/certification/features/web-api/web-api-server-1.0.2.feature:300)
[2020-04-06 13:35:40.572] [DEBUG] [headers] - http-outgoing-30 << HTTP/1.1 501 Not Implemented
[2020-04-06 13:35:40.572] [DEBUG] [headers] - http-outgoing-30 << Server: nginx
[2020-04-06 13:35:40.572] [DEBUG] [headers] - http-outgoing-30 << Date: Mon, 06 Apr 2020 18:35:42 GMT
[2020-04-06 13:35:40.572] [DEBUG] [headers] - http-outgoing-30 << Content-Type: application/json;odata.metadata=minimal
[2020-04-06 13:35:40.572] [DEBUG] [headers] - http-outgoing-30 << Connection: keep-alive
[2020-04-06 13:35:40.572] [DEBUG] [headers] - http-outgoing-30 << cache-control: no-cache
[2020-04-06 13:35:40.572] [DEBUG] [headers] - http-outgoing-30 << x-xss-protection: 1; mode=block
[2020-04-06 13:35:40.572] [DEBUG] [headers] - http-outgoing-30 << x-runtime: 0.034894
[2020-04-06 13:35:40.572] [DEBUG] [headers] - http-outgoing-30 << x-content-type-options: nosniff
[2020-04-06 13:35:40.572] [DEBUG] [headers] - http-outgoing-30 << odata-version: 4.0
[2020-04-06 13:35:40.572] [DEBUG] [headers] - http-outgoing-30 << x-powered-by: Phusion Passenger 5.0.6
[2020-04-06 13:35:40.572] [DEBUG] [headers] - http-outgoing-30 << status: 501 Not Implemented
web-api-commander/generic.resoscript
Line 258 in 648c263
These parameters have no equivalent "Required Fields and Values" section params so they come through null and cause a handful of tests to fail.
One test in particular is REQ-WA103-QO8
Consider making these tests case-insensitive:
REQ-WA103-QO13
REQ-WA103-QO14
REQ-WA103-QO15
These tests are likely noops in some databases:
REQ-WA103-QO16 (tolower)
REQ-WA103-QO17 (toupper)
Should these queries return the same count?
https://host/OData/DDv1.7/Property/$count?$filter=StreetName eq 'MAIN'
https://host/OData/DDv1.7/Property/$count?$filter=StreetName eq 'main'
Or these queries?
https://host/OData/DDv1.7/Property/$count?$filter=contains(StreetName, 'MAIN')
https://host/OData/DDv1.7/Property/$count?$filter=contains(StreetName, 'main')
The answer depends on the collation settings used by the underlying database. The OData and RESO specifications don't seem to offer an explicit opinion on the case-sensitivity of string values. The tests should be agnostic to casing to accommodate a given implementation's unspecified database collation settings.
Once the Payloads sampling tool is done, the results will need to be posted to the RESO Cert API. This issue covers creating the certification data to be used for reporting.
There were a couple of items caught in pre-review of the Commander for the recent conference demo. Thanks to Trung for finding both of these issues.
When running Gradle from the Windows command line, but not on the command line in OSX or Linux, and not in IntelliJ in Windows, OSX, or Linux, the response and file streams are not defaulting to UTF-8 and are therefore causing false negatives in validation.
The comparisons for Integer ne
aren't taking the possibility of null
into account. Nulls were previously being filtered by the queries, but the filtering was removed so we could test more of a representative sample of data. The comparisons need to account for this.
Additional matching was included in the DD specification that wasn't part of the MVP release, but should be added at some point. The DD matching code is like a sieve in the sense that we will continue to add matching techniques to catch cases that the previous matching heuristic didn't, which is aimed at reducing false positives from people using variations of standard items but not being flagged for them.
Currently the tool is using Levenshtein distance (allows variations for up to 25% of the word length), as well as case-insensitive fragment matching, both of which have a low error (false negative) rate when combined at around 5% (n=400). The Commander has a mechanism for dealing with false negatives by adding them to an ignored list, in which case they will never be offered as suggestions again.
Additional heuristics possible:
Given Edm.DateTime rather than Edm.DateTimeOffset, the metadata validation portion (from the command line) will say the metadata has passed, but it could still contain invalid types.
This issue covers creating the glue code for the following $expand
acceptance tests:
See here for test descriptions.
I'm trying to use test-commander to start to validate our RESO API. When I run the jar to do a "getMetadata", I get an error file generated with the following message:
ERROR: request failed!
URI: https://localhost:5001/odata/$metadata
Exception: org.apache.olingo.client.api.http.HttpClientException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Can you help me figure out why this is?
As outlined in RESO RCP-031 as well as the Data Dictionary 1.7 Testing Specification, there is an established set of data type mappings RESO uses for each Data Dictionary object.
Previously, only OData Edm.EnumType
enumerations were allowed for String List, Single items and OData.EnumType
with IsFlags="true"
OR Collection(Edm.EnumType)
for String List, Multi enumerations.
However, RESO is moving towards the adoption of allowing strings and collections of strings for single- and multi-valued lookups once the Lookup resource proposal has been adopted and the type mappings have been approved for those cases now.
Lookups will be addressed in a later specification such as Data Dictionary 1.8 but their values aren't being tested in Data Dictionary 1.7, just their data type mappings.
Currently, the fetch-by-key
Web API Core query will fail if an array of values is returned from the server, as it expects a single value.
If a values
array is provided instead, the tests fail but the error messages aren't very informative. Add an error message that tests for an array of values in this case and informs the user to pass a singleton instead.
There is a business need to support certification of APIs that may not include a set of specific DD fields - for example a lightweight API that includes a small subset of fields; none of which are Integer or Decimal. With that in mind we need the capability to skip tests on such fields without failing certification.
Another important consideration here is that ModificationTimestamp is not always going to be filterable, primarily to enforce replication restrictions in favor of live calls.
From Josh Darnell:
The Commander used to ignore any test that had an empty request URL in it, but as I've been moving in the direction of loading a fixed set of reference queries that can't be skipped, might no longer be a good option and I'd removed that code anyway. Perhaps a better solution would be to only test on known requirement IDs that are in the config, and if they're not present, skip the tests that rely on them. In other words, given requirementID is present in the RESOScript, run test.
This sounds like a great path forward to me.
The default branch has been renamed to main
.
Please update your references accordingly:
git branch -m master main
git fetch origin
git branch -u origin/main main
git remote set-head origin -a```
The Commander currently supports generating XML Metadata as well as Cucumber BDD tests from a RESO Data Dictionary spreadsheet.
The goal of this task is to create a generator for DDL so that a reference database may be generated, including all relevant relationships and keys, etc., which are currently provided through Standard Relationships.
This question may need to be routed through the workgroup instead, but I'm starting here. This test is looking specifically for a 400 error when a parameter value like "$SeLeCt" is used, however, our server simply ignores the parameter and doesn't process it as a proper $select as opposed to throwing a 400. We believe this is just as valid of a solution and still adheres to case sensitivity. An equivalent scenario is throwing a garbage parameter name such as "$testing" at a server which, as far as we understand, is within the spec to outright ignore as well.
@REQ-WA103-QO1.1 @core @2.4.4 @core-endorsement @OData-4.0
Scenario: REQ-WA103-QO1.1 - Query Support: $select case-sensitivity for OData 4.0 # src/main/java/org/reso/certification/features/web-api/web-api-server-1.0.2.feature:91
Using RESOScript: generic.gold.resoscript
Given a RESOScript file was provided # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:73)
RESOScript loaded successfully!
And Client Settings and Parameters were read from the file # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:80)
Bearer token loaded... first 4 characters:
Service root is: https://api.flexmls.com/Reso/OData
And a test container was successfully created from the given RESOScript # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:88)
Authentication Type: authorization_code
And the test container uses an authorization_code or client_credentials for authentication # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:95)
Request ID: REQ-WA103-QO1.1
Request URI: https://api.flexmls.com/Reso/OData/Property?$top=5&$SeLeCt=ListingKey,BedroomsTotal&$filter=BedroomsTotal%20eq%209
Request succeeded...91370 bytes received.
When a GET request is made to the resolved Url in "REQ-WA103-QO1.1" # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:311)
Reported OData-Version header value: '4.0'
And the server has an OData-Version header value of "4.0" or "4.01" # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:1019)
Asserted OData Version: 4.0, Server Version: 4.0
Asserted Response Code: 400, Response code: 200
Then the server responds with a status code of 400 if the server reports OData-Version "4.0" # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:370)
java.lang.AssertionError: ERROR: asserted response code (400) does not match the one returned from the server (200) !
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.reso.certification.stepdefs.WebAPIServer_1_0_2.lambda$new$17(WebAPIServer_1_0_2.java:379)
at ?.the server responds with a status code of 400 if the server reports OData-Version "4.0"(file:web-api-commander-master/src/main/java/org/reso/certification/features/web-api/web-api-server-1.0.2.feature:94)
There is at least one XML entity in the generic resoscript file that makes a reference to a single quoted value, whereas it's actually looking for an unquoted value.
web-api-commander/generic.resoscript
Line 125 in 648c263
Further down we see slightly different notation for what should be the same format :
https://github.com/RESOStandards/web-api-commander/blob/master/generic.resoscript#L174
https://github.com/RESOStandards/web-api-commander/blob/master/generic.resoscript#L179
Super minor overall, but would be good to clean these up and make them uniform.
Certain cases need to be validated, such as valid URLs being passed as well as format parameters.
Also, need to replace the submitted RESOScript requests for the given endorsement with the reference ones to ensure the correct requests are used. Have been doing this by hand, and it can be automated.
There are certain changes needed to make the Commander more Jenkins friendly:
scenario.write
to log information about what was found rather than logging to standard error.As was discussed in workgroup meetings, we omitted the $count=true
option from the Web API Core tests and need to add it.
While working on the $expand
tests, I had to make some tweaks. This issue captures the reasoning behind the changes that are in the related PR.
Line 895 reads like it belongs inside the if (clientProperty.getName().equals(expandField)) {
statement (https://github.com/RESOStandards/web-api-commander/blob/master/src/main/java/org/reso/certification/stepdefs/WebAPIServer_1_0_2.java#L897). I haven't been able to get the type
check to pass even with valid Navigation Property metadata. Maybe it should be removed entirely?
If the expand
field's value is null, the test doesn't fail since there is no guarantee that every record will have a valid reference to other records.
This item is to track the conversion of Web API Server testing from Gold/Platinum to Core, which means removing any non-Core tests from the testing tool.
Items to be removed are:
Removal of the testing rules for the DataSystem endpoint has also been recommended by the Certification subgroup.
all()
may return either the exact value requested OR the empty list. All test needs to be adjusted to account for this case.@darnjo, opened this one up per our email discussion.
When server metadata doesn't 100% pass validation a number of totally unrelated web API cert tests fail. Here's one example:
And resource metadata for "Parameter_EndpointResource" contains the fields in the given select list
java.lang.AssertionError: ERROR: Field name 'ListingKey' is not present in server metadata!
I can verify with 100% certainty that the ListingKey and BedsTotal fields are both in the metadata (although for the purpose of this test should that even matter?), so I'm wondering if it's choking on another aspect of the validation and throwing a false negative.
@REQ-WA103-QO4 @core @2.4.4 @filterability-endorsement
Scenario: REQ-WA103-QO4 - Query Support: $filter - Integer Comparison: gt # src/main/java/org/reso/certification/features/web-api/web-api-server-1.0.2.feature:141
Using RESOScript: generic.gold.resoscript
Given a RESOScript file was provided # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:73)
RESOScript loaded successfully!
And Client Settings and Parameters were read from the file # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:80)
Service root is: https://api.flexmls.com/Reso/OData
And a test container was successfully created from the given RESOScript # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:88)
Authentication Type: authorization_code
And the test container uses an authorization_code or client_credentials for authentication # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:95)
Request ID: REQ-WA103-QO4
Request URI: https://api.flexmls.com/Reso/OData/Property?$top=5&$select=ListingKey,BedroomsTotal&$filter=BedroomsTotal%20gt%209
Request succeeded...1018 bytes received.
When a GET request is made to the resolved Url in "REQ-WA103-QO4" # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:311)
Asserted Response Code: 200, Server Response Code: 200
Then the server responds with a status code of 200 # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:316)
Reported OData-Version header value: '4.0'
And the server has an OData-Version header value of "4.0" or "4.01" # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:1019)
Response is valid JSON!
Response: {
"@odata.context" : "https://api.flexmls.com/Reso/OData/$metadata#Property",
"@odata.nextLink" : "https://api.flexmls.com/Reso/OData/Property?%24filter=BedroomsTotal+gt+9&%24select=ListingKey%2CBedroomsTotal&%24skiptoken=20140916145247358266000000&%24top=5",
"value" : [ {
"ListingKey" : "20140916144954333968000000",
"BedroomsTotal" : 12,
"@odata.id" : "https://api.flexmls.com/Reso/OData/Property('20140916144954333968000000')"
}, {
"ListingKey" : "20140916145028952092000000",
"BedroomsTotal" : 14,
"@odata.id" : "https://api.flexmls.com/Reso/OData/Property('20140916145028952092000000')"
}, {
"ListingKey" : "20140916145039632072000000",
"BedroomsTotal" : 10,
"@odata.id" : "https://api.flexmls.com/Reso/OData/Property('20140916145039632072000000')"
}, {
"ListingKey" : "20140916145146671730000000",
"BedroomsTotal" : 10,
"@odata.id" : "https://api.flexmls.com/Reso/OData/Property('20140916145146671730000000')"
}, {
"ListingKey" : "20140916145247358266000000",
"BedroomsTotal" : 11,
"@odata.id" : "https://api.flexmls.com/Reso/OData/Property('20140916145247358266000000')"
} ]
}
And the response is valid JSON # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:353)
Results count is: 3
And the response has results # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:421)
Searching metadata for fields in given select list: [ListingKey, BedroomsTotal]
And resource metadata for "Parameter_EndpointResource" contains the fields in the given select list # org.reso.certification.stepdefs.WebAPIServer_1_0_2.(WebAPIServer_1_0_2.java:771)
java.lang.AssertionError: ERROR: Field name 'ListingKey' is not present in server metadata!
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertNotNull(Assert.java:713)
at org.reso.certification.stepdefs.WebAPIServer_1_0_2.lambda$null$44(WebAPIServer_1_0_2.java:779)
at ?.resource metadata for "Parameter_EndpointResource" contains the fields in the given select list(file:web-api-commander-master/src/main/java/org/reso/certification/features/web-api/web-api-server-1.0.2.feature:147)
If the metadata is the blocker on these tests it may even be better to outright refuse to fully execute the suite until a set of metadata tests pass, thus turning it into a two step process:
That feels quite a bit more performant compared to looping through metadata tests as a part of each individually web API cert scenario and makes the suite more resilient overall - eg. you can be sure that a failure in a test is related to the actual functionality you're trying to test.
This issue tracks the work on the model generator for the Web API Reference Server project.
This issue is for adding Data Dictionary testing to the RESO Commander.
See the Data Dictionary Test specification for more information.
In general, this means validation of the following:
Edm.EnumType
with IsFlags="false"
)Edm.EnumType
with IsFlags="true"
or Collection(Edm.EnumType)
)Additionally:
--Dstrict=true
to automated tests will cause tests to fail on invalid metadataOutstanding Issues:
This issue covers the work on the initial sampling code that can iterate through a given resource and pull k records at a time. This work will come in two phases, single- and multi-threaded since in practice the sampling process will take too long to run on a single thread if there are multiple resources to sample.
Currently, fractionalsecond
s are only being compared to two places past the decimal, but some implementations offer more precision. This issue is to increase the precision of the comparisons.
The goal of this issue is to reorganize the README and formalize the tasks for the Payloads testing tool.
These tests were determined to not be very effective for testing server functionality, and it's often hard to produce a 403 or 501 error consistently without some additional effort. Often the 403 test was returning 404 as written, and the 501 test was generating 400s or 500s.
When providers are certified, there's a JSON report with parameters that's generated from the config file used by the applicant.
We need to update the current generator to be inline with what the cert site accepts.
It was recently pointed out that we have some incomplete client settings tags in the generic.resoscript and generic.gold.resoscript files. This issue has been created to fix them.
This work covers the documentation phase of the Payloads testing tool work.
This issue is to track work on the following changes to the Web API Server Core 1.0.2 testing rules from the workgroup meetings:
any()
and all()
queries.I've got tests failing based on the default queries for filter-coll-enum-all and filter-coll-enum-any. The filter clause for those queries are defined like this:
AccessibilityFeatures/all(enum: enum eq org.reso.metadata.enums.AccessibilityFeatures'AccessibleApproachWithRamp')
However, because I need to persist my enums as classes in my EntityModel, the enums are a property of the containing class. That would mean the filter query would need to be something like:
AccessibilityFeatures/all(enum: enum/MyEnumProperty eq org.reso.metadata.enums.AccessibilityFeatures'AccessibleApproachWithRamp')
where MyEnumProperty is the enum-based property within my AccessibilityFeatures lookup class that contains the lookup value.
Unless I'm missing something, this test doesn't seem to be canonical oData for an "all" query.
Thanks in advance!
Error message during testing:
$.value[0].Resources[3].TimeZoneOffset: string found, integer expected
The value of TimeZoneOffset
that's causing the error is "GMT+0".
According to the WebAPI 1.0.2 Spec, this is the correct format: https://members.reso.org/display/API2/2.6.1+DataSystem+Resource
TimeZoneOffset | The TimeZone Offset provided in standard TimeZone notation of GMT[+|-]X, where X is the number of hours in the offset.
The goal of this issue is to:
web-api-commander/generic.resoscript
Line 731 in cccd671
Parameter_ExpandField included in the $select param fails on our server since we're only allowing fields included in the $metadata to be used within that param. Including it in the $expand parameter should be enough to fulfill the obligations of the test, no?
The Commander currently supports generating XML Metadata as well as Cucumber BDD tests from a RESO Data Dictionary spreadsheet.
The goal of this task is to create a generator for JSON Schema so that schema validation may be performed on data returned during Payloads sampling to ensure it matches the RESO Common Schema and type information in cases where standard items exist.
Hi Team,
A faced with the issue when try to retrieve metadata into output file.
Steps to reproduce:
Preconditions: Windows 10, Java version "1.8.0_251".
java -jar web-api-commander.jar --getMetadata --serviceRoot=https://myApi.com/api/v1/OData/endpoint--bearerToken=myToken --outputFile=C:\metadata.xml
Actual result: the execution of data is shown in web-api-commander\build\libs\commander.log file. The metadata is not written to C:\metadata.xml file.
Expected result: the execution of data is shown in web-api-commander\build\libs\commander.log file. The metadata is written to C:\metadata.xml file.
Generating OAS doc using current build fails to create valid JSON doc, possibly due to https://github.com/oasis-tcs/odata-openapi/blob/master/tools/V4-CSDL-to-OpenAPI.xsl
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.