Introduction to the test
Code snippets in the documents (especially in docs/reference) are tested by this class.
Can be excuted by Gradle task ./gradlew :docs:integTest
The work is being done in oss-docs branch
To skip a code snippets being tested, we can exclude the file here:
https://github.com/opendistro-for-elasticsearch/search/blob/ab65227510d51cdefd68988b296325f92dad71a3/docs/build.gradle#L85
Test failures
Step to reproduce:
All tests: Running ./gradlew check or ./gradlew :docs:integTest
Specific test: such as running ./gradlew ':docs:integTest' --tests "org.elasticsearch.smoketest.DocsClientYamlTestSuiteIT.test {yaml=reference/release-notes/highlights/line_162}"
Update:
[02/17/2021] There are solutions for all the tests, but to be reviewed.
Reason for the failure
- Related to "Data Streams" feature
- API usage of "ILM":
Can be fixed by removing _ilm API uses in the code snippets.
The fix in "oss-docs" branch: 8ae568c
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/change-mappings-and-settings/ ...}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/set-up-a-data-stream/ ...}
DocsClientYamlTestSuiteIT. test {yaml=reference/indices/get-data-stream/ ...}
Caused by: java.lang.AssertionError: expected [2xx] status code but api [raw[method=PUT path=_ilm/policy/my-data-stream-policy]] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"invalid_index_name_exception","reason":"Invalid index name [_ilm], must not start with '_', '-', or '+'","index_uuid":"_na_","index":"_ilm","stack_trace":"[_ilm] InvalidIndexNameException[Invalid index name [_ilm], must not start with '_', '-', or '+']\n ...
- "no _data_stream meta field mapper"
Can be solved by partially reverting elastic/elasticsearch@40b9fd4 :
Adding DataStreamTimestampFieldMapper.java back.
Adding builtInMetadataMappers.put(DataStreamTimestampFieldMapper.NAME, DataStreamTimestampFieldMapper.PARSER); back to IndicesModule.java
The experimental fix in "oss-docs" branch: 49ad8de
(the following commits are used to fix the tests outside docs directory by keep reverting codes from above Elastic's commit: 0efae0a b863060 87b1932 6046a41 )
[update on 03/11/2021 - the final solution is to remove Data Stream from the documents, related PRs: #150 #226 #287]
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/use-a-data-stream/ ...}
DocsClientYamlTestSuiteIT. test {yaml=reference/indices/create-data-stream/line_14}
DocsClientYamlTestSuiteIT. test {yaml=reference/indices/data-stream-stats/ ...}
DocsClientYamlTestSuiteIT. test {yaml=reference/indices/delete-data-stream/line_25}
DocsClientYamlTestSuiteIT. test {yaml=reference/indices/put-index-template/line_301}
DocsClientYamlTestSuiteIT. test {yaml=reference/indices/rollover-index/line_231}
Caused by: java.lang.AssertionError: expected [2xx] status code but api [raw[method=PUT path=_index_template/my-data-stream-template]] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"composable template [my-data-stream-template] template after composition is invalid","stack_trace":"[composable template [my-data-stream-template] template after composition is invalid]; nested: IllegalArgumentException[composable template [my-data-stream-template] template after composition is invalid]; nested: XContentParseException[[index_template] unknown field [data_stream]];
https://github.com/opendistro-for-elasticsearch/search/blob/7114dd3d8c1e20c44358e03a78d9170578711eb4/server/src/main/java/org/elasticsearch/cluster/metadata/MetadataIndexTemplateService.java#L1125
- "x-pack" contents to be removed
index.routing.allocation.include._tier_preference
Can be fixed by reverting the docs updates in the commit elastic/elasticsearch@1bfebd5
The fix in "oss-docs" branch: 52d87eb 5b8c6c2
DocsClientYamlTestSuiteIT. test {yaml=reference/api-conventions/line_404}
DocsClientYamlTestSuiteIT. test {yaml=reference/api-conventions/line_435}
DocsClientYamlTestSuiteIT. test {yaml=reference/migration/migrate_7_0/api/line_41}
Caused by: java.lang.AssertionError: $body didn't match expected value:
$body:
my-index-000001:
settings:
index.creation_date: same [1613368525450]
index.number_of_replicas: same [1]
index.number_of_shards: same [1]
index.provided_name: same [my-index-000001]
index.routing.allocation.include._tier_preference: expected [data_content] but not found
index.uuid: same [ac0pdW0ATsqIGy7wgzSu8A]
index.version.created: same [7100399]
- Freeze index
Can be fixed by removing the use of freeze index API
The fix in "oss-docs" branch: 32b8f37
DocsClientYamlTestSuiteIT. test {yaml=reference/indices/resolve/line_50}
Caused by: java.lang.AssertionError: expected [2xx] status code but api [raw[method=POST path=freeze-index/_freeze]] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"parse_exception","reason":"request body is required","stack_trace":"ElasticsearchParseException[request body is required] ...
- Point in time API
DocsClientYamlTestSuiteIT. test {yaml=reference/release-notes/highlights/line_129}
DocsClientYamlTestSuiteIT. test {yaml=reference/release-notes/highlights/line_162}
DocsClientYamlTestSuiteIT. test {yaml=reference/search/search-your-data/paginate-search-results/line_178}
DocsClientYamlTestSuiteIT. test {yaml=reference/search/search-your-data/paginate-search-results/line_55}
For the first two, can be fixed by removing the description of PIT from release highlights for ES 7.10.
For the last two, can be fixed by reverting the docs before PIT is introduced (elastic/elasticsearch@1b51acb).
The fix in "oss-docs" branch: 79fdaaf 025219f
Caused by: java.lang.AssertionError: expected [2xx] status code but api [raw[method=POST path=my-index-000001/_pit]] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"parse_exception","reason":"request body is required","stack_trace":"ElasticsearchParseException[request body is required] ...
- EQL
Can be fixed by removing the description of EQL from release highlights for ES 7.10.
The fix in "oss-docs" branch: 79fdaaf
DocsClientYamlTestSuiteIT. test {yaml=reference/release-notes/highlights/line_203}
Caused by: java.lang.AssertionError: expected [2xx] status code but api [raw[method=GET path=my-index-000001/_eql/search]] returned [404 Not Found] [{"_index":"my-index-000001","_type":"_eql","_id":"search","found":false}]
- "Constant Keyword" field
Can be fixed by removing the paragraphs by reverting the changes in elastic/elasticsearch@cb868d2
The fix in "oss-docs" branch: 76008d5
DocsClientYamlTestSuiteIT. test {yaml=reference/how-to/search-speed/line_447}
Caused by: java.lang.AssertionError: expected [2xx] status code but api [raw[method=PUT path=bicycles]] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"No handler for type [constant_keyword] declared on field [cycle_type]","stack_trace":"MapperParsingException[No handler for type [constant_keyword] declared on field [cycle_type]]\n\tat org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:314) ...
- Keystore password unset for oss distribution
Can be fixed by setting keystorepassword in build script even for "oss" distribution, which is only set for "default" distribution.
The fix in "oss-docs" branch: 57a9992 (only the first line of code change in the commit)
DocsClientYamlTestSuiteIT. test {yaml=reference/cluster/nodes-reload-secure-settings/line_63}
Caused by: java.lang.AssertionError: $body didn't match expected value:
$body:
_nodes:
failed: same [0]
successful: same [1]
total: same [1]
cluster_name: same [integTest]
nodes:
5FH51NZASVuQstuRK3I-FA:
name: same [node-0]
reload_exception: unexpected but found [{type=security_exception, reason=Provided keystore password was incorrect, caused_by={type=i_o_exception, reason=javax.crypto.AEADBadTagException: Tag mismatch!, caused_by={type=a_e_a_d_bad_tag_exception, reason=Tag mismatch!}}}]
Appendix: all failures in the generated test report
DocsClientYamlTestSuiteIT. test {yaml=reference/api-conventions/line_404}
DocsClientYamlTestSuiteIT. test {yaml=reference/api-conventions/line_435}
DocsClientYamlTestSuiteIT. test {yaml=reference/cluster/nodes-reload-secure-settings/line_63}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/change-mappings-and-settings/line_128}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/change-mappings-and-settings/line_149}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/change-mappings-and-settings/line_182}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/change-mappings-and-settings/line_217}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/change-mappings-and-settings/line_243}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/change-mappings-and-settings/line_287}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/change-mappings-and-settings/line_312}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/change-mappings-and-settings/line_341}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/change-mappings-and-settings/line_396}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/change-mappings-and-settings/line_443}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/change-mappings-and-settings/line_495}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/change-mappings-and-settings/line_516}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/change-mappings-and-settings/line_544}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/change-mappings-and-settings/line_592}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/change-mappings-and-settings/line_616}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/change-mappings-and-settings/line_647}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/change-mappings-and-settings/line_669}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/change-mappings-and-settings/line_99}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/set-up-a-data-stream/line_31}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/use-a-data-stream/line_110}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/use-a-data-stream/line_122}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/use-a-data-stream/line_138}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/use-a-data-stream/line_146}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/use-a-data-stream/line_162}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/use-a-data-stream/line_206}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/use-a-data-stream/line_231}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/use-a-data-stream/line_257}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/use-a-data-stream/line_321}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/use-a-data-stream/line_335}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/use-a-data-stream/line_345}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/use-a-data-stream/line_60}
DocsClientYamlTestSuiteIT. test {yaml=reference/data-streams/use-a-data-stream/line_80}
DocsClientYamlTestSuiteIT. test {yaml=reference/how-to/search-speed/line_447}
DocsClientYamlTestSuiteIT. test {yaml=reference/indices/create-data-stream/line_14}
DocsClientYamlTestSuiteIT. test {yaml=reference/indices/data-stream-stats/line_173}
DocsClientYamlTestSuiteIT. test {yaml=reference/indices/data-stream-stats/line_51}
DocsClientYamlTestSuiteIT. test {yaml=reference/indices/delete-data-stream/line_25}
DocsClientYamlTestSuiteIT. test {yaml=reference/indices/get-data-stream/line_190}
DocsClientYamlTestSuiteIT. test {yaml=reference/indices/get-data-stream/line_65}
DocsClientYamlTestSuiteIT. test {yaml=reference/indices/put-index-template/line_301}
DocsClientYamlTestSuiteIT. test {yaml=reference/indices/resolve/line_50}
DocsClientYamlTestSuiteIT. test {yaml=reference/indices/rollover-index/line_231}
DocsClientYamlTestSuiteIT. test {yaml=reference/migration/migrate_7_0/api/line_41}
DocsClientYamlTestSuiteIT. test {yaml=reference/release-notes/highlights/line_129}
DocsClientYamlTestSuiteIT. test {yaml=reference/release-notes/highlights/line_162}
DocsClientYamlTestSuiteIT. test {yaml=reference/release-notes/highlights/line_203}
DocsClientYamlTestSuiteIT. test {yaml=reference/search/search-your-data/paginate-search-results/line_178}
DocsClientYamlTestSuiteIT. test {yaml=reference/search/search-your-data/paginate-search-results/line_55}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent