opensearch-project / opensearch-java

Java Client for OpenSearch

OpenSearch Java Client

• Welcome!
• Sample Code
• Project Resources
• Code of Conduct
• User Guide
• Snapshot Builds
• Compatibility with OpenSearch
• Security
• License
• Copyright

Welcome!

opensearch-java is a community-driven, open source fork of elasticsearch-java licensed under the Apache v2.0 License. For more information, see opensearch.org. This client is meant to replace the existing OpenSearch Java High Level REST Client.

Sample Code

Please see the USER_GUIDE for code snippets.

Project Resources

• Project Website
• Documentation
• JavaDoc
• Maven Central
• Snapshot Builds
• Need help? Try Forums
• Project Principles
• Contributing to OpenSearch
• Maintainer Responsibilities
• Release Management
• Admin Responsibilities
• Security

Code of Conduct

This project has adopted the Amazon Open Source Code of Conduct. For more information see the Code of Conduct FAQ, or contact [email protected] with any additional questions or comments.

User Guide

See User Guide.

Snapshot Builds

The snapshots builds are published to sonatype using publish-snapshots.yml workflow. Each push event to the main branch triggers this workflow.

Compatibility with OpenSearch

See Compatibility.

Security

If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our vulnerability reporting page or directly via email to [email protected]. Please do not create a public GitHub issue.

License

This project is licensed under the Apache v2.0 License.

Copyright

Copyright OpenSearch Contributors. See NOTICE for details.

opensearch-java's Issues

set javadoc encoding to UTF-8

Refer comment here: #49

jackson-databind-2.13.2.jar: 1 vulnerabilities (highest severity is: 7.5) - autoclosed

Vulnerable Library - jackson-databind-2.13.2.jar

General data-binding functionality for Jackson: works on core streaming API

Library home page: http://github.com/FasterXML/jackson

Path to dependency file: /java-client/build.gradle.kts

Path to vulnerable library: /aches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.13.2/926e48c451166a291f1ce6c6276d9abbefa7c00f/jackson-databind-2.13.2.jar,/home/wss-scanner/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.13.2/926e48c451166a291f1ce6c6276d9abbefa7c00f/jackson-databind-2.13.2.jar

Found in HEAD commit: 02a313f39fd898c43748f649621cef9fb8cfe84e

Vulnerabilities

CVE	Severity	CVSS	Dependency	Type	Fixed in	Remediation Available
CVE-2020-36518	High	7.5	jackson-databind-2.13.2.jar	Direct	com.fasterxml.jackson.core:jackson-databind:2.12.6.1,2.13.2.1	✅

Details

CVE-2020-36518

Vulnerable Library - jackson-databind-2.13.2.jar

General data-binding functionality for Jackson: works on core streaming API

Library home page: http://github.com/FasterXML/jackson

Path to dependency file: /java-client/build.gradle.kts

Path to vulnerable library: /aches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.13.2/926e48c451166a291f1ce6c6276d9abbefa7c00f/jackson-databind-2.13.2.jar,/home/wss-scanner/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.13.2/926e48c451166a291f1ce6c6276d9abbefa7c00f/jackson-databind-2.13.2.jar

Dependency Hierarchy:

• ❌ jackson-databind-2.13.2.jar (Vulnerable Library)

Found in HEAD commit: 02a313f39fd898c43748f649621cef9fb8cfe84e

Found in base branch: main

Vulnerability Details

jackson-databind before 2.13.0 allows a Java StackOverflow exception and denial of service via a large depth of nested objects.
WhiteSource Note: After conducting further research, WhiteSource has determined that all versions of com.fasterxml.jackson.core:jackson-databind up to version 2.13.2 are vulnerable to CVE-2020-36518.

Publish Date: 2022-03-11

URL: CVE-2020-36518

CVSS 3 Score Details (7.5)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: None
  • Scope: Unchanged
• Impact Metrics:
  • Confidentiality Impact: None
  • Integrity Impact: None
  • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: FasterXML/jackson-databind#2816

Release Date: 2022-03-11

Fix Resolution: com.fasterxml.jackson.core:jackson-databind:2.12.6.1,2.13.2.1

⛑️ Automatic Remediation is available for this issue

---

⛑️ Automatic Remediation is available for this issue.

Update maven file name

Can we avoid creating the maven-metadata-local.xml file and rename to maven-metadata.xml. The fix for this shouldn't be in the publishing job but we should update the build output itself with expected file name.

PR: #53

[BUG] Test failure

What is the bug?
Test failure in linux system

How can one reproduce the bug?

Run ./gradlew clean unitTest
Downloading https://services.gradle.org/distributions/gradle-6.8.3-bin.zip
..........10%..........20%..........30%...........40%..........50%..........60%..........70%...........80%..........90%..........100%

Welcome to Gradle 6.8.3!

Here are the highlights of this release:
 - Faster Kotlin DSL script compilation
 - Vendor selection for Java toolchains
 - Convenient execution of tasks in composite builds
 - Consistent dependency resolution

For more details see https://docs.gradle.org/6.8.3/release-notes.html

Starting a Gradle Daemon (subsequent builds will be faster)
> Task :buildSrc:compileJava NO-SOURCE
> Task :buildSrc:compileGroovy NO-SOURCE
> Task :buildSrc:processResources NO-SOURCE
> Task :buildSrc:classes UP-TO-DATE
> Task :buildSrc:jar
> Task :buildSrc:assemble
> Task :buildSrc:compileTestJava NO-SOURCE
> Task :buildSrc:compileTestGroovy NO-SOURCE
> Task :buildSrc:processTestResources NO-SOURCE
> Task :buildSrc:testClasses UP-TO-DATE
> Task :buildSrc:test NO-SOURCE
> Task :buildSrc:check UP-TO-DATE
> Task :buildSrc:build
> Task :java-client:clean UP-TO-DATE

> Task :java-client:compileJava
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :java-client:processResources NO-SOURCE
> Task :java-client:classes
> Task :java-client:compileTestJava
> Task :java-client:processTestResources NO-SOURCE
> Task :java-client:testClasses

> Task :java-client:unitTest

org.opensearch.clients.opensearch.experiments.ParsingTests > testVariants FAILED
    java.lang.NullPointerException at ParsingTests.java:133

12 tests completed, 1 failed


> Task :java-client:unitTest FAILED
FAILURE: Build failed with an exception.
4 actionable tasks: 3 executed, 1 up-to-date

* What went wrong:
Execution failed for task ':java-client:unitTest'.
> There were failing tests. See the report at: file:///home/runner/work/opensearch-java/opensearch-java/java-client/build/reports/tests/unitTest/index.html

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 1m 22s
Error: Process completed with exit code 1.

What is the expected behavior?
Test should succeed.

What is your host/environment?

• OS: [ubuntu/AL 2012] any linux distribution
• Version [e.g. 22]
• Plugins

Do you have any screenshots?
N/A

Do you have any additional context?
N/A

checkstyle-8.37.jar: 1 vulnerabilities (highest severity is: 3.3)

Vulnerable Library - checkstyle-8.37.jar

Path to dependency file: /build.gradle.kts

Path to vulnerable library: /home/wss-scanner/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/29.0-jre/801142b4c3d0f0770dd29abea50906cacfddd447/guava-29.0-jre.jar

Found in HEAD commit: 10b6c21f7429e46f367fbd8d7cc0d23297893f93

Vulnerabilities

CVE	Severity	CVSS	Dependency	Type	Fixed in	Remediation Available
CVE-2020-8908	Low	3.3	guava-29.0-jre.jar	Transitive	N/A	❌

Details

CVE-2020-8908

Vulnerable Library - guava-29.0-jre.jar

Guava is a suite of core and expanded libraries that include utility classes, google's collections, io classes, and much much more.

Library home page: https://github.com/google/guava

Path to dependency file: /build.gradle.kts

Path to vulnerable library: /home/wss-scanner/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/29.0-jre/801142b4c3d0f0770dd29abea50906cacfddd447/guava-29.0-jre.jar

Dependency Hierarchy:

• checkstyle-8.37.jar (Root Library)
  • ❌ guava-29.0-jre.jar (Vulnerable Library)

Found in HEAD commit: 10b6c21f7429e46f367fbd8d7cc0d23297893f93

Found in base branch: main

Vulnerability Details

A temp directory creation vulnerability exists in all versions of Guava, allowing an attacker with access to the machine to potentially access data in a temporary directory created by the Guava API com.google.common.io.Files.createTempDir(). By default, on unix-like systems, the created directory is world-readable (readable by an attacker with access to the system). The method in question has been marked @deprecated in versions 30.0 and later and should not be used. For Android developers, we recommend choosing a temporary directory API provided by Android, such as context.getCacheDir(). For other Java developers, we recommend migrating to the Java 7 API java.nio.file.Files.createTempDirectory() which explicitly configures permissions of 700, or configuring the Java runtime's java.io.tmpdir system property to point to a location whose permissions are appropriately configured.

Publish Date: 2020-12-10

URL: CVE-2020-8908

CVSS 3 Score Details (3.3)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Local
  • Attack Complexity: Low
  • Privileges Required: Low
  • User Interaction: None
  • Scope: Unchanged
• Impact Metrics:
  • Confidentiality Impact: Low
  • Integrity Impact: None
  • Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8908

Release Date: 2020-12-10

Fix Resolution: v30.0

Where is PercolateQueryBuilder class in opensearch java client ?

I don't see org.opensearch.percolator.PercolateQueryBuilder in opensearch high level client, i am using below dependency

    <dependency>
        <groupId>org.opensearch.client</groupId>
        <artifactId>opensearch-rest-high-level-client</artifactId>
        <version>1.2.3</version>
    </dependency>

What is the alternate for org.elasticsearch.percolator.PercolateQueryBuilder

Add auto backport functionality

Coming from meta issue opensearch-project/opensearch-clients#14:

• Add auto-backport PR creation Github action to make sure all the PRs are backported for a particular release.
• Add auto-delete for merged auto-backport PRs to clean up the branches created during auto-backport.
• Add documentation for the same.

Ensure DCO Workflow Check

Coming from opensearch-project/project-meta#17

A Developer Certificate of Origin is required on commits in the OpenSearch-Project.

See doc.yml for an example workflow. Ensure CONTRIBUTING.md to has a section on the DCO per the project template.

• DCO Check Workflow
• CONTRIBUTING.md DCO Section

[FEATURE] add a module-info.java to support JPMS properly

I'm currently trying to build a modular jar of my app that depends on:

<dependency>
    <groupId>org.opensearch.client</groupId>
    <artifactId>opensearch-rest-high-level-client</artifactId>
    <version>1.2.4</version>
    <scope>compile</scope>
</dependency>

but as the opensearch java clients don't have module-info.java files it's defaulting to try and do the modularity based on the classes in the file. This would potentially be okay if it wasn't for the fact that the org.opensearch.client package is available in two places.

module myapp reads package org.opensearch.client from both opensearch.rest.high.level.client and opensearch.rest.client

This is probably something that can be solved by properly supporting the module system.

[BUG] publish to maven local is not publishing sha1 and md5 checksums

What is the bug?
./gradlew publishToMavenLocal does not generate the checksums

How can one reproduce the bug?
Steps to reproduce the behavior:

1. /gradlew publishToMavenLocal
2. check the files

What is the expected behavior?
sha1 and md5 checksums should be generated

Do you have any additional context?
work around has been added here: https://github.com/opensearch-project/opensearch-java/pull/51/files#diff-e21da378399d7f35543533d279a39bf3d1024092580f40330fd8bc56379d6164R87-R109
which can be removed once the checksum generation is handled by gradle

Add features that are not part of upstream

• Sync with Upstream
• Remove Xpack feature
• Remove docs

Supporting async search using RestHighLevelClient

We are trying to port our existing code using RestHighLevelClient from elastic search to OpenSearch.

For elastic search we used code similar to:

SearchSourceBuilder searchSource = new SearchSourceBuilder()
        .query(QueryBuilders.matchAllQuery()); 
String[] indices = new String[] { "my-index" }; 
SubmitAsyncSearchRequest request
        = new SubmitAsyncSearchRequest(searchSource, indices);

we trying to find implemented similar behavior with OpenSeacrh but could not find any examples/documentation.
We understand that probably implementation of async search with OpenSearch is different from elastic and is done using plugins as per https://opensearch.org/docs/latest/search-plugins/async/index/
Our question is whether async search can be invoked using RestHighLevelClient or some other java client and if not when supporting of async search is planned?

Thank you

[Setup] Integration Test Setup

What is the issue?
I'm setting up integration tests. Using intellij I was able to debug that asserts in the PR are passing, but gradle doesn't exit.

---

How can one reproduce the issue?
Steps to reproduce the behavior:

docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.0

./gradlew test --tests "org.opensearch.clients.opensearch.integTest.PingAndInfoIT" -Dtests.rest.cluster="localhost:9200" -Dtests.cluster="localhost:9200" -Dtests.clustername="docker-cluster" --no-daemon --debug

---

What is the expected behavior?
Once all asserts pass, it's expected that process is terminated successfully.

---

What is your host/environment?

• OS: Mac OS X 10.15.7 x86_64
• Code: #16

[BUG] bulk updates failing with Json RawValue

Hello guys!

What is the bug?
When trying to update parts of documents with a Json RawValue, it fails with a parse exception saying the field we are trying to modify is unknown. However, a same operation with the normal update request seems to work.

How can one reproduce the bug?
For example I have this indexed in OpenSearch:

{
  "message": "trying out OpenSearch",
  "field":"Should be unchanged"
}

Try to do a bulk update on the field message with a Json RawValue:

BulkRequest.Builder bulkBuilder = new BulkRequest.Builder();
bulkBuilder.operations(
    op -> op.update(idx -> idx
        .index("index")
        .id("1")
        .document(new RawValue("{\"message\": \"mastering out OpenSearch\"}"))
        .routing("1")
    ));
client.bulk(bulkBuilder.build());

Getting the error:

org.opensearch.client.opensearch._types.OpenSearchException: Request failed: [x_content_parse_exception] [1:2] [UpdateRequest] unknown field [message]

While if I use the normal update method it's fine:

client.update(new UpdateRequest.Builder()
    .index("index")
    .id("1")
    .doc(new RawValue("{\"message\": \"mastering out OpenSearch\"}"))
    .routing("1")
    .build());

What is the expected behavior?
It should work like for a normal update in this case?

[BUG] Cannot Send Bulk Requests

What is the bug?
When attempting to use the .bulk endpoint, the following error is thrown: "The bulk request must be terminated by a newline [\n]"

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Build the json for a bulk request (either adding entries to a JsonObject or building a JsonString)
2. Build a BulkRequest giving your json as the value
3. Pass the BulkRequest as the parameter for the client.bulk() endpoint
4. See error message

What is the expected behavior?
The bulk request should be accepted and actioned.

What is your host/environment?

• OS: macOS Big Sur
• Version 11.6
• oOpenJDK version 1.8.0
• OpenJDK Runtime Environment Corretto-8.302.08.1

Do you have any additional context?
I attempted to use both Jakarta JsonObject and Jakarta JsonString. With the object I used the builder .add method to add entries for the actions and the document information, with the JsonString I built a string that held the json and ended in the new line character and that still didn't work.

Add Developer Guide

Add developer guide to help contributors to set up development environment

[FEATURE] Add integration test badges to README

Is your feature request related to a problem?
Lots of awesome tests in https://github.com/opensearch-project/opensearch-java/pull/122/files, but I like CI/CD badges so I know tests are passing on main.

What solution would you like?
Add badges to README.

[FEATURE] Add support for AWS Sigv4 request signer

Add support for signed requests via AWS sigv4.

[BUG] /_nodes/stats throwing serialization error

What is the bug?

import org.opensearch.clients.opensearch.nodes.StatsRequest;
import org.opensearch.clients.opensearch.nodes.StatsResponse;
...
...
StatsRequest statsRequest = new StatsRequest.Builder().build();
StatsResponse statsResponse = client.nodes().stats(statsRequest);
...
...

Throws:

jakarta.json.stream.JsonParsingException: Unexpected JSON event 'VALUE_STRING', expected [START_ARRAY]

What is the expected behavior?
Node stats response.

cannot use opensearch java client2.0

The official document of version 2.0 uses the Java client sample code, but cannot find the class.how to use java client 2.0?

my jdk is 8

[FEATURE] Reduce the size of the test matrix

Is your feature request related to a problem?

We run too many tests in the CI matrix, takes a long time and is wasteful.

What solution would you like?

• Remove JDK 14, 15, only run 11 and 17
• Run integration tests against OpenSearch on a single JDK (e.g. some on 11, some on 17)

Do you have any additional context?

Use a matrix like this: https://github.com/slack-ruby/slack-ruby-bot-server/blob/master/.github/workflows/test-mongodb.yml#L10

Add `publishing` tasks

Add publishing task with opensearch details.
This was removed in #33

Security plugin endpoints

Is your feature request related to a problem?
The security plugin provides endpoints such as creating users and roles. Those endpoints don't have a client equivalent (it also doesn't look like the security plugin has a client that we can use).

What solution would you like?
I would like to be able to create/update/delete users and roles from a high level rest client.

[BUG] TermsQuery should be a list of terms, not only a single term

What is the bug?

TermsQuery is implemented as having a single value, which makes it identical to TermQuery, but in reality it should be a list of values which is the whole purpose of this query type.

What is the expected behavior?

The value property should be values, and a vararg list of values to match.

Do you have any additional context?
ES 7.10 docs

Add Developer Guide / Testing requirements

If a user follows current README, they will see build failure due to integration test. Please update instruction accordingly. Also, add separate sections for unit testing and dev testing steps for developers.

Milestone 2 tasks

In this milestone, we focus on cleaning up existing code and make it ready for OpenSearch development

• Remove X-Pack and non oss features/tests
• Update namespace
• Updates license headers
• Remove docs
• Remove commits not related to >7.10 from main branch
• Update CI to run with OpenDistro docker image with out security plugin

Test and publish compatibility matrix with various OpenSearch versions

Add support to run integration tests across various versions of OpenSearch and publish a compatibility matrix for the client version vs OpenSearch version.

PR Reference: #122
Compatibility: https://github.com/opensearch-project/opensearch-java/blob/main/COMPATIBILITY.md

[FEATURE] Pull dependency versions from OpenSearch

Is your feature request related to a problem?
Coming from: #160 (comment)
All dependencies java client needs are manually updated but not fetched from centralized location.
This causes pain to update dependencies in all repositories.

What solution would you like?
One solution we could use a centralized version catalog by gradle[1]. (Suggested by @reta)

[1] https://docs.gradle.org/current/userguide/platforms.html

Remove references to deprecated service: JCenter

From opensearch-project/opensearch-build#1456 and https://discuss.opendistrocommunity.dev/t/jcenter-build-issues/8293

What’s the problem?
JCenter, which is a repository used in our builds, ceased operations today. Gradle was pulling dependencies from JCenter, and some repos have a direct dependency on it. This broke that the build process for OpenSearch engine, all branches so that the last successful build of 1.3 was at 7:03 AM PT Jan 12th.

References in this repository, query.

jackson-databind-2.12.6.jar: 1 vulnerabilities (highest severity is: 7.5)

Vulnerable Library - jackson-databind-2.12.6.jar

General data-binding functionality for Jackson: works on core streaming API

Library home page: http://github.com/FasterXML/jackson

Path to dependency file: /java-client/build.gradle.kts

Path to vulnerable library: /home/wss-scanner/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.12.6/fac216b606c1086e36acea6e572ee61572ad1670/jackson-databind-2.12.6.jar,/aches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.12.6/fac216b606c1086e36acea6e572ee61572ad1670/jackson-databind-2.12.6.jar

Vulnerabilities

CVE	Severity	CVSS	Dependency	Type	Fixed in	Remediation Available
CVE-2020-36518	High	7.5	jackson-databind-2.12.6.jar	Direct	jackson-databind-2.10 - 2.10.1;com.fasterxml.jackson.core.jackson-databind - 2.6.2.v20161117-2150	❌

Details

CVE-2020-36518

Vulnerable Library - jackson-databind-2.12.6.jar

General data-binding functionality for Jackson: works on core streaming API

Library home page: http://github.com/FasterXML/jackson

Path to dependency file: /java-client/build.gradle.kts

Path to vulnerable library: /home/wss-scanner/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.12.6/fac216b606c1086e36acea6e572ee61572ad1670/jackson-databind-2.12.6.jar,/aches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.12.6/fac216b606c1086e36acea6e572ee61572ad1670/jackson-databind-2.12.6.jar

Dependency Hierarchy:

• ❌ jackson-databind-2.12.6.jar (Vulnerable Library)

Found in base branch: main

Vulnerability Details

jackson-databind before 2.13.0 allows a Java StackOverflow exception and denial of service via a large depth of nested objects.
WhiteSource Note: After conducting further research, WhiteSource has determined that all versions of com.fasterxml.jackson.core:jackson-databind up to version 2.13.2 are vulnerable to CVE-2020-36518.

Publish Date: 2022-03-11

URL: CVE-2020-36518

CVSS 3 Score Details (7.5)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: None
  • Scope: Unchanged
• Impact Metrics:
  • Confidentiality Impact: None
  • Integrity Impact: None
  • Availability Impact: High

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://nvd.nist.gov/vuln/detail/CVE-2020-36518

Release Date: 2022-03-11

Fix Resolution: jackson-databind-2.10 - 2.10.1;com.fasterxml.jackson.core.jackson-databind - 2.6.2.v20161117-2150

Release version 2.0.0 for Java client

Release the client version 2.0.0. Before releasing, the following items need to be done:

• Use OpenSearch 2.0 in gradle dependencies in place of the snapshot version.
• Add OpenSearch 2.0 to the integration test matrix.
• Bump the client version to 2.0.0.
• Cut a 2.0 branch for the client.

[FEATURE] Ensure compatibility with JDK 8, 14, 15, 17

Is your feature request related to a problem?
It's unclear what versions of JDK is this client compatible with. Tests seem to run on JDK14.

What solution would you like?
Include JDKs 8, 14, 15 and 17 to the test matrix and document what can be used.

[BUG] can't sort on field when document of that field contains null. fixed in elasticsearch-java.

What is the bug?
A full convo can be found here regarding the bug:
https://discuss.elastic.co/t/failure-with-sorts-and-new-co-elastic-clients-elasticsearch-java-client/291511/12

tl;dr: Sorting on a field where documents contain null breaks.

What is the expected behavior?
I am able to sort on a field where some documents contain NULL in those fields.

Elastic Searches Fix
https://github.com/elastic/elasticsearch-java/pull/68/files

[BUG] testClusterGetSettingsWithDefault fails intermittently

What is the bug?

See https://github.com/opensearch-project/opensearch-java/runs/5645720812?check_suite_focus=true, tests failed on main against 1.3.0.

What is the expected behavior?
Tests pass.

[FEATURE] Be able to set total number of fields on index

Is your feature request related to a problem?

I cannot using the java client set index.mappings.total_fields.limit on an IndexSettings request, and have to do so manually since my project uses the java client instead of the REST client.

What solution would you like?

I would like to be able to set the total fields limit for an index, same as number of shards, on the IndexSettings object for an IndexTemplateMapping.

What alternatives have you considered?

Using the REST client in addition to the java client, but I really don't want to have to use both clients in my project just so I can set the total fields on an index mapping update.

Update to Gradle 7

Is your feature request related to a problem? Please describe.
The OpenSearch Java Client is still built using Gradle 6.6.x whereas the Gradle release train has moved to 7.x already. The OpenSearch core is about to be switched to Gradle 7.3 (see please opensearch-project/OpenSearch#1609), it would make sense to switch the build to Gradle 7.3 as well.

Describe the solution you'd like
Update Gradle to 7.3

Describe alternatives you've considered (Optional)
N/A

Additional context
See please opensearch-project/OpenSearch#1246 and opensearch-project/opensearch-plugins#107

[PROPOSAL] Restore support for Java 8

What kind of business use case are you trying to solve? What are your requirements?

Many users still need Java 8 for their clients for maximum Java compatibility.

What is the problem? What is preventing you from meeting the requirements?

With the introduction of OpenSearch 2.0.0, the OpenSearch 2.0.0 build has updated to Java 11. This also includes the opensearch-rest-client and opensearch-rest-high-level-client clients.

Users of clients who wish to update their client version must now update to Java 11.

Additionally, the opensearch-java client relies on the opensearch-rest-client. Per opensearch-project/opensearch-clients#17, this is the client which is meant to have the most compatibility.

Here is an example of updating Data Prepper (which currently supports Java 8):

> No matching variant of org.opensearch.client:opensearch-rest-high-level-client:2.0.0-rc1 was found. The consumer was configured to find a runtime of a library compatible with Java 8, packaged as a jar, and its dependencies declared externally but:
    - Variant 'apiElements' capability org.opensearch.client:opensearch-rest-high-level-client:2.0.0-rc1 declares a library, packaged as a jar, and its dependencies declared externally:
        - Incompatible because this component declares an API of a component compatible with Java 11 and the consumer needed a runtime of a component compatible with Java 8
    - Variant 'runtimeElements' capability org.opensearch.client:opensearch-rest-high-level-client:2.0.0-rc1 declares a runtime of a library, packaged as a jar, and its dependencies declared externally:
        - Incompatible because this component declares a component compatible with Java 11 and the consumer needed a component compatible with Java 8

What are you proposing? What do you suggest we do to solve the problem or improve the existing situation?

I propose that the OpenSearch Core's opensearch-rest-client and opensearch-rest-high-level-client projects be made to use Java 8. The Gradle build can setup the sourceCompatibility and targetCompatibility to ensure that it uses only Java 8 language features.

An alternative solution would be to remove the dependency that opensearch-java has on opensearch-rest-client.

What are your assumptions or prerequisites?

None.

What are remaining open questions?

Can Gradle toolchains or another mechanism help ensure that the client projects do not use any Java 11 APIs?

[FEATURE] Add java client support for remote store APIs

What solution would you like?
As per the discussion: opensearch-project/OpenSearch#3576 (comment), this change will add support for remote store APIs

[BUG] NPE due to org.opensearch.client.json.LazyDeserializer

LazyDeserializer.unwrap() can return null if two threads attempt to use it at the same time.
This causes NPEs in classes that try to use it.

    JsonpDeserializer<T> d = deserializer;
    if (d == null) {                               //<-- if d is null
        synchronized (this) {
            if (deserializer == null) {     //<-- but another thread set this while we were waiting, then we don't go in here
                d = ctor.get();
                deserializer = d;
            }
        }
    }
    return d                                        //<-- and d is still null when we return it

Since it's a concurrency problem, and it only happens once per type, you need to be unlucky to experience this.

[BUG] IllegalStateException encountered when creating index with mappings

What is the bug?
IllegalStateException encountered when creating index with mappings

How can one reproduce the bug?

1. Run opensearch as a docker container on your local machine: docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" -d opensearchproject/opensearch:latest
2. Using the opensearch-java client version 0.1.0, write and run the following code.

public class OpenSearchTest {
    public static void main(String[] args) {
        final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
        credentialsProvider.setCredentials(
                AuthScope.ANY, new UsernamePasswordCredentials("admin", "admin"));


        RestClient restClient = RestClient.builder(new HttpHost("localhost", 9200, "https")).
                setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
                    @Override
                    public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
                        return httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
                    }
                }).build();
        Transport transport = new RestClientTransport(restClient, new JacksonJsonpMapper());
        OpenSearchClient client = new OpenSearchClient(transport);

        //Create the index
        String index = "sample-index";

        JsonObject mappings = Json.createObjectBuilder()
                .add("_source", Json.createObjectBuilder().add("_enabled", true))
                .add("properties", Json.createObjectBuilder()
                        .add("example_field", Json.createObjectBuilder()
                                .add("type", "boolean")
                                .add("store", true)
        )).build();

        CreateRequest createIndexRequest = new CreateRequest.Builder().index(index).mappings(mappings).build();

        try {
            client.indices().create(createIndexRequest);
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }
}

You should see the following error at the top of the stack trace:
java.lang.IllegalStateException: No ObjectCodec defined for the generator, can only serialize simple wrapper types (type passed org.glassfish.json.JsonObjectBuilderImpl$JsonObjectImpl)

What is the expected behavior?
Json object is successfully serialized and passed as a mapping to the opensearch service

What is your host/environment?

• OS: [MacOS
• Version 11.6
• Plugins

[FEATURE] Add support for inclusive naming

Add support for inclusive naming by deprecating the master nomenclature.

See meta issue opensearch-project/opensearch-clients#16 for details.

[BUG] Bad request response when calling the PutMapping api

What is the bug?
After calling the index api to make a put mapping request the following response is received:

org.opensearch.client.ResponseException: method [PUT], host [localhost:9200], URI [/memory/_mapping], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters:  [source_field : {enabled=true}]"}],"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters:  [source_field : {enabled=true}]"},"status":400}

How can one reproduce the bug?
Steps to reproduce the behavior:
After creating an opensearch client, create a 'PutMappingRequest object and fill it out as below:

      PutMappingRequest.Builder mappingsRequestBuilder = new PutMappingRequest.Builder().index(indexName);
        mappingsRequestBuilder = mappingsRequestBuilder.sourceField(
                new SourceField.Builder()
                    .enabled(mappingsJson.getJsonObject("_source").getBoolean("enabled"))
                    .build());
        mappingsRequestBuilder = mappingsRequestBuilder.routingField(
                new RoutingField.Builder()
                    .required(mappingsJson.getJsonObject("_routing").getBoolean("required"))
                    .build());

      this.client.indices().putMapping(mappingsRequestBuilder.build());

What is the expected behavior?
Successful update of the mapping metadata

What is your host/environment?

• OS:MacOS
• Version 11.6

Do you have any screenshots?
n/a

Do you have any additional context?
I believe this error occurs because the fields are incorrect in the serialization of the PutMappingRequest. For example on the 'source' field the key is 'source_field' when it should be 'source'

e.g. https://github.com/opensearch-project/opensearch-java/blob/main/java-client/src/main/java/org/opensearch/client/opensearch/indices/PutMappingRequest.java#L467

I believe this is the case of many fields in that object.

Thanks

Milestone 1 tasks

In this milestone, we focus on setting up main branch that can work with existing 7.10.2 version of ODFE

• Delete all older branches < 7.10
• Delete all released tags
• Delete main branch(this branch potentially tracks of all 8.x features as well)
• create main branch from 7.x branch
• Remove product checker commit if present (this is required to make clients work with oss cluster)
• Copy template files related to OpenSearch guidelines.
• set up CI and ensure tests pass for Elasticsearch OSS Docker image. This means all the test cases related to oss features from 7.10.2 executes successfully.

[FEATURE] Publish snapshot build

Is your feature request related to a problem?
opensearch-java-0.2-snapshot should be available in
https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/client/

What solution would you like?
New development should be readily available for consumers

What alternatives have you considered?
git clone repo from main, build and publish to maven local. Later use that in dependency.

Do you have any additional context?
Similar to OpenSearch

[BUG] 0.2.1 branch breaks completion suggesters

What is the bug?

The required fields for completion suggesters are incorrect. Some fields with default values are marked as required fields. In completion suggester, this includes transpositions and unicodeAware and therefore breaks code that doesn't set these.

Also, the response objects for completion suggestion do not work. Completion suggestion results are coming back as isTerm and therefore blowing up since they do not have score or some other values in the correct places, and all of their other fields are therefore missing.

@imRishN

Breaking changes for 2.0

Get started with incorporating the breaking changes for 2.0 from OpenSearch 2.0.

To do so:

1. Branch off a 1.x from main if not already done so that development for 2.0 can start on main.
2. Incorporate breaking changes as mentioned in opensearch-project/opensearch-clients#12.
3. Add integ tests support to run against unreleased OpenSearch.

Example PR from java-client: #132

[FEATURE] Add IAM authentication option / support for SigV4 Signer

If people want to connect to an Amazon OpenSearch Service cluster, they have to figure out how to use this client alongside some third-party signing library and/or the AWS SDK, which is non-trivial due to how we (by design) conceal the underlying HTTP requests that the client makes. We should offer IAM signing as an option, just like the OpenSearch CLI does.

Is your feature request related to a problem?
If you want to use this client with an Amazon OpenSearch Service cluster that has IAM authentication rather than basic authentication, good luck.

What solution would you like?

When initializing the client, accept an AWS IAM credential provider and sign requests automatically.

What alternatives have you considered?

An additional Java signing library for use on top of the client. But given that it would only work with the client, it seems better and easier to just add it to the client.

Do you have any additional context?

opensearch-project/OpenSearch#1400

[BUG] Compatibility of 2.0 client with 1.x OpenSearch

What is the bug?

Coming from opensearch-project/opensearch-clients#17 and looking at 02a313f compatibility with 1.x client has been removed. Re-add it.

I expect that a 2.0 REST client should work against 1.x assuming you're not using any new features that exist only in 2.0.

What is the expected behavior?
opensearch-java 2.0 works against OpenSearch 1.x

[FEATURE] Test against latest SNAPSHOTs of OpenSearch v. Next

Is your feature request related to a problem?

In #122 we have added integration tests against released versions of OpenSearch. We'd like to catch bugs for the next version of OpenSearch earlier.

What solution would you like?
Add -SNAPSHOT/latest builds of OpenSearch to the matrix. As the time of writing this would be 1.3.1 and 2.0. Ignore failures (continue-on-error) so that these tests don't block PRs.

[BUG] Tag for v1.0.0 is missing and no GitHub Releases.

What is the bug?

There is no tag created as 1.0.0 released https://search.maven.org/artifact/org.opensearch.client/opensearch-java; this will be confusing and difficult for users to know that this client has been formally released.

How can one reproduce the bug?
Steps to reproduce the behavior:

• Go to the repo page and see there is no Releases on the right side.
• Go to the tag page and there is no tag cut for v1.0.0.

What is the expected behavior?
Both the tag and Release should be present.

Do you have any screenshots?
Tag:

Release: