Comments (17)
Apologies if I am missing something here, please correct me if I am wrong.
I would encourage to not rely on CodecSettings anyhow (for this specific feature)
from opensearch.
@sarthakaggarwal97 Could the custom-codecs plugin itself create an register a static setting like codecs.experimental.enabled
and only create and add the new codec if that setting is explicitly set to true?
from opensearch.
@andrross Do you mean a static cluster setting? It would be like a feature flag right. Please correct me if I am wrong.
We might have to utilize this interface so that we can mark codecs as experimental. Another benefit of an interface change is that it gives a way for other plugins as well to use it. We have plugins like Security Analytics, K-nn who also currently maintain their own codecs, and might have similar use cases in the future.
I feel it will be a combination of both the setting and a clean way to mark the new codecs as experimental. We can then have assertions in place to block the create index call.
from opensearch.
@sarthakaggarwal97 Yeah it would functionally work the same way as a feature flag, but just be scoped to the custom-codecs plugin in this case. I think in order to determine exactly how we implement this feature we'll want to clearly define the user experience. If we extend the CodecSettings interface, how do we implement the behavior? What's a system admin's experience for enabling and disabling experimental codecs?
from opensearch.
@andrross since the validation to select the codec happens in EngineConfig, we might need to implement the feature flag within OpenSearch itself. CodecSettings interface would help us know whether the particular codec is experimental or not, while the feature flag would make experimental codecs available via CodecService.
Tagging @reta @mgodwan @backslasht for inputs.
from opensearch.
@andrross since the validation to select the codec happens in EngineConfig, we might need to implement the feature flag within OpenSearch itself. CodecSettings interface would help us know whether the particular codec is experimental or not, while the feature flag would make experimental codecs available via CodecService.
I would 💯 agree with @andrross that experimental codecs should be contained in custom-codecs
plugin, including the experimental support. The CodecSettings
is optional and we should not expect any codec to implement it (vs Apache Lucene's Codec
which is mandatory).
from opensearch.
I guess we should be able to have the setting in custom-codecs. We might have to still expose the experimental
via the CodecSettings interface. If we would not validate the experimental nature of codecs in the EngineConfig, and the codecs are not available in CodecService (because they are experimental), the shards would fail.
Apologies if I am missing something here, please correct me if I am wrong.
from opensearch.
If we would not validate the experimental nature of codecs in the EngineConfig, and the codecs are not available in CodecService (because they are experimental), the shards would fail.
@sarthakaggarwal97 Can you elaborate a bit on what you mean here? I get that there are sharp edges on the user experience (i.e. if you disable an experimental codec then any existing indexes using that codec will fail), but that is sort of expected for these opt-in experimental features until they are stabilized and fully supported. To me, the simplest option here is to just let custom-codecs choose whether to register experimental codecs at startup based on the existence of a static setting. What about that implementation doesn't work with the experience you have in mind?
from opensearch.
@andrross in EngineConfig when the validation happens upon index creation, it loads the codec from NamedSPI itself. The codecs are registered with NamedSPI through resources.
Is it possible to load a different resource file based on an experimental settings?
from opensearch.
Is it possible to load a different resource file based on an experimental settings?
It is not possible, the service loader mechanism does not support that (afaik)
from opensearch.
yeah, even I am not aware.
@andrross @reta what do you guys suggest in this case?
from opensearch.
To me, the most logical way to move forward is to contain the change in custom-codecs
(opensearch-project/custom-codecs#148): it provides own CodecService
(CustomCodecService
) and can filter out experimental codecs (driven by plugin specific settings).
from opensearch.
The challenge would be make the experimental codecs not available in the index.codec settings validations.
from opensearch.
The challenge would be make the experimental codecs not available in the index.codec settings validations.
Not sure I get it: If experimental codec is not enabled (in custom-codecs
), any attempts to configure it (including validating settings) would fail (since this is managed by our own CodecSettings
)
from opensearch.
If the codecs are registered by NamedSPI, they would be available over here
from opensearch.
If the codecs are registered by NamedSPI, they would be available over here
Same answer as for CodecSettings
, the CodecAliases
is ours and any attempts to use it (including validating settings) would fail. We cannot do this over any out of the box Apache Lucene codecs but we could do that for any codec in custom-codecs
plugin (some code changes may be need of cause)
from opensearch.
[Triage - attendees 1 2 3 4 5 6 7]
@sarthakaggarwal97 Thanks for creating this issue
from opensearch.
Related Issues (20)
- [BUG] Wazuh-indexer service warnings: Terminally Deprecated method has been called HOT 2
- Refactor FastFilterRewriteHelper
- [Profiling deep dive] Default aggregation vs. optimization code path
- [Feature Request] Fine grained control of (ingest|search) processors to install HOT 1
- [BUG] Potential Cluster Slowdown/Lags after merging #13748(#14348) #14338(#14391) in 2.15 HOT 12
- [BUG] flaky test Test derived_field supported type using search definition HOT 1
- [BUG] Batch async shard fetch holds up significant memory causing OOMs
- [PROPOSAL] Highlight REST API Changes in PRs HOT 2
- [Remote State] Timing and tracing for RemoteWritableEntityStore
- [Remote State] Optimize diff publication
- [AUTOCUT] Gradle Check Flaky Test Report for IndicesStoreIntegrationIT
- [Automation Enhancement] Mechanism to close the created Gradle Check AUTOCUT flaky test issues. HOT 3
- [AUTOCUT] Gradle Check Flaky Test Report for RemoteStoreRestoreIT
- [AUTOCUT] Gradle Check Flaky Test Report for AzureStorageServiceTests HOT 6
- [Refactor] Consider removing the `opensearch-dashboards` module from this repo
- [BUG] Inconsistent behavior on POST _aliases when two opposing actions are operating on the same index/alias pair HOT 1
- [AUTOCUT] Gradle Check Flaky Test Report for ICacheKeySerializerTests
- [AUTOCUT] Gradle Check Flaky Test Report for LangPainlessClientYamlTestSuiteIT
- [BUG] Option in NodesStatsRequest to return per-shard stats in response
- [AUTOCUT] Gradle Check Flaky Test Report for RecoveryWhileUnderLoadIT
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from opensearch.