Comments (3)
I removed ES 6 support from our tool, and will have to mention loudly in the release notes that someone needs to first upgrade from ES 6-7 before using the es-hadoop version that includes this issue, as it is no longer possible to make the migration path from 6-8 in the same binary.
Feel free to close this if you are ok with the impact
from elasticsearch-hadoop.
Can you provide more details on how to reproduce this? You are using es-hadoop 8.11.3 pointed at a 6.x elasticsearch with types? For some reason the es-hadoop client thinks that you are pointed at an 8.x elasticsearch (it discovers the version on startup). This is the logic that is supposed to handle it: https://github.com/elastic/elasticsearch-hadoop/blob/v8.11.3/mr/src/main/java/org/elasticsearch/hadoop/rest/Resource.java#L85.
Or are you trying to use an index name with types when reading from elasticsearch 8.x?
from elasticsearch-hadoop.
Maybe one sidenote for the upgrade story: Apache Lucene only writes the current major version (N) and can only read the previous major version (N-1) or current one (N). The Elasticsearch major versions also upgrade to a major Lucene version. So going from 6 to 8 will generally not work, since you'd need to be able to read N-2 — at least not without a reindex in which you could fix the _type.
It might not be perfect but many people will probably only keep tracing data for some time (let's say 30 to 90 days). If possible, I'd do a stepwise upgrade from 6 to 7 or within 7 to the new pattern and then 8 as the data ages out. Not perfect but maybe a reasonable tradeoff in upgrades?
from elasticsearch-hadoop.
Related Issues (20)
- Does elasticsearch-hadoop support IPv6 addresses HOT 1
- ELK to Databricks Data load HOT 1
- hello, is async http support? HOT 1
- Nested objects fail parsing in Spark SQL when empty objects present
- Latency Spike during Spark Structured Streaming HOT 2
- Upgrade to Spark 3.4.x HOT 3
- Unable to build the newly cloned project due to invalid dependency paths
- Spark dependency is not compatible resulting in compile error HOT 1
- 7.17.11 failed backwards compatibility for 5.5.3 HOT 4
- [Bug][CVE-2019-10172] found CVE in the latest release 8.12.2 and 8.9.1 HOT 1
- Es.write.operation documentation is deceptive on default values when used via spark HOT 1
- spark read long datatype error HOT 3
- ES-hadoop is not compatible with spark 3.5.1 HOT 7
- Stop using deprecated "stored" field in scripts for updates
- Automatic index creation fails with Elasticsearch >= 8.6.0 HOT 6
- Option es.read.field.*.include unable to take field names containing a colon
- Unable to index with elasticsearch-spark on serverless elasticsearch HOT 1
- What's the status on Elasticsearch 8.13 support? HOT 3
- Add support for creating suggester fields with weights
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from elasticsearch-hadoop.