Policy about type name in index name is too harsh about elasticsearch-hadoop

[codefromthecrypt]

I removed ES 6 support from our tool, and will have to mention loudly in the release notes that someone needs to first upgrade from ES 6-7 before using the es-hadoop version that includes this issue, as it is no longer possible to make the migration path from 6-8 in the same binary.

Feel free to close this if you are ok with the impact

[masseyke]

Can you provide more details on how to reproduce this? You are using es-hadoop 8.11.3 pointed at a 6.x elasticsearch with types? For some reason the es-hadoop client thinks that you are pointed at an 8.x elasticsearch (it discovers the version on startup). This is the logic that is supposed to handle it: https://github.com/elastic/elasticsearch-hadoop/blob/v8.11.3/mr/src/main/java/org/elasticsearch/hadoop/rest/Resource.java#L85.
Or are you trying to use an index name with types when reading from elasticsearch 8.x?

[xeraa]

Maybe one sidenote for the upgrade story: Apache Lucene only writes the current major version (N) and can only read the previous major version (N-1) or current one (N). The Elasticsearch major versions also upgrade to a major Lucene version. So going from 6 to 8 will generally not work, since you'd need to be able to read N-2 — at least not without a reindex in which you could fix the _type.

It might not be perfect but many people will probably only keep tracing data for some time (let's say 30 to 90 days). If possible, I'd do a stepwise upgrade from 6 to 7 or within 7 to the new pattern and then 8 as the data ages out. Not perfect but maybe a reasonable tradeoff in upgrades?