Elasticsearch 8 support about pelias

[missinglink]

Apologies for the delay on this, I have the ES8 upgrade work (merging/testing what Michael has done) on my to-do list but I've had a very busy start to the year.

Thank you for your patience, expect more movement in the following weeks.

[missinglink]

"I don't think there's a specific plan for 9 yet; that would probably also mean Lucene 10 under the hood which IMO doesn't have a concrete plan yet either." -some guy elastic employee on reddit

The linked post also mentions that 7 has been supported since Apr 2019 (almost 5 years at time of writing) so moving to 8 this year should avoid running on unsupported versions until ~2027.

[michaelkirk]

By way of an update, as of yesterday, the projects/north-america integration tests running on es8 are at parity with the master branch.

Aggregate test results
Pass: 233
Improvements: 18
Expected Failures: 28
Placeholders: 0
Regressions: 84
Total tests: 363
Took 9351ms
Test success rate 76.86%

full output of cd docker/projects/north-america && pelias test run :

• output on master
• output on mkirk/elastic8

These PR's are ready for review/merging:

• pelias/config#142
• pelias/docker#317

[missinglink]

Hi all, I've just merged pelias/config#142 which drops support for elasticsearch v6 so that work can continue on supporting v8.

Version 7 remains the officially supported version.

see: https://github.com/pelias/config/releases/tag/v6.0.0

[michaelkirk]

Hah, I suppose p99 is a dumb metric when my buckets often have less than 100 points. Here's a new chart plotting median response time (now in millis) and also including the request count for context.

It still shows the es7 node was really slow. It might be under-provisioned, but I don't think I changed anything since deploying es8.

The deployment configuration is here if you're curious: https://github.com/headwaymaps/headway/blob/main/k8s/configs/planet/pelias-elasticsearch-deployment.yaml

Notably I got ES_JAVA_OPTS=-Xmx8g" from https://github.com/pelias/documentation/blob/master/full_planet_considerations.md

Anyway, I suppose my small measurements aren't super useful empirically, but I wanted to say that I'm using this, and it's working for me in case anyone else was waiting on a positive sign that this might be worth trying out.

[michaelkirk]

In case it wasn't clear, I have (I believe) a fully functioning pelias stack on top of elastic8. And I was hoping to feed those changes into the project in a graceful way (starting with the two PR's in my last comment).

Is supporting elastic8 something the project maintainers are interested in shepherding forward? Is there anything I can do to make that process easier?

[missinglink]

Hi @michaelkirk, yeah I think we'll definitely like to keep up with major releases at some point so thanks for all your work 🙇

Honestly I'm still feeling the pain of upgrading to 7 which IIRC was still easier than going from v2-5 😢

It always ends up being a huge amount of dev work, support & maintenance to get everyone upgraded.

Some fairly large companies are still using ancient versions from 2016 😲

The last couple upgrades were backwards compatible one version and that worked out reasonably well despite us having code paths which supports both versions.

In the past we've been motivated to upgrade by increased search performance or reduced disk use.

When looking through the changelog for 8 I didn't see any compelling reason to upgrade this time around.

Can I ask what your motivation is for this?

Maybe you'd like to be able to use the new vector search functionality or co-locate Pelias on an elastic server which also uses those newer features?

The reason I ask is that I personally feel like stability is more valuable than being on the latest version unless there is a compelling reason to upgrade.

[michaelkirk]

My own needs are pretty humble at this point. There's an annoying timeout bug (pelias/docker#217 (comment)) that doesn't occur with elastic 8.

Additionally, I wanted to dig into some performance issues I was experiencing, but was quickly demoralized when I realized the elasticsearch client library I was spending time studying is deprecated and not being maintained. It feels like a dead end.

So my thinking was pretty simply, let's get on firmer footing and then try to make forward progress. I appreciate that you have to support a variety of commercial customers and that any changes can jeopardize their experience.

The work I've done is still compatible with elastic7. Does that change the equation for you?

[missinglink]

The work I've done is still compatible with elastic7. Does that change the equation for you?

That's great, I am 👍 for dropping support for 6 so we can increase support for 8, ideally merging things in smaller manageable batches to avoid having to do it all at once.

[michaelkirk]

Thanks for the feedback @missinglink.

dropping support for 6 so we can increase support for 8, ideally merging things in smaller manageable batches to avoid having to do it all at once.

The non-draft PR's I open are done with this in mind. e.g. pelias/config#142 and pelias/docker#317 were done with the understanding that they could be released inependently at any time without causing trouble for existing users.

If there's an open PR that you feel is "too large" please let me know and I'll try to cleave it down.

P.S. one other thing I like about elastic8 is it seems to run fine on aarch64. =)

[cozydvlpr]

My current infrastructure offers OpenSearch 2.11. I'm patching here and there to check if I can make Pelias run on that. Following this issue. However, no Elasticsearch client will allow to connect to an up to date OpenSearch. Is it possible to make the dbclient lib more customizable to allow injecting a different client?

[alex-habrock-SHG]

@michaelkirk Appreciate you starting this thread. Looking at updating our ES deployment to 8 and what to see what that would mean for Pelias. What is the current status?

[michaelkirk]

Hi @Alex-SHG - no big changes other than what's already been covered in this thread.

I'm still hoping the people managing the project will review and merge pelias/config#142, and then I'll continue cleaving bits off of my other es8 branches (mentioned in the issue description) in hopes of getting them upstreamed.

If you want to try pelias on es8 today, feel free to build the stack from those branches in this issues description.

[dawidl11]

Thank you for the topic and your information!

We are using Elasticsearch 7 on our Pelias instances and we are getting more and more questions about when we can move to version 8.
The background to this is probably the fear that we will remain on an ES version that is no longer supported (https://www.elastic.co/support/eol).
Is it possible to predict approximately when the switch to 8 will be possible? Are the fears described above justified, or should we not worry here, and it will not happen that Pelias only supports outdated, unsupported ES versions?

[michaelkirk]

Is it possible to predict approximately when the switch to 8 will be possible?

I'm not a maintainer of this project - just a contributor, so I definitely couldn't predict, but if you're feeling bold, the issue description includes links to all the changes I found necessary to run pelias on es8. Note that I'm only using the /autocomplete endpoint, but I have also verified there were no new regression in the north_america and portland integration test suites which also exercise the /search endpoint.

The pelias project exists as a constellation of repositories, so there is a fair bit of sequencing that would need to happen to get all those changes merged without unintentionally breaking things for existing users. The next ones up, which are ready to review/merge, are:

• pelias/schema#490
• missinglink/elastictest#6

[dawidl11]

Thanks for the information!

[missinglink]

I landed a bunch of PRs today to make the importers (osm, oa, wof, etc..) work with es8.

I've tested the portland-metro project with both v7 and v8, it imports all the data and serves it without error 🎉

diff --git a/projects/portland-metro/docker-compose.yml b/projects/portland-metro/docker-compose.yml
index 98ae9da..50f69d6 100644
--- a/projects/portland-metro/docker-compose.yml
+++ b/projects/portland-metro/docker-compose.yml
@@ -102,7 +102,7 @@ services:
       - "./pelias.json:/code/pelias.json"
       - "${DATA_DIR}:/data"
   elasticsearch:
-    image: pelias/elasticsearch:7.16.1
+    image: pelias/elasticsearch:8.12.2-beta
     container_name: pelias_elasticsearch
     user: "${DOCKER_USER}"
     restart: always

[michaelkirk]

By way of an update, I've been happily using pelias on es8 for maps.earth (www and iOS) for about 10 days. No issues so far. In fact I have seen a nice decrease in response time.

I don't get very much traffic though — the whole range only covers 5k requests.

[missinglink]

I have seen a nice decrease in response time

Nice, thanks for sharing. I'm interested in what might be causing this the reduced latency as I didn't find anything online that would suggest v8 had many changes to the core search performance compared to v7. Maybe the new GC engine in the JVM upgrade? but likely not..

What's the scale of the Y-axis? Seconds or Milliseconds? If it's seconds then the machine is quite under-provisioned, making benchmarks difficult to extrapolate to larger clusters.

At some point we will do some split testing of ES7 vs ES8 under load & sample some performance data over say ~10MM requests across search/autocomplete/reverse and compare them between versions.

It can be a bit tricky to compare directly as new servers take time to 'settle' (ie. cache warming etc.) and some AWS servers are just worse than others, even in the same server class 🤷