ncolomer / elasticsearch-osmosis-plugin Goto Github PK

Hi,

When trying to import an osm file (single ES-Node-seems fine) I got this exception - (System: Ubuntu Server 14.04 LTS / 2 Cores - 7GB):

xxx:/usr/share/elasticsearch# osmosis --read-pbf /osm/extract/guyane-latest.osm.pbf --write-elasticsearch cluster.hosts="localhost"
Jul 04, 2016 7:38:43 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Osmosis Version 0.45
Jul 04, 2016 7:38:43 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Preparing pipeline.
Jul 04, 2016 7:38:43 AM org.openstreetmap.osmosis.plugin.elasticsearch.client.ElasticsearchClientBuilder buildTransportClient
INFO: Connecting to elasticsearch cluster 'elasticsearch' via [localhost] using TransportClient
07:38:43.664 [main] INFO org.elasticsearch.plugins - [Skin] loaded [], sites []
07:38:43.694 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [force_merge], type [fixed], size [1], queue_size [null]
07:38:43.701 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [percolate], type [fixed], size [2], queue_size [1k]
07:38:43.712 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [fetch_shard_started], type [scaling], min [1], size [4], keep_alive [5m]
07:38:43.713 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [listener], type [fixed], size [1], queue_size [null]
07:38:43.714 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [index], type [fixed], size [2], queue_size [200]
07:38:43.714 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [refresh], type [scaling], min [1], size [1], keep_alive [5m]
07:38:43.714 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [suggest], type [fixed], size [2], queue_size [1k]
07:38:43.714 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [generic], type [cached], keep_alive [30s]
07:38:43.715 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [warmer], type [scaling], min [1], size [1], keep_alive [5m]
07:38:43.716 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [search], type [fixed], size [4], queue_size [1k]
07:38:43.716 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [flush], type [scaling], min [1], size [1], keep_alive [5m]
07:38:43.716 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [fetch_shard_store], type [scaling], min [1], size [4], keep_alive [5m]
07:38:43.716 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [management], type [scaling], min [1], size [5], keep_alive [5m]
07:38:43.717 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [get], type [fixed], size [2], queue_size [1k]
07:38:43.717 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [bulk], type [fixed], size [2], queue_size [50]
07:38:43.717 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [snapshot], type [scaling], min [1], size [1], keep_alive [5m]
07:38:44.086 [main] DEBUG org.elasticsearch.common.network - configuration:

lo
inet 127.0.0.1 netmask:255.0.0.0 scope:host
inet6 ::1 prefixlen:128 scope:host
UP LOOPBACK mtu:65536 index:1

eth0
inet 10.0.0.4 netmask:255.255.255.0 broadcast:10.0.0.255 scope:site
inet6 fe80::20d:3aff:fe22:ed45 prefixlen:64 scope:link
hardware 00:0D:3A:22:ED:45
UP MULTICAST mtu:1500 index:2

07:38:44.114 [main] DEBUG org.elasticsearch.common.netty - using gathering [true]
07:38:44.129 [main] DEBUG org.elasticsearch.client.transport - [Skin] node_sampler_interval[5s]
07:38:44.144 [main] DEBUG o.e.n.c.socket.nio.SelectorUtil - Using select timeout of 500
07:38:44.144 [main] DEBUG o.e.n.c.socket.nio.SelectorUtil - Epoll-bug workaround enabled = false
07:38:44.176 [main] DEBUG org.elasticsearch.client.transport - [Skin] adding address [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}]
07:38:44.276 [elasticsearch[Skin][management][T#1]] DEBUG org.elasticsearch.transport.netty - [Skin] connected to node [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}]
07:38:44.388 [elasticsearch[Skin][transport_client_worker][T#1]{New I/O worker #1}] INFO org.elasticsearch.client.transport - [Skin] failed to get local cluster state for {#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}, disconnecting...
org.elasticsearch.transport.RemoteTransportException: [Failed to deserialize response of type [org.elasticsearch.action.admin.cluster.state.ClusterStateResponse]]
Caused by: org.elasticsearch.transport.TransportSerializationException: Failed to deserialize response of type [org.elasticsearch.action.admin.cluster.state.ClusterStateResponse]
at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:179) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:138) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: java.lang.ExceptionInInitializerError: null
at org.elasticsearch.Version.fromId(Version.java:508) ~[elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.elasticsearch.Version.readVersion(Version.java:280) ~[elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.elasticsearch.cluster.node.DiscoveryNode.readFrom(DiscoveryNode.java:327) ~[elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.elasticsearch.cluster.node.DiscoveryNode.readNode(DiscoveryNode.java:310) ~[elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.elasticsearch.cluster.node.DiscoveryNodes.readFrom(DiscoveryNodes.java:594) ~[elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.elasticsearch.cluster.node.DiscoveryNodes$Builder.readFrom(DiscoveryNodes.java:674) ~[elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.elasticsearch.cluster.ClusterState.readFrom(ClusterState.java:699) ~[elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.elasticsearch.cluster.ClusterState$Builder.readFrom(ClusterState.java:677) ~[elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.elasticsearch.action.admin.cluster.state.ClusterStateResponse.readFrom(ClusterStateResponse.java:58) ~[elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:177) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
... 23 common frames omitted
Caused by: java.lang.IllegalArgumentException: An SPI class of type org.apache.lucene.codecs.PostingsFormat with name 'Lucene50' does not exist. You need to add the corresponding JAR file supporting this SPI to your classpath. The current classpath supports the following names: [es090, completion090, XBloomFilter]
at org.apache.lucene.util.NamedSPILoader.lookup(NamedSPILoader.java:109) ~[elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.apache.lucene.codecs.PostingsFormat.forName(PostingsFormat.java:112) ~[elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.elasticsearch.common.lucene.Lucene.(Lucene.java:68) ~[elasticsearch-osmosis-plugin-2.1.0.jar:na]
... 33 common frames omitted
07:38:44.390 [elasticsearch[Skin][transport_client_worker][T#1]{New I/O worker #1}] DEBUG org.elasticsearch.transport.netty - [Skin] disconnecting from [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}] due to explicit disconnect call
07:38:44.392 [elasticsearch[Skin][transport_client_worker][T#1]{New I/O worker #1}] WARN org.elasticsearch.transport.netty - [Skin] exception caught on transport layer [[id: 0xc470f506, /127.0.0.1:37351 :> localhost/127.0.0.1:9300]], closing connection
java.lang.IllegalStateException: Message not fully read (response) for requestId [0], handler [org.elasticsearch.client.transport.TransportClientNodesService$SniffNodesSampler$1$1@7af7b083], error [false]; resetting
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:145) ~[elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Jul 04, 2016 7:38:44 AM org.openstreetmap.osmosis.core.Osmosis main
SEVERE: Execution aborted.
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}]]
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:290)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:207)
at org.elasticsearch.client.transport.support.TransportProxyClient.execute(TransportProxyClient.java:55)
at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:283)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:347)
at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:841)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:85)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:59)
at org.openstreetmap.osmosis.plugin.elasticsearch.client.ElasticsearchClientBuilder.checkConnection(ElasticsearchClientBuilder.java:116)
at org.openstreetmap.osmosis.plugin.elasticsearch.client.ElasticsearchClientBuilder.build(ElasticsearchClientBuilder.java:60)
at org.openstreetmap.osmosis.plugin.elasticsearch.ElasticSearchWriterFactory.buildElasticsearchClient(ElasticSearchWriterFactory.java:83)
at org.openstreetmap.osmosis.plugin.elasticsearch.ElasticSearchWriterFactory.createTaskManagerImpl(ElasticSearchWriterFactory.java:29)
at org.openstreetmap.osmosis.core.pipeline.common.TaskManagerFactory.createTaskManager(TaskManagerFactory.java:60)
at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.buildTasks(Pipeline.java:50)
at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.prepare(Pipeline.java:112)
at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:86)
at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
at org.codehaus.classworlds.Launcher.main(Launcher.java:47)

I installed the osmosis and the plugin per instructions,
but it looks like ElasticSearch version 5.x.x is not supported?

ElasticSearch cluster Is up:

curl http://localhost:9200/_cluster/health\?pretty\=true
{
"cluster_name" : "elasticsearch",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 16,
"active_shards" : 16,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 16,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 50.0
}

Osmosis error:

11:59:51.842 [main] DEBUG org.elasticsearch.client.transport - [Tag] adding address [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}]
11:59:52.016 [elasticsearch[Tag][management][T#1]] DEBUG org.elasticsearch.transport.netty - [Tag] connected to node [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}]
11:59:52.037 [elasticsearch[Tag][generic][T#1]] DEBUG org.elasticsearch.transport.netty - [Tag] disconnecting from [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}], channel closed event
11:59:52.104 [elasticsearch[Tag][generic][T#2]] INFO org.elasticsearch.client.transport - [Tag] failed to get local cluster state for {#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}, disconnecting...
org.elasticsearch.transport.NodeDisconnectedException: [][localhost/127.0.0.1:9300][cluster:monitor/state] disconnected
May 21, 2017 11:59:52 AM org.openstreetmap.osmosis.core.Osmosis main
SEVERE: Execution aborted.
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}]]
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:290)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:207)
at org.elasticsearch.client.transport.support.TransportProxyClient.execute(TransportProxyClient.java:55)
at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:283)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:347)
at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:841)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:85)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:59)
at org.openstreetmap.osmosis.plugin.elasticsearch.client.ElasticsearchClientBuilder.checkConnection(ElasticsearchClientBuilder.java:116)
at org.openstreetmap.osmosis.plugin.elasticsearch.client.ElasticsearchClientBuilder.build(ElasticsearchClientBuilder.java:60)
at org.openstreetmap.osmosis.plugin.elasticsearch.ElasticSearchWriterFactory.buildElasticsearchClient(ElasticSearchWriterFactory.java:83)
at org.openstreetmap.osmosis.plugin.elasticsearch.ElasticSearchWriterFactory.createTaskManagerImpl(ElasticSearchWriterFactory.java:29)
at org.openstreetmap.osmosis.core.pipeline.common.TaskManagerFactory.createTaskManager(TaskManagerFactory.java:60)
at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.buildTasks(Pipeline.java:50)
at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.prepare(Pipeline.java:112)
at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:86)
at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
at org.codehaus.classworlds.Launcher.main(Launcher.java:47)

ElasticSearch Error:
[2017-05-19T16:07:01,871][INFO ][o.e.c.r.a.DiskThresholdMonitor] [FvT7WdR] rerouting shards: [high disk watermark exceeded on one or more nodes]
[2017-05-19T16:07:06,969][WARN ][o.e.t.n.Netty4Transport ] [FvT7WdR] exception caught on transport layer [[id: 0x240b7a45, L:/172.18.0.3:9300 - R:/172.18.0.1:41884]], closing connection
java.lang.IllegalStateException: Received message from unsupported version: [2.0.0] minimal compatible version is: [5.0.0]
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1354) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74) ~[transport-netty4-5.4.0.jar:5.4.0]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) [netty-codec-4.1.9.Final.jar:4.1.9.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:280) [netty-codec-4.1.9.Final.jar:4.1.9.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:396) [netty-codec-4.1.9.Final.jar:4.1.9.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248) [netty-codec-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:624) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:524) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:478) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:438) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.9.Final.jar:4.1.9.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

We just exported with this plugin the europe file. Compressed about 15 gb. Exported its about 649 gb.
It seems a little bit steep. How much would be the whole planet osm pbf file exported?

Ways are referencing to node ids but it would be of more use to

1. fetch the node id its associated lat+lon from elasticsearch
2. store this information in the way document as geo_point (or multiple?)

this enables searching for positions of street.

Feature's type needs to be capitalized (and CamelCased) to be valid:

• point => Point
• linestring => LineString
• polygon => Polygon

Check GeoJSON examples from the official spec.

Somehow response.getField stopped working, causing a lot of tests to fail.

Map<String, Object> shape = (Map<String, Object>) response.getField("shape").getValue();

It seems getSource is a good replacer

Map<String, Object> shape = (Map<String, Object>) response.getSource().get("shape");

Then recompile with elasticsearch 0.90.10 and it work fine with last release, but not work with beta line 1.0.0.Beta1, 1.0.0.Beta2 of elasticsearch.
During compilation with 1.0.0.Beta1, 1.0.0.Beta2 error:
[ERROR] /root/osmosis/src/main/java/org/openstreetmap/osmosis/plugin/elasticsearch/model/shape/ESShape.java:[7,35] cannot find symbol
symbol : class GeoShapeConstants
location: package org.elasticsearch.common.geo
[ERROR] /root/osmosis/src/main/java/org/openstreetmap/osmosis/plugin/elasticsearch/model/shape/ESShape.java:[203,46] package GeoShapeConstants does not exist

Hello!
I'm following your Getting started manual and I get exception when I start to use osmosis:

osmosis --read-pbf ~/osm/extract/guyane-latest.osm.pbf
--write-elasticsearch cluster.hosts="localhost:9300"

INFO: [Zodiak] failed to get local cluster state for [#transport#-1][inet[localhost/127.0.0.1:9300]], disconnecting...
org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[localhost/127.0.0.1:9300]][cluster/state] request_id [0] timed out after [5003ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:342)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

Jun 17, 2015 2:14:49 PM org.openstreetmap.osmosis.core.Osmosis main
SEVERE: Execution aborted.
org.elasticsearch.client.transport.NoNodeAvailableException: No node available
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:202)
at org.elasticsearch.client.transport.support.InternalTransportClusterAdminClient.execute(InternalTransportClusterAdminClient.java:85)
at org.elasticsearch.client.support.AbstractClusterAdminClient.health(AbstractClusterAdminClient.java:83)
at org.elasticsearch.action.admin.cluster.health.ClusterHealthRequestBuilder.doExecute(ClusterHealthRequestBuilder.java:93)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:62)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:57)
at org.openstreetmap.osmosis.plugin.elasticsearch.client.ElasticsearchClientBuilder.checkConnection(ElasticsearchClientBuilder.java:110)
at org.openstreetmap.osmosis.plugin.elasticsearch.client.ElasticsearchClientBuilder.build(ElasticsearchClientBuilder.java:59)
at org.openstreetmap.osmosis.plugin.elasticsearch.ElasticSearchWriterFactory.buildElasticsearchClient(ElasticSearchWriterFactory.java:83)
at org.openstreetmap.osmosis.plugin.elasticsearch.ElasticSearchWriterFactory.createTaskManagerImpl(ElasticSearchWriterFactory.java:29)
at org.openstreetmap.osmosis.core.pipeline.common.TaskManagerFactory.createTaskManager(TaskManagerFactory.java:60)
at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.buildTasks(Pipeline.java:50)
at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.prepare(Pipeline.java:112)
at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:86)
at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:329)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:239)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
at org.codehaus.classworlds.Launcher.main(Launcher.java:47)

Any ideas what's wrong?

This specialized index aims to extend the main Way indice by building a new one with the following features:

• indexed locations instead of ids as a geo_point array. There is some limitations (see this issue and its related post) but this should nonetheless help find ways within a bounding box as for nodes.
• In addition, we could also compute quite easily the way's centroid prior to indexing using the spatial4j library.

This indice should be placed in the main index to allow geo query for both nodes and ways.

I believe it's not currently possible to pass parameters such as the number of shards / replica for the created index ( http://www.elasticsearch.org/guide/reference/api/admin-indices-create-index.html ).

Changing this parameter can sometimes helps with performance.

Thanks

Hello,

why are not the data analyzed (especially property "name") in ES index?

Thanks

Probably this was caused by something else, but I managed to stop the runtime error here by replacing

if (!response.isExists()) continue;

With

if (response.isSourceEmpty()) continue;

Should be replaced with refresh()

We can use the ElasticSearch Bulk Java API to buffer requests and send them by batch. This could improve insertion performance by reducing the total number of opened HTTP connection.

See http://www.elasticsearch.org/guide/reference/java-api/bulk.html for code samples

Typo in the method name :

Since elasticsearch 1.x does not return shape fields from the index, only from source, the current EntityDao cannot build a Way using the results of buildMultiGetRequest.

Choices are to fetch the entire source object or to store the geometry as a separate non-analyzed field, perhaps encoded as OGC WKT for compactness.

Due to the complex input structure and index representation of shapes, it is not currently possible to sort shapes or retrieve their fields directly. The geo_shape value is only retrievable through the _source field.

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-geo-shape-type.html

I'm on ES 1.0.0, and the plugin just keeps on failing with this:

INFO: [Angler] failed to get local cluster state for [#transport#-1][inet[/127.0.0.1:9300]], disconnecting...

Even though it's a single node cluster, it is up and working, and I tried passing in various options to no avail. I'm running it more or less like this (tried all kinds of options here):

osmosis --read-pbf my.osm.pbf --wes cluster.hosts=localhost:9300 cluster.name='elasticsearch_bstst'

Is this the ES version which is to blame here?

Only provide osmosis version in attributes. As build repository should not change in the future, the url can be builded from the needed version.

See builds @ http://bretth.dev.openstreetmap.org/osmosis-build/

This specialized index aims to index multi-level boundaries (country / state / province / city) in order to query for region memberships of a location.

This implies following tasks:

• implement the Relation entity in the EntityDao
• index Relations in the main index
• implement the Boundary specialized index builder (see the boundary=administrative wiki page)