ncolomer / elasticsearch-osmosis-plugin Goto Github PK
View Code? Open in Web Editor NEWAn Osmosis plugin that index OpenStreetMap data into elasticsearch
License: Apache License 2.0
An Osmosis plugin that index OpenStreetMap data into elasticsearch
License: Apache License 2.0
Hi,
When trying to import an osm file (single ES-Node-seems fine) I got this exception - (System: Ubuntu Server 14.04 LTS / 2 Cores - 7GB):
xxx:/usr/share/elasticsearch# osmosis --read-pbf /osm/extract/guyane-latest.osm.pbf --write-elasticsearch cluster.hosts="localhost"
Jul 04, 2016 7:38:43 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Osmosis Version 0.45
Jul 04, 2016 7:38:43 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Preparing pipeline.
Jul 04, 2016 7:38:43 AM org.openstreetmap.osmosis.plugin.elasticsearch.client.ElasticsearchClientBuilder buildTransportClient
INFO: Connecting to elasticsearch cluster 'elasticsearch' via [localhost] using TransportClient
07:38:43.664 [main] INFO org.elasticsearch.plugins - [Skin] loaded [], sites []
07:38:43.694 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [force_merge], type [fixed], size [1], queue_size [null]
07:38:43.701 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [percolate], type [fixed], size [2], queue_size [1k]
07:38:43.712 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [fetch_shard_started], type [scaling], min [1], size [4], keep_alive [5m]
07:38:43.713 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [listener], type [fixed], size [1], queue_size [null]
07:38:43.714 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [index], type [fixed], size [2], queue_size [200]
07:38:43.714 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [refresh], type [scaling], min [1], size [1], keep_alive [5m]
07:38:43.714 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [suggest], type [fixed], size [2], queue_size [1k]
07:38:43.714 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [generic], type [cached], keep_alive [30s]
07:38:43.715 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [warmer], type [scaling], min [1], size [1], keep_alive [5m]
07:38:43.716 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [search], type [fixed], size [4], queue_size [1k]
07:38:43.716 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [flush], type [scaling], min [1], size [1], keep_alive [5m]
07:38:43.716 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [fetch_shard_store], type [scaling], min [1], size [4], keep_alive [5m]
07:38:43.716 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [management], type [scaling], min [1], size [5], keep_alive [5m]
07:38:43.717 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [get], type [fixed], size [2], queue_size [1k]
07:38:43.717 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [bulk], type [fixed], size [2], queue_size [50]
07:38:43.717 [main] DEBUG org.elasticsearch.threadpool - [Skin] creating thread_pool [snapshot], type [scaling], min [1], size [1], keep_alive [5m]
07:38:44.086 [main] DEBUG org.elasticsearch.common.network - configuration:
lo
inet 127.0.0.1 netmask:255.0.0.0 scope:host
inet6 ::1 prefixlen:128 scope:host
UP LOOPBACK mtu:65536 index:1
eth0
inet 10.0.0.4 netmask:255.255.255.0 broadcast:10.0.0.255 scope:site
inet6 fe80::20d:3aff:fe22:ed45 prefixlen:64 scope:link
hardware 00:0D:3A:22:ED:45
UP MULTICAST mtu:1500 index:2
07:38:44.114 [main] DEBUG org.elasticsearch.common.netty - using gathering [true]
07:38:44.129 [main] DEBUG org.elasticsearch.client.transport - [Skin] node_sampler_interval[5s]
07:38:44.144 [main] DEBUG o.e.n.c.socket.nio.SelectorUtil - Using select timeout of 500
07:38:44.144 [main] DEBUG o.e.n.c.socket.nio.SelectorUtil - Epoll-bug workaround enabled = false
07:38:44.176 [main] DEBUG org.elasticsearch.client.transport - [Skin] adding address [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}]
07:38:44.276 [elasticsearch[Skin][management][T#1]] DEBUG org.elasticsearch.transport.netty - [Skin] connected to node [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}]
07:38:44.388 [elasticsearch[Skin][transport_client_worker][T#1]{New I/O worker #1}] INFO org.elasticsearch.client.transport - [Skin] failed to get local cluster state for {#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}, disconnecting...
org.elasticsearch.transport.RemoteTransportException: [Failed to deserialize response of type [org.elasticsearch.action.admin.cluster.state.ClusterStateResponse]]
Caused by: org.elasticsearch.transport.TransportSerializationException: Failed to deserialize response of type [org.elasticsearch.action.admin.cluster.state.ClusterStateResponse]
at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:179) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:138) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: java.lang.ExceptionInInitializerError: null
at org.elasticsearch.Version.fromId(Version.java:508) ~[elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.elasticsearch.Version.readVersion(Version.java:280) ~[elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.elasticsearch.cluster.node.DiscoveryNode.readFrom(DiscoveryNode.java:327) ~[elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.elasticsearch.cluster.node.DiscoveryNode.readNode(DiscoveryNode.java:310) ~[elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.elasticsearch.cluster.node.DiscoveryNodes.readFrom(DiscoveryNodes.java:594) ~[elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.elasticsearch.cluster.node.DiscoveryNodes$Builder.readFrom(DiscoveryNodes.java:674) ~[elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.elasticsearch.cluster.ClusterState.readFrom(ClusterState.java:699) ~[elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.elasticsearch.cluster.ClusterState$Builder.readFrom(ClusterState.java:677) ~[elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.elasticsearch.action.admin.cluster.state.ClusterStateResponse.readFrom(ClusterStateResponse.java:58) ~[elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:177) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
... 23 common frames omitted
Caused by: java.lang.IllegalArgumentException: An SPI class of type org.apache.lucene.codecs.PostingsFormat with name 'Lucene50' does not exist. You need to add the corresponding JAR file supporting this SPI to your classpath. The current classpath supports the following names: [es090, completion090, XBloomFilter]
at org.apache.lucene.util.NamedSPILoader.lookup(NamedSPILoader.java:109) ~[elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.apache.lucene.codecs.PostingsFormat.forName(PostingsFormat.java:112) ~[elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.elasticsearch.common.lucene.Lucene.(Lucene.java:68) ~[elasticsearch-osmosis-plugin-2.1.0.jar:na]
... 33 common frames omitted
07:38:44.390 [elasticsearch[Skin][transport_client_worker][T#1]{New I/O worker #1}] DEBUG org.elasticsearch.transport.netty - [Skin] disconnecting from [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}] due to explicit disconnect call
07:38:44.392 [elasticsearch[Skin][transport_client_worker][T#1]{New I/O worker #1}] WARN org.elasticsearch.transport.netty - [Skin] exception caught on transport layer [[id: 0xc470f506, /127.0.0.1:37351 :> localhost/127.0.0.1:9300]], closing connection
java.lang.IllegalStateException: Message not fully read (response) for requestId [0], handler [org.elasticsearch.client.transport.TransportClientNodesService$SniffNodesSampler$1$1@7af7b083], error [false]; resetting
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:145) ~[elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [elasticsearch-osmosis-plugin-2.1.0.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Jul 04, 2016 7:38:44 AM org.openstreetmap.osmosis.core.Osmosis main
SEVERE: Execution aborted.
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}]]
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:290)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:207)
at org.elasticsearch.client.transport.support.TransportProxyClient.execute(TransportProxyClient.java:55)
at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:283)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:347)
at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:841)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:85)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:59)
at org.openstreetmap.osmosis.plugin.elasticsearch.client.ElasticsearchClientBuilder.checkConnection(ElasticsearchClientBuilder.java:116)
at org.openstreetmap.osmosis.plugin.elasticsearch.client.ElasticsearchClientBuilder.build(ElasticsearchClientBuilder.java:60)
at org.openstreetmap.osmosis.plugin.elasticsearch.ElasticSearchWriterFactory.buildElasticsearchClient(ElasticSearchWriterFactory.java:83)
at org.openstreetmap.osmosis.plugin.elasticsearch.ElasticSearchWriterFactory.createTaskManagerImpl(ElasticSearchWriterFactory.java:29)
at org.openstreetmap.osmosis.core.pipeline.common.TaskManagerFactory.createTaskManager(TaskManagerFactory.java:60)
at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.buildTasks(Pipeline.java:50)
at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.prepare(Pipeline.java:112)
at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:86)
at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
at org.codehaus.classworlds.Launcher.main(Launcher.java:47)
I installed the osmosis and the plugin per instructions,
but it looks like ElasticSearch version 5.x.x is not supported?
ElasticSearch cluster Is up:
curl http://localhost:9200/_cluster/health\?pretty\=true
{
"cluster_name" : "elasticsearch",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 16,
"active_shards" : 16,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 16,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 50.0
}
Osmosis error:
11:59:51.842 [main] DEBUG org.elasticsearch.client.transport - [Tag] adding address [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}]
11:59:52.016 [elasticsearch[Tag][management][T#1]] DEBUG org.elasticsearch.transport.netty - [Tag] connected to node [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}]
11:59:52.037 [elasticsearch[Tag][generic][T#1]] DEBUG org.elasticsearch.transport.netty - [Tag] disconnecting from [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}], channel closed event
11:59:52.104 [elasticsearch[Tag][generic][T#2]] INFO org.elasticsearch.client.transport - [Tag] failed to get local cluster state for {#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}, disconnecting...
org.elasticsearch.transport.NodeDisconnectedException: [][localhost/127.0.0.1:9300][cluster:monitor/state] disconnected
May 21, 2017 11:59:52 AM org.openstreetmap.osmosis.core.Osmosis main
SEVERE: Execution aborted.
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}]]
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:290)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:207)
at org.elasticsearch.client.transport.support.TransportProxyClient.execute(TransportProxyClient.java:55)
at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:283)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:347)
at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:841)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:85)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:59)
at org.openstreetmap.osmosis.plugin.elasticsearch.client.ElasticsearchClientBuilder.checkConnection(ElasticsearchClientBuilder.java:116)
at org.openstreetmap.osmosis.plugin.elasticsearch.client.ElasticsearchClientBuilder.build(ElasticsearchClientBuilder.java:60)
at org.openstreetmap.osmosis.plugin.elasticsearch.ElasticSearchWriterFactory.buildElasticsearchClient(ElasticSearchWriterFactory.java:83)
at org.openstreetmap.osmosis.plugin.elasticsearch.ElasticSearchWriterFactory.createTaskManagerImpl(ElasticSearchWriterFactory.java:29)
at org.openstreetmap.osmosis.core.pipeline.common.TaskManagerFactory.createTaskManager(TaskManagerFactory.java:60)
at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.buildTasks(Pipeline.java:50)
at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.prepare(Pipeline.java:112)
at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:86)
at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:330)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:238)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
at org.codehaus.classworlds.Launcher.main(Launcher.java:47)
ElasticSearch Error:
[2017-05-19T16:07:01,871][INFO ][o.e.c.r.a.DiskThresholdMonitor] [FvT7WdR] rerouting shards: [high disk watermark exceeded on one or more nodes]
[2017-05-19T16:07:06,969][WARN ][o.e.t.n.Netty4Transport ] [FvT7WdR] exception caught on transport layer [[id: 0x240b7a45, L:/172.18.0.3:9300 - R:/172.18.0.1:41884]], closing connection
java.lang.IllegalStateException: Received message from unsupported version: [2.0.0] minimal compatible version is: [5.0.0]
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1354) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74) ~[transport-netty4-5.4.0.jar:5.4.0]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) [netty-codec-4.1.9.Final.jar:4.1.9.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:280) [netty-codec-4.1.9.Final.jar:4.1.9.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:396) [netty-codec-4.1.9.Final.jar:4.1.9.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248) [netty-codec-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:624) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:524) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:478) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:438) [netty-transport-4.1.9.Final.jar:4.1.9.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.9.Final.jar:4.1.9.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
We just exported with this plugin the europe file. Compressed about 15 gb. Exported its about 649 gb.
It seems a little bit steep. How much would be the whole planet osm pbf file exported?
Ways are referencing to node ids but it would be of more use to
this enables searching for positions of street.
Somehow response.getField stopped working, causing a lot of tests to fail.
Map<String, Object> shape = (Map<String, Object>) response.getField("shape").getValue();
It seems getSource is a good replacer
Map<String, Object> shape = (Map<String, Object>) response.getSource().get("shape");
Then recompile with elasticsearch 0.90.10 and it work fine with last release, but not work with beta line 1.0.0.Beta1, 1.0.0.Beta2 of elasticsearch.
During compilation with 1.0.0.Beta1, 1.0.0.Beta2 error:
[ERROR] /root/osmosis/src/main/java/org/openstreetmap/osmosis/plugin/elasticsearch/model/shape/ESShape.java:[7,35] cannot find symbol
symbol : class GeoShapeConstants
location: package org.elasticsearch.common.geo
[ERROR] /root/osmosis/src/main/java/org/openstreetmap/osmosis/plugin/elasticsearch/model/shape/ESShape.java:[203,46] package GeoShapeConstants does not exist
Hello!
I'm following your Getting started manual and I get exception when I start to use osmosis:
osmosis --read-pbf ~/osm/extract/guyane-latest.osm.pbf
--write-elasticsearch cluster.hosts="localhost:9300"
INFO: [Zodiak] failed to get local cluster state for [#transport#-1][inet[localhost/127.0.0.1:9300]], disconnecting...
org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[localhost/127.0.0.1:9300]][cluster/state] request_id [0] timed out after [5003ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:342)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Jun 17, 2015 2:14:49 PM org.openstreetmap.osmosis.core.Osmosis main
SEVERE: Execution aborted.
org.elasticsearch.client.transport.NoNodeAvailableException: No node available
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:202)
at org.elasticsearch.client.transport.support.InternalTransportClusterAdminClient.execute(InternalTransportClusterAdminClient.java:85)
at org.elasticsearch.client.support.AbstractClusterAdminClient.health(AbstractClusterAdminClient.java:83)
at org.elasticsearch.action.admin.cluster.health.ClusterHealthRequestBuilder.doExecute(ClusterHealthRequestBuilder.java:93)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:62)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:57)
at org.openstreetmap.osmosis.plugin.elasticsearch.client.ElasticsearchClientBuilder.checkConnection(ElasticsearchClientBuilder.java:110)
at org.openstreetmap.osmosis.plugin.elasticsearch.client.ElasticsearchClientBuilder.build(ElasticsearchClientBuilder.java:59)
at org.openstreetmap.osmosis.plugin.elasticsearch.ElasticSearchWriterFactory.buildElasticsearchClient(ElasticSearchWriterFactory.java:83)
at org.openstreetmap.osmosis.plugin.elasticsearch.ElasticSearchWriterFactory.createTaskManagerImpl(ElasticSearchWriterFactory.java:29)
at org.openstreetmap.osmosis.core.pipeline.common.TaskManagerFactory.createTaskManager(TaskManagerFactory.java:60)
at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.buildTasks(Pipeline.java:50)
at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.prepare(Pipeline.java:112)
at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:86)
at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:329)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:239)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
at org.codehaus.classworlds.Launcher.main(Launcher.java:47)
Any ideas what's wrong?
This specialized index aims to extend the main Way indice by building a new one with the following features:
This indice should be placed in the main index to allow geo query for both nodes and ways.
I believe it's not currently possible to pass parameters such as the number of shards / replica for the created index ( http://www.elasticsearch.org/guide/reference/api/admin-indices-create-index.html ).
Changing this parameter can sometimes helps with performance.
Thanks
Hello,
why are not the data analyzed (especially property "name") in ES index?
Thanks
Probably this was caused by something else, but I managed to stop the runtime error here by replacing
if (!response.isExists()) continue;
With
if (response.isSourceEmpty()) continue;
Should be replaced with refresh()
We can use the ElasticSearch Bulk Java API to buffer requests and send them by batch. This could improve insertion performance by reducing the total number of opened HTTP connection.
See http://www.elasticsearch.org/guide/reference/java-api/bulk.html for code samples
Typo in the method name :
Since elasticsearch 1.x does not return shape fields from the index, only from source, the current EntityDao cannot build a Way using the results of buildMultiGetRequest.
Choices are to fetch the entire source object or to store the geometry as a separate non-analyzed field, perhaps encoded as OGC WKT for compactness.
Due to the complex input structure and index representation of shapes, it is not currently possible to sort shapes or retrieve their fields directly. The geo_shape value is only retrievable through the _source field.
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-geo-shape-type.html
I'm on ES 1.0.0, and the plugin just keeps on failing with this:
INFO: [Angler] failed to get local cluster state for [#transport#-1][inet[/127.0.0.1:9300]], disconnecting...
Even though it's a single node cluster, it is up and working, and I tried passing in various options to no avail. I'm running it more or less like this (tried all kinds of options here):
osmosis --read-pbf my.osm.pbf --wes cluster.hosts=localhost:9300 cluster.name='elasticsearch_bstst'
Is this the ES version which is to blame here?
Only provide osmosis version in attributes. As build repository should not change in the future, the url can be builded from the needed version.
See builds @ http://bretth.dev.openstreetmap.org/osmosis-build/
This specialized index aims to index multi-level boundaries (country / state / province / city) in order to query for region memberships of a location.
This implies following tasks:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.