Comments (19)
That's a new one for me. Do you happen to have a stacktrace? I'm curious where it gets thrown.
from lyra.
Sorry, should have included that to begin with:
java.net.NoRouteToHostException: No route to host
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618)
at com.rabbitmq.client.ConnectionFactory.createFrameHandler(ConnectionFactory.java:445)
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:504)
at net.jodah.lyra.internal.ConnectionHandler$3.call(ConnectionHandler.java:214)
at net.jodah.lyra.internal.ConnectionHandler$3.call(ConnectionHandler.java:205)
at net.jodah.lyra.internal.RetryableResource.callWithRetries(RetryableResource.java:46)
at net.jodah.lyra.internal.ConnectionHandler.createConnection(ConnectionHandler.java:205)
at net.jodah.lyra.internal.ConnectionHandler.createConnection(ConnectionHandler.java:179)
at net.jodah.lyra.internal.ConnectionHandler.(ConnectionHandler.java:61)
at net.jodah.lyra.Connections.create(Connections.java:65)
Right after that is my code calling Connections.create(options, retryConfig);, then the threads and web framework I'm using.
from lyra.
If the connection is done by hostname, and the host name resolves in DNS, that is thrown (assuming the host isn't up). If an IP is specified, and that host isn't available, the same stack trace happens with that.
If a hostname is provided that doesn't resolve in DNS, a DNS related exception is thrown, and that isn't retryable, which makes sense to me.
from lyra.
I tried reproducing this on my machine (running OSX) and the only way I could get a NoRouteToHostException was by adding a route (route add ...
) that rejects packets. I would think this is more of a fatal condition - something that retrying wouldn't help. I'm surprised you're hitting this just by having a machine down. Maybe it's OS specific? Any idea?
from lyra.
Turns out it appears to be OS specific. I tried replicating on Windows and was not able to. I haven't had a chance to try on RedHat, but I'm getting the error on Ubuntu 13.10. So it appears that this is a Linux problem.
from lyra.
If it's just an Ubuntu thing (perhaps it's a specific JRE version?) I'd lean towards not making this exception retryable. For any failure that isn't likely to be resolved while retrying, we should just throw right away.
Another option I've though of is to make the set of failures that are retryable a configurable thing. Still, having that be OS specific is not ideal.
from lyra.
Closing for now unless this exception proves to be something broader or something that should be recoverable.
from lyra.
Vary sad, because it is often reproducible when servers with rabbit just restarted on amazon ec2.
from lyra.
@jhalterman while NoRouteToHostException
is clearly an infrastructure-level issue, I wonder if it's worth making it re-triable, possibly as an option. It is indeed correct that on AWS, when instances are restarted or autoscaling group adds new ones, routing can be temporarily interrupted. AWS is an 800 pound gorilla in the infra room, so may be worth making an exception for.
@sergeyleyko what do you use for the hostname parameter, private IPs or private DNS? Public DNS?
from lyra.
@michaelklishin we use just custom DNS name assigned to EC2 instance. (not Public DNS generated by AWS).
@jhalterman by the way, I cannot find the way to add this No route to host Exception to retryableExceptions list in config.
from lyra.
@sergeyleyko so your DNS is not managed by AWS in any way?
from lyra.
@michaelklishin Yes, it is managed by 3rd party service. Just pointed to ec2 instance
from lyra.
@sergeyleyko then this is not AWS-specific. Still, I think it's reasonable to expect NoRouteToHostException
to be re-triable in my opinion.
from lyra.
@michaelklishin you are right. by the way, to add exception just use config.getRecoverableExceptions().add(newExceptionToRecover)
from lyra.
@sergeyleyko If NoRouteToHost actually is recoverable, I'm fine with adding it to the default list. Curious: how long does it usually take to recover after a restart and a NoReouteToHostException? How long before the DNS issues are resolved?
from lyra.
@jhalterman I was wondering if adding NoRouteToHostException is going to be added to the default list (or is this something that we can edit?). We're getting this issue on a deployment where the RabbitMQ instance goes down briefly each night and right now we have to do a corresponding restart of our services using Lyra on ubuntu instances.
from lyra.
@numbat I'm super tied up on other projects at the moment, but two options:
- Add the exception via
config.getRecoverableExceptions().add(newExceptionToRecover)
- Send me a quick pull request to get this added permanently. I believe the place to add it is here: https://github.com/jhalterman/lyra/blob/master/src/main/java/net/jodah/lyra/config/Config.java#L29
from lyra.
Cheers will do, thanks!
from lyra.
Was addressed in 99ca312.
from lyra.
Related Issues (20)
- Reconnection after RabbitMQ shutdown is not successful HOT 5
- when java.net.UnknownHostException is thrown the recovery failed
- What about moving to the latest 'amqp-client' version?
- Forgotten copying of automaticRecovery in ConnectionOptions.copy() HOT 1
- Documentation on ConnectionFactory used in ConnectionOptions HOT 2
- support withUri() in ConnectionOptions HOT 5
- Moving to latest RabbitMQ Java Client 4.0 HOT 5
- Why to copy ConnectionFactory on each create HOT 7
- No communication after recovery HOT 4
- Consumers not getting messages after broker restart HOT 8
- Connection name is not passed HOT 8
- RabbitMQ flooded by new connections HOT 5
- Lyra recover mechanism can prevent from the JVM to shutdown HOT 6
- Get rid of static executor service HOT 2
- Rapid growth of connections when network conditions are unreliable HOT 3
- retryableExceptions configuration and implementation HOT 3
- Use a `Predicate` for retry exception strategy
- Getting Rabbitmq HeartBeat Exception HOT 3
- Lyra is no longer under active development. HOT 1
- Connection recovery fails and gives up even though recovery policy is recoverAlways() HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lyra.