Comments (19)
is there any update on this? This problem really floods logs when running on GCE. setting the logger seems to provide an initial fix, but feels wrong
from grpc-go.
We are working on improving the logging system, #922 is the first step.
from grpc-go.
On Mon, Mar 16, 2015 at 6:38 PM, jellevandenhooff [email protected]
wrote:
I am running "go version go1.4.1 darwin/amd64". I accidentally pointed a
grpc client at an address that didn't speak grpc. Afterwards, grpc printed
a lot of messages in the log that were not helpful at best and distracting
at worst. I would prefer grpc to a) not generate as many errors, perhaps
with some back-off mechanism, and b) not print as many errors.Specifically, my terminal filled with hundreds of lines of the form
2015/03/16 21:08:01 transport: http2Client.notifyError got notified that
the client transport was broken unexpected EOF.
2015/03/16 21:08:01 transport: http2Client.notifyError got notified that
the client transport was broken unexpected EOF.
2015/03/16 21:08:01 transport: http2Client.notifyError got notified that
the client transport was broken unexpected EOF.
2015/03/16 21:08:01 transport: http2Client.notifyError got notified that
the client transport was broken unexpected EOF.
2015/03/16 21:08:01 transport: http2Client.notifyError got notified that
the client transport was broken unexpected EOF.I tried sticking in a "c.failFast = true" in grpc.Invoke, but that did not
help.I had got another user request which complained that there was no log
messages when the transport sees some error triggering reconnect. So I
added this log which seems spam you now. :) Let me see if there is a way I
can win in both worlds.—
Reply to this email directly or view it on GitHub
#120.
from grpc-go.
Ah -- I guess the messages themselves might by only a symptom of the underlying problem. I don't know if instantaneous reconnecting is desired behavior, but if it is not, then adding some back-off would probably also make me happy :)
Thanks!
from grpc-go.
Exponential back-off is already there (https://github.com/grpc/grpc-go/blob/master/clientconn.go#L164). You actually hit a different case: your connect actually succeeded because that port is listening but when you sent the first rpc it got rejected by the peer because the peer does not speak grpc. Therefore, you already hit the initial reconnect interval.
from grpc-go.
How do you feel about moving the exponential back-off into clientConn so it also backs-off if a server is crashing or misbehaving? I could try and see what it looks like if you like that idea.
from grpc-go.
I was thinking can we reconnect a little bit of more aggressive, I mean, I need to restart client mostly of the time when I restart server doesn't feels right. It killing my productivity.
Maybe reconnect if there is new rpc calls in client? Maybe just a little of aggressive, my server has no complain of that. What I want is thought of like zeromq or nanomsg, you don't need to care if server was going down, you just send messages.
from grpc-go.
Got the same high volume of error messages under the following scenario:
- Set up generic TCP listener which accepts and closes connections
- Send a request to server using the grpc hello world tutorial
It looks like there is an infinite loop in the Invoke function in /grpc/call.go
as follows:
- Invoke creates callInfo with
callInfo.failFast=false
at line 109 - If
sendRequest()
in line 169 fails withtransport.ConnectionError
a subsequentcontinue
jumps to loop start at line 143 - If
recvResponse()
in line 181 fails withtransport.ConnectionError
a subsequentcontinue
jumps to loop start at line 143 - Loop does not exit because
callInfo.failFast=false
blocks error handling at line 150
Setting callInfo.failFast=true
aborts the loop at line 143 and prevents the infinite loop.
In this case where the server explicitely closes the connection, because it does not recognise the grpc protocol, the client should not attempt retry.
There may be other transport.ConnectionError
cases for which limited retries are appropriate, but unbounded retries are not a good idea.
The following test code reproduces the above scenario
import (
"fmt"
"testing"
"net"
"golang.org/x/net/context"
"google.golang.org/grpc"
demo "google.golang.org/grpc/examples/helloworld/helloworld"
)
const hw_addr = "127.0.0.0:50000"
func TestGrpcReject(t *testing.T) {
// Create a simple server that just closes connections
lis, err := net.Listen("tcp", hw_port)
if err != nil { fmt.Printf("Listener not started: %v\n", err); return }
go func() {
for {
conn, _ := lis.Accept()
fmt.Printf("Server closing connection\n")
conn.Close()
}
}()
// Set up client and send request to server
conn, err := grpc.Dial(hw_address, grpc.WithInsecure())
client := demo.NewGreeterClient(conn)
client.SayHello(context.Background(), &demo.HelloRequest{Name: "World"})
}
from grpc-go.
I also felt this was extremely annoying in a dev environment (I have multiple services connecting to a non-critical grpc service and they would cumulatively spit out over 10k log lines in 5 seconds if I turned that service off), so I just replace the grpclog Logger with grpclog.SetLogger(log.New(ioutil.Discard, "", 0)) when in dev.
from grpc-go.
If you replace it with glogger (https://github.com/grpc/grpc-go/blob/master/grpclog/glogger/glogger.go), all the logs will go to some files instead of stderr unless you configure it explicitly.
from grpc-go.
I am experiencing same issue and using grpclog.SetLogger
. However, as I see it the problem is that there is no context about the error: if I am using different gRPC services I will never know by reading this log message which one is specifically the failing service.
Would it be possible to add to this log message more context e.g. the hostname or service name?
from grpc-go.
It would be very nice if the underlined backoff mechanism covers this case as well.
from grpc-go.
While I appreciate #922, the many log messages are a symptom of many failed connection attempts. I think it'd be worthwhile also reducing connection frequency, because even if we're not spewing logs, we're still hammering some poor unsuspecting service.
from grpc-go.
I was going to mention @jellevandenhooff's observation too, I thought there was some sort of exponential backoff involved when reconnecting.
from grpc-go.
Can you share more information about your program? Like what the error you got was and what server were you connecting to? What we want to know is the root cause of the connection error.
We have backoff mechanism if the connection can't be established at the first place.
One possible situation where the flood can happen is that the connection is established successfully, and disconnects immediately after that. We should do something (backoff or even stop retrying) depending on the reason.
from grpc-go.
The root cause of the error is exactly as you described, and a backoff as you propose is the solution. My PR from last year sketched out an approach for implementing that, but with all the changes to grpc, doesn't quite merge anymore.
from grpc-go.
The root cause of the error is exactly as you described
I assume by this you mean "connection is established successfully, and disconnects immediately after that."
Then what is the reason for this to happen in your case? (The server is a misbehavior server? or a non-gRPC server?)
from grpc-go.
@menghanl in my case the server did not have properly setup the certificate.
from grpc-go.
closing in favor of #954
from grpc-go.
Related Issues (20)
- Why does connection to `localhost` fail on local ? HOT 15
- transition from TRANSIENT_FAILURE to IDLE HOT 1
- Is waitForReady required for retry policies? HOT 7
- Add godoc linter HOT 5
- How can I configure a usable gRPC-Go Retry Policy? HOT 9
- failed to install [email protected] HOT 13
- design tradeoffs about not including ctx in method argument for generated stream handler HOT 3
- google.protobuf.Empty as response serializes to not empty response depending on request headers HOT 2
- How to make server-stream run in a non-blocking state and handle other requests HOT 10
- Improve Logger Performance When Disabled HOT 5
- Flaky Test: RingHash_ReattemptWhenGoingFromTransientFailureToIdle
- xdsclient: reduce the number of ADS requests
- grpctest: minor improvements to the test logger implementation
- xdsclient: support environment variable protection for fallback
- Incorporate `mem` package into custom HTTP/2 Framer
- Examples using the `experimental` stats package HOT 2
- about grpc.NewClient,It will not check the validity of the target ? HOT 1
- Minor typos HOT 7
- clientconn: Subconn state may transition from CONNECTING directly to IDLE HOT 1
- Flaky Test: Balancer_TwoAddresses_WeightExpiration HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from grpc-go.