Comments (10)
@satishviswanathan Do you get any sort of exception on etcd server restart ?
from dotnet-etcd.
@shubhamranjan i'm not seeing exception but seems like the application is losing the connection with etcd and subsequent watch notification is not received after restart.
from dotnet-etcd.
Noted. Etcd restarts haven't been handled yet for etcd watches are neither is there a provision for an exception handler. This are is something we need to improvise on.
@yoricksmeets Did you get a chance to work on the watch manager ?
from dotnet-etcd.
I connect to etcd cluster with dotnet-etcd version 3.2.0. the cluster was compose of three etcd nodes.
In beginning I have connected to the cluster succussfully and watch some datas changed.
But when I shutdown etcd nodes one by one, the applicaton thorow a unhandled exception as follows:
Grpc.Core.RpcException
HResult=0x80131500
Message=Status(StatusCode=Unknown, Detail="Stream removed")
Source=System.Private.CoreLib
StackTrace:
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Grpc.Core.Internal.ClientResponseStream`2.d__5.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at dotnet_etcd.EtcdClient.<>c__DisplayClass122_1.<b__0>d.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at dotnet_etcd.EtcdClient.d__122.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Threading.Tasks.Task.<>c.b__139_1(Object state)
at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
from dotnet-etcd.
@yanhua1012 Can you explain a bit as it happens ?. For e.g.
- 1 node goes down, 1 error occurs and subsequent work and so on.......
from dotnet-etcd.
step1. the etcd client connect to etcd cluster successfully and watch some datas
step2. shutdown the first node of cluster
step3. shutdown the second node of cluster
step4. start the second node of cluster
then one unhandled exception as follows happened (not immediately)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult()
at Grpc.Core.Internal.ClientResponseStream`2.d__5.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at dotnet_etcd.EtcdClient.<>c__DisplayClass122_1.<b__0>d.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at dotnet_etcd.EtcdClient.d__122.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Threading.Tasks.Task.<>c.b__139_1(Object state)
at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
from dotnet-etcd.
another case
step1. the etcd client connect to etcd cluster successfully and watch some datas
step2. shutdown the first node of cluster
step3. shutdown the second node of cluster
step4. shutdown the third node of cluster
then one unhandled exception as follows happened (not immediately)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at dotnet_etcd.EtcdClient.d__122.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Threading.Tasks.Task.<>c.b__139_1(Object state)
at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
from dotnet-etcd.
@shubhamranjan , Just reaching out to see if this scenario is being considered in future release as a feature.
from dotnet-etcd.
I'm using etcd client in my web api application and have implemented watch which works fine. However i'm facing the following issue.
Etcd connection gets disconnected
To Reproduce Steps to reproduce the behavior:
- During web api bootstrap initialize the etcd client and read the keys , registered a watch and loaded the IConfiguration with the keys.
- Change any value of the key in etcd
- We are able to get the call back with the updated value.
- Stop etcd and start it again.
- Change the value of any key
- Unable to receive the watch callback. It seems like with the etcd restart the connection was lost and has not reestablished the connection.
Expected behavior Application should be able to receive the callback notification with the updated key / value
silent reconnect to watch may be dangerous.
if watch starts from specific revision, callbacks called again from these revision and its break watch stream ordered guarantee
https://etcd.io/docs/v3.3/learning/api_guarantees/#consistency
if watch start from current revision, reconnect can skip some events
we are sometimes got both these problems.
but root of the problem is that occasionally connection didnt closed, and watch wait infintly. and Client code cant handle this.
that can be easy reproduce with kubectl port-forward
to any etcd in k8s, then start local etcdctl watch on etcd and stop port-forwarding.
problem may be solved with grpc pings, but they are available only from dotnet6+
https://docs.microsoft.com/en-us/aspnet/core/grpc/performance?view=aspnetcore-6.0#keep-alive-pings
from dotnet-etcd.
Thanks @setood . I see a lot of great features have been added to the grpc client. I will try to re design the whole client again.
IMO, I think it should be fair to say that people should be able to easily upgrade their dotnet versions, considering breaking changes are not that much in most areas.
from dotnet-etcd.
Related Issues (20)
- Legacy Grpc.Core fails on Linux Alpine container HOT 8
- Support IConfiguration and DI
- Exception type when cancelling async operations HOT 4
- Prefix request with Limit HOT 2
- How to read past version of keys HOT 2
- add separated community contrib project or repo HOT 1
- GrpcChannelOptions parameter configuration HOT 2
- Lock permission denied HOT 5
- Possible NullReference HOT 3
- Add optional ExceptionHandler in LeaseKeepAlive HOT 1
- Invalid port specified HOT 1
- method name is misspelled HOT 1
- Cannot use constructor EtcdClient(string) anymore HOT 1
- Exception while opening a watch on etcd grpc-proxy
- Watch stops listening to changes after server restart HOT 3
- IClusterProvider I need to implement this interface. HOT 2
- Failing to initiate with ambiguous and crashing with this exception 6.0.2-beta
- lock error HOT 2
- missing Election request HOT 1
- Switch to license expression
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dotnet-etcd.