Comments (9)
The error comes from:
https://github.com/Azure/azure-functions-eventhubs-extension/blob/dev/src/Microsoft.Azure.WebJobs.Extensions.EventHubs/Listeners/EventHubListener.cs#L160
The method called when the underlying client experiences an error while receiving. EventProcessorHost will take care of recovering from the error and continuing to pump messages.
https://docs.microsoft.com/en-us/dotnet/api/microsoft.azure.eventhubs.processor.ieventprocessor.processerrorasync?view=azure-dotnet
So nothing to worry about, your messages did not drop as there wasn't checkpoint operation.
from azure-functions-eventhubs-extension.
Thanks @alrod
So if I understand correctly this is a bug. Because an expected exception (System.Net.Sockets.SocketException) is being logged with a high severity?
What's the best place to report/fix this?
from azure-functions-eventhubs-extension.
I am not sure if this a valid bug:
The method called when the underlying client experiences an error while receiving. EventProcessorHost will take care of recovering from the error and continuing to pump messages.
https://docs.microsoft.com/en-us/dotnet/api/microsoft.azure.eventhubs.processor.ieventprocessor.processerrorasync?view=azure-dotnet
Errors during pumping new messages are expected and EventHub guranties a message a will be delivered at least once.
How often do you get SocketExceptions
? only during a night?
from azure-functions-eventhubs-extension.
I guess the bug for me is the serverity (just like in #5).
Consider:
- It's expected. So it's not exceptional.
- If theres no action I can take (the event processor host recovers itself, and I don't think I even have access to it?) - then why do I need to know about it?
I am trying to monitor the event hub -> function app connection so that I can quickly react to any issues. For example, just today a company I'm working with revoked the shared access key, and but watching for errors I could quickly let them know and restore the string.
Having a couple of message with a high serveriy level where the only action we can take is to ignore them does not scalable. Especially when they happen at two in the morning!
from azure-functions-eventhubs-extension.
(To answer your question - we've had 16 over the last week)
from azure-functions-eventhubs-extension.
we can't change severity level for specific exception types. we consider ReceiverDisconnectedException and LeaseLostException as information because these exceptions can happen as part of normal partition balancing across instances.
16 per week is fairly low amount, I only can recommend adjust your alerting logic:
- Filter by specific exception types.
- Send an alert if you have more then
x
erros duringy
period of time.
from azure-functions-eventhubs-extension.
Ok, I guess I'm just missing context here
The issue with only sending an alert if there are more than x a minute is that we risk missing actual exceptions. (For example, we get the odd storage exception that happens due to an error we need to resolve).
And if I filter by the socket exception, then it feels like we're all acknowledging that all socket errors expected in normal operation; and that there's no risk to ignoring them?
I guess I'm just missing the context as to why we can't alter the severity of the exception that we know is not exceptional?
Maybe there is an opportunity to document a list of exceptions, like these, which are thrown as part of the internals of the function app when discussing monitoring event hub integrations?
(As an aside, 16 might not sound like much, but there's only 10 devices pushing data; which is why it doesn't feel very scalable)
from azure-functions-eventhubs-extension.
Or to put it another way. It feels like you're saying it's reasonable to expect anyone who wants to monitor an event hubs integration to build the following query:
exceptions
| where severityLevel > 1 // only alert on errors and warnings
| where innermostMessage != "The lease ID specified did not match the lease ID for the blob." // known bug: https://github.com/Azure/azure-functions-eventhubs-extension/issues/42
| where problemId != "System.Net.Sockets.SocketException at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw" // unexceptional: https://github.com/Azure/azure-functions-eventhubs-extension/issues/43
| where not (problemId startswith "System.ArgumentOutOfRangeException at Microsoft.Azure.EventHubs.Processor.PartitionContext") // unexceptional: https://stackoverflow.com/questions/51602965/azure-eventhub-function-leaselostexception
from azure-functions-eventhubs-extension.
Closing due to inactivity.
from azure-functions-eventhubs-extension.
Related Issues (20)
- Consumptionplan wont scale out if eventhub connection is managedIdentity HOT 1
- Error : Event Hub Trigger The response ended prematurely, with at least 156 additional bytes expected. HOT 2
- Stop Listener when disposed HOT 2
- Can not have enough instances HOT 2
- Using single dispatch, extension sometimes logs incorrect trigger details
- This repo is missing important files HOT 2
- Two different events on one trigger HOT 10
- Incorrect EventHubTrigger properties received in deviceLifecycleEvents when "cardinality:"one"
- Currently, eventhub triggers require the role to be granted on eventhub level, not only on consumer group.
- Single slow function execution leads to all events on the same partition being delayed.
- EventHubTrigger vs. EventProcessor and CPU utilization #3036
- Add retry logic to EventHubAsyncCollector.SendBatchAsync in the `partitioned-output` branch HOT 2
- [Bug] "Partion" Typo should be "Partition"
- Event Hub Extension Feature Request: Log the details about the Event Hub being monitored in the Bindings and also the Storage account. HOT 1
- EventHub outbinding having an issue when a same function try to send an event to two different EH namespace but having same EH name HOT 2
- Retry loop should cancel on lease lost HOT 2
- The lease ID specified did not match the lease ID for the blob HOT 5
- Include the Function name on log messages HOT 1
- Log eventhub namespace and eventhub name to detect cases when another triggers listens the same eventhub.
- Scale monitor does not respect initial offset position
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from azure-functions-eventhubs-extension.