Comments (14)
This is on our roadmap, we're aiming to add proper logging support later this year.
PRs are of course welcome before that :)
Corresponding Python client's logger, start here:
https://github.com/confluentinc/confluent-kafka-python/blob/master/confluent_kafka/src/confluent_kafka.c#L1669
from confluent-kafka-go.
I implement this recently, and use both separate goroutine and channel: #295
And i think this feature is quite useful for online issue debug.
from confluent-kafka-go.
Broker connection errors are considered temporary and the client will try to recover automatically, that's why these errors are not directly propagated as per-message errors but rather global errors (or logs).
The not-yet-produced messages will remain in internal queues until the relevant broker(s) comes up again or the messages time out (message.timeout.ms
, def 5 minutes).
Regarding the logging output: it is not currently possible but it is on the roadmap.
Do you have a preference for how you would like to receive log messages? A Log event on the .Events channel? A separate channel? something else?
from confluent-kafka-go.
Hi,
I think a separate "notification" log channel would be great. Otherwise possibly allow someone the option of the librdkafka library just calling the standard logger in https://golang.org/pkg/log/, thereby letting them handle sending the log to a file or whatever they want via that package's options.
from confluent-kafka-go.
Hi, I vote for a separate channel to receive a structured as well-defined errors, no strings please.
from confluent-kafka-go.
May also want to give the option to just configure the log callback, which can be done in librdkafka
via rd_kafka_conf_set_log_cb
. The current lack of control over log output is unworkable in a number of environments.
from confluent-kafka-go.
That is somewhat problematic since the librdkafka log_cb may be called from internal librdkafka threads, it might be safer to emit log messages as Events and let the application poll/consume them.
This is what we did in the Python client.
from confluent-kafka-go.
I wonder about possible issues with that approach, as it would delay the logging of messages, which may be lot in the case of some fatal failure in the meantime. But it would certainly be an improvement on the current state.
from confluent-kafka-go.
A separate channel perhaps? It would have to enabled (by default disabled) using a special config key I guess so that the producer/consumer does not deadlock when the channel is not being received upon.
from confluent-kafka-go.
That could work. A separate go routine can consume the log messages.
from confluent-kafka-go.
Is it some news about this point?
In a golang app using confluent-kafka-go, how to handle with connection to broker & librdkafka err logs?
Thx
from confluent-kafka-go.
I believe it is implemented in v0.11.6:
for e := range p.Producer.Events() {
switch ev := e.(type) {
case *kafka.Message:
// reply message ....
case kafka.Error:
log.Errorf("%v", ev)
if ev.Code() == kafka.ErrAllBrokersDown {
// do what is required when brokers are down
}
}
I believe this issue can be closed
from confluent-kafka-go.
v0.11.6 added proper Error reporting, but there is still no support for specifying an alternate logger. All logs currently go to stderr.
We're scheduling this for early 2019, unless someone from the community wants to help out.
This boils down to enabling librdkafka log queue and modifying the event loop in event.go to translate librdkafka log events to the Go log framework.
from confluent-kafka-go.
Merged in #421.
from confluent-kafka-go.
Related Issues (20)
- Failed to build confluent-kafka-go on arm64 machine. HOT 2
- golang confluent kafka producer client running into error - Failed to create thread: Resource temporarily unavailable (11) HOT 1
- Automatic offset commit led to the loss of some messages HOT 2
- delivery.report.only.error check does not work in case type of the value is not boolean
- Async Producer Panic HOT 1
- Is the producer a tcp keepalive connection
- Signal: bus error when running on macos sonoma 14.4.1 HOT 1
- allow us to unsubscribe from a topic more easily HOT 4
- Producer messages stuck flushing even though there weren't any writes. HOT 4
- Consumers unable to join group until group leader is restarted HOT 3
- Newly added consumers with different version of library in group are not consuming from subscribed topics.
- Schema reference for JSON schemas HOT 1
- consumer recived repeat message after crash
- Does confluent-kafka-go producer PartitionAny is round robin produce message to kafka? HOT 1
- Can we set krb5.conf path for kinit?
- Duplicate messages consumed during rebalancing HOT 2
- Schema Registry misleading error message when unauthorized
- Segmentation violation Error with confluent kafka GO sdk v2 HOT 3
- Can't provide my own http.Client to schemaregistroy.Client via schemaregistry.Config HOT 2
- Question about producer and deliveryChan HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from confluent-kafka-go.