kmtong / logback-redis-appender Goto Github PK

com.cwbase.logback.RedisAppender calls methods redis.clients.jedis.JedisPool.returnBrokenResource(...) and redis.clients.jedis.JedisPool.returnResource(...)
that are marked as protected in jedis-3.1.0 (and my guess is that it's disabled since jedis-3.0.0).

redis.clients.jedis.Jedis.close() should be called instead

Hello,

We are exploring to see if we could use logback-redis-appender to collect logs from 20+ micro services to single location. Most of the feature we need are already baked in but were wondering if there was support to lookup tags from MDC property map.
For example -

<tags>
      ${request_id}
</tags>

Where request_id is set in the MDC map. We need the value of request_id to be added to the json tags field.

Thanks

can i add some external log info such as traceId、spanId to final json message?

For some unknown reason logstash fails to put the log events added with the RedisAppender to Redis into elasticsearch. When I modify the RedisAppender to use the net.logstash.logback.layout.LogstashLayout it works.

I propose that the Layout is further decoupled from the appender by allowing the user to set or override the appender to use.

I have write value to redis, but I can't get it. How to get it? BLPOP, or BRPOP?

Every log posted to Redis via the logback appender is throwing Readtimeout execption, Redis is feeding the ELK stack we can see all logs passing through to Elastic and Kibana, so there is no loss of logs. But could not understand the reason for below exception. Any pointers/help please?

Redis is running on AWS EC2 instance and the log posting app is on my local set up. Strangely this error is not observed when log are posting from a client running with in AWS on another EC2 instance.

Using the logback Redis appender for posting logs to Redis com.cwbase logback-redis-appender 1.1.2

at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:201)
at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40)
at redis.clients.jedis.Protocol.process(Protocol.java:132)
at redis.clients.jedis.Protocol.read(Protocol.java:196)
at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:288)
at redis.clients.jedis.Connection.getIntegerReply(Connection.java:213)
at redis.clients.jedis.Jedis.rpush(Jedis.java:797)
at com.cwbase.logback.RedisAppender.append(RedisAppender.java:43)
at com.cwbase.logback.RedisAppender.append(RedisAppender.java:16)
at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:88)
at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:48)
at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:273)
at ch.qos.logback.classic.Logger.callAppenders(Logger.java:260)
at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:442)
at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:396)
at ch.qos.logback.classic.Logger.info(Logger.java:600)
at org.wmc.sampletest.App.main(App.java:24)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:108)
at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:195)
... 16 more

Hi, we faced the problem, that we have a cluster of Kibana servers and yesterday the master changed.
In the logback.xml we had:

<appender name="LOGSTASH" class="com.cwbase.logback.RedisAppender">
   <host>redis-01</host>
   <port>6379</port>
   <key>***</key>
   <type>****</type>
   <tags>****</tags>
   <sourceHost>${HOSTNAME}</sourceHost>
</appender>

now the master is redis-02 and the logs could not be written in redis-01 anymore . It would be helpful to be able to set up the cluster of servers.

Currently, logs written to the RedisAppender are sent using the same thread as the one that called the logger.

This can lead to issues if the Redis server is down, if there is unexpected network latency or any other delay in communication.

If there are no plans to change this behavior, I suggest that the documentation clearly states this. The documentation should also refer users to logback's AsyncAppender and highly recommend its usage.

Thanks!

couldn't find any license information about the project

what is the license? MIT License? Apache v 2.0= could you provide some or add it to your repository? that would be great.

Sometimes if you use haproxy or other proxies in the middle of the connection towards redis, the proxy time outs the client. This leads to nasty error/warning messages being logged. Setting testOnBorrow on GenericObjectPoolConfig in RedisAppender solves this by validating the connection when a new one is pulled from the pool.

Is there a reason why this is happening?

https://github.com/kmtong/logback-redis-appender/blob/master/src/main/java/com/cwbase/logback/JSONEventLayout.java#L136