frankchen021 / bithon Goto Github PK

View Code? Open in Web Editor NEW

14.0 14.0 4.0 15.73 MB

An observability platform mainly for Java

License: Apache License 2.0

Java 74.01% Shell 0.06% JavaScript 21.97% HTML 2.15% CSS 1.24% SCSS 0.52% ANTLR 0.05%

agent alerting apm java logging metrics tracing

bithon's People

Contributors

Stargazers

Watchers

Forkers

kai8406 nguyenminhduc9988 yuanlihan

bithon's Issues

Merge requestTime/responseTime of redis metrics together as RT

jedis provides requestTime/response respectively, it adds some complexity for implementation. And from users' side, they usually don't care about how long a command is sent to server and how long it takes to return the response, instead, they care about how long a client command complets.

And it's hard to provide such response time for some client libs such as lettuce which utilizes async operations heavily.

This issue tracks the merge of current requestTime/responseTime together as a new responseTime

Add command line args to disable agent

it's convenient when debugging without deleting the -javaagent argument to disable the agent

Integrate web-server into collector for standalone evaluation

Duplicated thread pool

Refresh dashboard/chart incrementally

Currently, all refresh requires reload ALL data in given intervals. This should be improved by loading data incrementally.

To do this, the chart component at client side should holds a timestamp when it load data last time.

Support Aggregations for endpoint topo

Handle messages by batch
Aggregate metrics for topo to reduce records greatly for JDBC storage

Add targetHost dimension to http-client-metrics

Currently, targetHost and targetUri are stored together as URL, it's best to separate host and uri so that it provides flexible ways to query.

Decouple message format of Kafka collector from current KafkaSink

Current Kafka collector requires metric messages in array format, which limits the extensibility of Bithon to easily support messages come from 3rd systems.

This issues describes how we plan to solve it:
for Kafka collector:

it accepts a single metric object in a single kafka message
it accpets multiple metric objects in a single kafka message. Contrast to the array format message, each of metric object is serialized into string, and are concat by empty characters such as carriage return
Kafka collector polls kafka messages in a batch and turns the batched messages into existing SizedIterator object

for kafka sink,
it serialize an array of metric object as described in scenario 2 above.

For 3rd systems, it usually sends a single metric object in a single kafka message as described in scenario 1 above.

Schema Manager

automatically imported base schemas from current json to schema storage
schema storage equals to metadata storage
management api to add schemas
management api to update retention time of metrics
simple UI to manage schemas

Add support to C3P0 connection pool metrics

OKHttp3 Trace Support

Add GROUP-BY semantic to chart component

For example, with the support of GROUP-BY semantics, metrics of young gc and full gc could be put on one chart.

Add MongoDb server view

Add a separated class loader to replace shaded classes and resolve jar compatibilities

Currently, all interceptors are loaded in a separated class loader. This works well except involving shading libraries.

For underlying communication such as thrift, we're now using version 0.14 the latest at the moment, but some apps are using older versions such as 0.11. Since thrift is not shaded and much difficult for us to shade the generated code and communicate layer code, there would be version conflicts.

A better way to solve it is to implement a agent class loader which loads all classes required by agent. Once this class loader is provided, the shading module could also be eliminated which would be a little easier to build and maintain this project.

NoClassDefFoundError

2021-04-11 11:53:54.810 [Agent WARN] - [main:AopLogger] Error occurred during invoking LoggerCallAppenders.after()
java.lang.NoClassDefFoundError: ch/qos/logback/classic/spi/ILoggingEvent
	at com.sbss.bithon.agent.plugin.logback.interceptor.LoggerCallAppenders.onMethodLeave(LoggerCallAppenders.java:41)
	at com.sbss.bithon.agent.boot.aop.AroundMethodAop.intercept(AroundMethodAop.java:79)
	at com.sbss.bithon.agent.boot.aop.MethodAop.intercept(MethodAop.java:47)
	at ch.qos.logback.classic.Logger.callAppenders(Logger.java)
	at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
	at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
	at ch.qos.logback.classic.Logger.log(Logger.java:765)
	at com.alibaba.druid.support.logging.SLF4JImpl.info(SLF4JImpl.java:72)
	at com.alibaba.druid.pool.DruidDataSource.init$original$pmpVFQl2(DruidDataSource.java:1002)
	at com.alibaba.druid.pool.DruidDataSource.init$original$pmpVFQl2$accessor$UqqJrtCB(DruidDataSource.java)
	at com.alibaba.druid.pool.DruidDataSource$auxiliary$O0MTLNRK.invoke(Unknown Source)
	at com.sbss.bithon.agent.boot.aop.AroundMethodAop.intercept(AroundMethodAop.java:64)
	at com.sbss.bithon.agent.boot.aop.MethodAop.intercept(MethodAop.java:47)
	at com.alibaba.druid.pool.DruidDataSource.init(DruidDataSource.java)
	at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1311)
	at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1307)
	at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:109)
	at org.springframework.boot.autoconfigure.h2.H2ConsoleAutoConfiguration.lambda$h2Console$0(H2ConsoleAutoConfiguration.java:72)
	at org.springframework.beans.factory.ObjectProvider.ifAvailable(ObjectProvider.java:93)
	at org.springframework.boot.autoconfigure.h2.H2ConsoleAutoConfiguration.h2Console(H2ConsoleAutoConfiguration.java:71)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:652)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:637)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1336)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1176)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:207)
	at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:211)
	at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:202)
	at org.springframework.boot.web.servlet.ServletContextInitializerBeans.addServletContextInitializerBeans(ServletContextInitializerBeans.java:96)
	at org.springframework.boot.web.servlet.ServletContextInitializerBeans.<init>(ServletContextInitializerBeans.java:85)
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.getServletContextInitializerBeans(ServletWebServerApplicationContext.java:255)
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.selfInitialize(ServletWebServerApplicationContext.java:229)
	at org.springframework.boot.web.embedded.tomcat.TomcatStarter.onStartup(TomcatStarter.java:53)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5166)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384)
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909)
	at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:843)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384)
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909)
	at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
	at org.apache.catalina.core.StandardService.startInternal(StandardService.java:434)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
	at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
	at org.apache.catalina.startup.Tomcat.start(Tomcat.java:486)
	at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:123)
	at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.<init>(TomcatWebServer.java:104)
	at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getTomcatWebServer(TomcatServletWebServerFactory.java:440)
	at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getWebServer(TomcatServletWebServerFactory.java:193)
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:178)
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:158)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:545)
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:143)
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758)
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750)
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:405)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)
	at com.sbss.bithon.server.starter.StarterApplication.main(StarterApplication.java:31)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:107)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)

Replace fastJSON by jaskson at agent side

Disruptor evaluation

to replace file queue

Support to save traces to ES

Support customized URL normalization rule

For URL like

http://dev.discovery.bithon.cn/eureka/apps/DEMO/172.16.1.1:demo:8080

applying customized rule, it could be folded into

http://dev.discovery.bithon.cn/eureka/apps/{appName}

http://dev.discovery.bithon.cn/eureka/apps/DEMO/{instance}

Add database dimension to mongodb metrics

License Setup

Fix license for those code snippets from Apache/Druid
Choose a license for this project

Apply max/min/avg request/response time to redis metrics

Current redis-metrics show request/response time by SUM aggregation, avg/min/max aggregations are better

Add Kafka metrics

Producer metrics
Consumer metrics
Topo

No Handler for message [mongo-metrics]

ERROR 51824 --- [pool-4-thread-5] c.s.b.s.c.sink.local.LocalMetricSink : No Handler for message [mongo-metrics]

Renaming agent-bootstrap and agent-boot

These two names are confusing.

Add bytes metrics to web-request

Add alerting system

HTTP client metrics processed not correctly

Introduced by #61

Flatten JVM metrics definition in thrift

Current jvm metrics contains several metric categories such as CPU/Memory/Thread. Since the message format of thrift is defined to reflect these categories, it introduces some differences compared to other metrics, which causes code difference in ThriftMetricCollector

This issue propose to flatten the metric definition in thrift to unify the metric processing code pattern

Add Webflux support

web-request metrics update

callCount --> total
add 'okCount'

Add Kafka server view

Failed to load metrics in 12h/24h

Add requestBytes/responseBytes metrics in JDK HttpClient

sun.net.NetworkClient

    public void openServer(String server, int port)
        throws IOException, UnknownHostException {
        if (serverSocket != null)
            closeServer();
        serverSocket = doConnect (server, port);
        try {
            serverOutput = new PrintStream(new BufferedOutputStream(
                                        serverSocket.getOutputStream()),
                                        true, encoding);
        } catch (UnsupportedEncodingException e) {
            throw new InternalError(encoding +"encoding not found", e);
        }
        serverInput = new BufferedInputStream(serverSocket.getInputStream());
    }

Add dubbo support

Add interceptors for Spring Beans to support trace

Add Redis server view

Support to open a chart in dialog

Open chart in dialog
re-select time range in dialog
compare chart with history

A new Generic Metrics Binary Format

Currently, to develop a new metric, we have to do 4 things as below:

a thrift/gRPC proto-type has to be defined first and then use thrift/gRPC compiler to compile the new structure
add a new sendxxx method to thrift/gRPC interface
implement the newly added thrift/gRPC interface at server side
add a new conversion function to ThriftMessageConverters to convert user defined object to thrift/gRPC defined messages

Although thrift/gRPC has provided a high efficient ways to transport metrics from client to server , current way is still not friendly for agent development.

To change that , a new generic metrics binary format is proposed as below.

The reason why new format is still binary instead of something like JSON is that it has the highest performance and lest bytes transmitted on network, especially for metrics like http call and request which usually have large records for high QPS systems.

In the modern days, there're many binary serialization format we could choose. One is AVRO.

But since the pattern of metrics is the same(what's different is the number of dimensions and metrics), and we don't want to generate serialization/deserialization code based on predefined schema(If schema is needed at agent side, why not we still use the current thrift way ?) , it's better to provide customer defined message format.

For 3rd systems that will be integrated, the customized message format is not a burden because it's always recommended to support these 3rd systems at server side.

Generic Metrics Binary Format

Metric Batch

Every time the client send a batch of metrics with different dimension values to server.

Type	Value
string	data source name
list< string >	schema
list< MetricSet >	metricSets

Metric Row

MetriSet

A metric-set is composed of a timestamp, a set of dimensions, a set of metric values. Types of all dimensions are string.

Type	Value
long	timestamp
list < string >	dimensions
list < MetricValue >	metrics

MetricValue

A metric value stores the actual value for a metric. There're 2 types of metric value now:

double
long
And in future, there will be Histogram.

A TLV scheme is used to serialize metric value to binary. And for (long/double) value encoding , ZigZag is applied.

T value table

The table below lists the value of T field of a MetricValue. A byte is enough to represent this field.

T	Meaning
1	Long
2	Double
3	Histogram

Deserialization

Since the format does not contain any names of dimensions/metrics, a schema is still needed at collector side as it does now.
The schema is used when turning this binary format into a generic MetricSet object which is based on a HashMap.

Time-series/Group-by query should fill zero for missing time point

If not provided, there's only one data point shown in the graph as above.

Support to filter incoming HTTP requests at agent side

Support to filter requests by user-agent
Support to filter requests by uri rules
Support to config filter rules

Wrong menu item link for redis metrics

Add TTL to metrics

It's important to have ttl defined for metrics so that number of records won't be grown too large.

For JDBC storage, there must be a daemon thread to do cleanup periodically.
For systems which have TTL concept, such as mongo, es, it could be simplified by applying ttl configuration to these system.

And TTL could be configured for each data source

Refactor metric menu items

Currently, menu items are built through MetricSidebar client component. There're some problems:

application topo, is an internal data source, it should be hidden from end user
it's difficult to sort the items returned from back end service, because the sort itself is related UI only
Garbage collection, has been shown in 'JVM' and should be hidden from end user

So, it's better to define menu items at client side only.

Application selection state
Instance selection state
Interval selection state