Memory leak suspect in BasicMonitorRegistry about servo

[sunl]

@marckamerbeek please look at : spring-cloud/spring-cloud-netflix#947

[brharrington]

My guess something is just registering a lot of monitors. If you dump the contents of the registry does anything stand out?

[marckamerbeek]

Thanks for answering. Yesterday I disabled Servo with -Dspring.metrics.servo.enabled=false. That seems to be working. No memory leak now. This morning I enabled it again in production to see what kind of monitors are registered with /manage/metrics. I made a snapshot and will compare this with the result after a few days. I'll keep you posted.

[marckamerbeek]

After one day running on production, the metrics shown in /manage/metrics is not increasing substantially. I got 133 metrices. Memory usage is increasing, so it looks like the memory leak is happening again.

[marckamerbeek]

Did some more investigation on the content of the hashmap that is growing. The hashmap is showing 1.2 million instances of AtomicLong:

Following this AtomicLong Reference shows each time this trace:

So it looks like it has something to do with the use of Netflix Eureka in Zuul.

[hxs8839]

I faced a similar issue with Servo on a long duration load test. Heap dumps showed that it primarily had the metrics from List in updateImpl() we were sending via the observer that was long delivered but still in heap. Added a finally block to explicitly nullify the List and other local objects created in the updateImpl method of the observer which seemed to help with those objects getting garbage collected

[marckamerbeek]

Thanks @hxs8839 for your comment. I finally removed Netflix Eureka(was planned) and the problem is gone now. I now use the kubernetes dns with loadbalancer to find my services. I don't have the time to investigate the combination Netflix Servo and Netflix Eureka and why its causing problems.

[brharrington]

@hxs8839 based on the memory profiles included in this ticket that sounds like a different problem. Feel free to submit a PR if you think it would help.

Looking at NamedConnectionPool in Eureka and the memory profiles above, it looks like something is creating NamedConnectionPool instances and not unregistering.

[hxs8839]

@brharrington agreed. Yeah I made the change for my custom observer, Working on if we can generalize it

[marckamerbeek]

@brharrington. I'm not in the position right now to spend a lot of time on this. Hopefully somebody else can profit from this. Thanks anyway.

[yanickxia]

i have same problem

hashmap has too many key

version: com.netflix.servo:servo-core:0.10.1 by SpringBoot Dalston.SR4

[brharrington]

See the linked spring ticket (spring-cloud/spring-cloud-netflix#947), it seems like a bug in how spring is registering metrics.

[yanickxia]

@brharrington #947 not a prefect solution, sometime our api-gateway don't konw how many queryparm in url, so the code will be

this.restTemplate.postForEntity(url + "?" + requestVo.getQueryString(), formEntity, String.class).getBody()

RestTemplate has no dynamic queryparm method, if use urlBuilder then question begin again

[wyzssw]

Disable ServoMetrics by add spring.metrics.servo.enabled=false to application.properties。
Servo is the arch-criminal