Comments (14)
@marckamerbeek please look at : spring-cloud/spring-cloud-netflix#947
from servo.
My guess something is just registering a lot of monitors. If you dump the contents of the registry does anything stand out?
from servo.
Thanks for answering. Yesterday I disabled Servo with -Dspring.metrics.servo.enabled=false. That seems to be working. No memory leak now. This morning I enabled it again in production to see what kind of monitors are registered with /manage/metrics. I made a snapshot and will compare this with the result after a few days. I'll keep you posted.
from servo.
After one day running on production, the metrics shown in /manage/metrics is not increasing substantially. I got 133 metrices. Memory usage is increasing, so it looks like the memory leak is happening again.
from servo.
Did some more investigation on the content of the hashmap that is growing. The hashmap is showing 1.2 million instances of AtomicLong:
Following this AtomicLong Reference shows each time this trace:
So it looks like it has something to do with the use of Netflix Eureka in Zuul.
from servo.
I faced a similar issue with Servo on a long duration load test. Heap dumps showed that it primarily had the metrics from List in updateImpl() we were sending via the observer that was long delivered but still in heap. Added a finally block to explicitly nullify the List and other local objects created in the updateImpl method of the observer which seemed to help with those objects getting garbage collected
from servo.
Thanks @hxs8839 for your comment. I finally removed Netflix Eureka(was planned) and the problem is gone now. I now use the kubernetes dns with loadbalancer to find my services. I don't have the time to investigate the combination Netflix Servo and Netflix Eureka and why its causing problems.
from servo.
@hxs8839 based on the memory profiles included in this ticket that sounds like a different problem. Feel free to submit a PR if you think it would help.
Looking at NamedConnectionPool in Eureka and the memory profiles above, it looks like something is creating NamedConnectionPool instances and not unregistering.
from servo.
@brharrington agreed. Yeah I made the change for my custom observer, Working on if we can generalize it
from servo.
@brharrington. I'm not in the position right now to spend a lot of time on this. Hopefully somebody else can profit from this. Thanks anyway.
from servo.
version: com.netflix.servo:servo-core:0.10.1 by SpringBoot Dalston.SR4
from servo.
See the linked spring ticket (spring-cloud/spring-cloud-netflix#947), it seems like a bug in how spring is registering metrics.
from servo.
@brharrington #947 not a prefect solution, sometime our api-gateway don't konw how many queryparm in url, so the code will be
this.restTemplate.postForEntity(url + "?" + requestVo.getQueryString(), formEntity, String.class).getBody()
RestTemplate has no dynamic queryparm method, if use urlBuilder then question begin again
from servo.
Disable ServoMetrics by add spring.metrics.servo.enabled=false to application.properties。
Servo is the arch-criminal
from servo.
Related Issues (20)
- PollScheduler maximum threads? Configurable? HOT 2
- CloudWatchMetricObserver.putMetricData swallows AmazonServiceException HOT 2
- CloudWatchMetricObserver not pushing metrics to CloudWatch HOT 3
- Observer for InfluxDb HOT 2
- Booleans not changed into int values by MonitorRegistryMetricPoller HOT 4
- AwsInjectableTag should be lazily initialised HOT 1
- Support CloudWatch high-resolution metrics HOT 5
- StepCounters loose increments when poll is behind schedule HOT 3
- Heapdump support? HOT 1
- release build failed because it tried to publish for jdk9
- System.getProperties() in DefaultMonitorRegistry.java HOT 3
- JmxMetricPoller: Trouble adding Tabular metrics HOT 2
- Sometimes deploying a zuul project can cause such problems. HOT 1
- .travis.yml: The 'sudo' tag is now deprecated in Travis CI
- upgrade Guava dependency to something recent HOT 6
- Ufyfug
- Hello
- Integration of servo-core into OSS-Fuzz HOT 1
- Netflib
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from servo.