Comments (6)
And example how OldGen heap looks like with ZGC on Java 21 with this problem
from vert.x.
@bolbat And I was very wrong, it seems there's no way to prevent indexes to keep on growing there, so, the fix made by @vietj should be the one to solve AND improve performance in one go (as the original Pr I sent was meant to do, sorry for that :"( )
from vert.x.
Confirming that problem was fixed in 4.5.2-SNAPSHOT
Heap looks stable, InternalThreadLocalMap map index and size is near 50 elements, WebClient works fine.
Heap OldGen graph for last 2 days from one node:
from vert.x.
If someone would need code to debug this situation I will share my as example
Code is very ad hock, just for fast debugging
Add this dependencies to ByteBuddy library
<dependency>
<groupId>net.bytebuddy</groupId>
<artifactId>byte-buddy</artifactId>
</dependency>
<dependency>
<groupId>net.bytebuddy</groupId>
<artifactId>byte-buddy-agent</artifactId>
</dependency>
Instrument related code with something like this
Call instrumentForDebug on some early application startup stage
public static void instrumentForDebug() {
ByteBuddyAgent.install();
new ByteBuddy()
.redefine(FastThreadLocal.class)
.visit(Advice.to(FastThreadLocalInstantiationInterceptor.class).on(ElementMatchers.isDefaultConstructor()))
.make()
.load(InternalThreadLocalMap.class.getClassLoader(), ClassReloadingStrategy.fromInstalledAgent())
.getLoaded();
}
public static final class FastThreadLocalInstantiationInterceptor {
public static final Map<String, AtomicInteger> STATS = Maps.mutable.empty();
@OnMethodEnter
public static void intercept() {
final String stackTraceString = ExceptionUtils.getStackTrace(new Throwable());
final AtomicInteger existing = STATS.get(stackTraceString);
if (existing == null) {
STATS.put(stackTraceString, new AtomicInteger(1));
} else {
existing.incrementAndGet();
}
}
}
Prepare some debug information and log it or get by some debug API like I did
private void debugFastThreadLocalInstantiationStats(final StringBuilder sb, final int minInstantiations) {
line(sb.append("FastThreadLocal Instantiation Stats"));
FastThreadLocalInstantiationInterceptor.STATS.forEach((k, v) -> {
final int instantiations = v.get();
if (instantiations >= minInstantiations) {
line(sb.append("Instantiation[").append(instantiations).append("] ")
.append("StackTrace: ").append(k));
line(sb);
}
});
}
private void line(final StringBuilder sb) {
sb.append(System.lineSeparator());
}
from vert.x.
thanks @bolbat
from vert.x.
FYI, I think you could the the leak detection of a sync profiler (adding --live to the cmd line options) to detect this or just allocation profiling.
Thanks for the analysis, although by reading this I don't see the problem in the fast thread local, but instead in the high number of combiner executors created: any idea why @vietj ?
Said that, fast thread locals could be "smarter" in resizing the array, if it knows that a combiner is no longer reachable, and could do a better job.
This means making it autocloseable and handle thread local dispose somehow and we should have a clean life cycle for it already, just need to use it to enforce thread local disposal
from vert.x.
Related Issues (20)
- HTTP trailers in requests HOT 1
- java.util.NoSuchElementException thrown after the HttpClientRequest.reset() is called. HOT 2
- Vert.x stress test problem after 16xxx request sent HOT 8
- io.vertx.ext.web.RoutingContext json() method not reporting json.EncodeException HOT 1
- performance degradation when no handler HOT 1
- Resource segregation
- Received HTTP message with no request in progress HOT 4
- Access the vertx instance from HttpClient objects HOT 1
- Event executor provider SPI
- Micrometer configuration lifecycle violation HOT 2
- HttpServerRequest connection.close() does not trigger exceptionHandler() HOT 1
- HAManager couldn't see quorum if during its initialization cluster manager has all nodes joined already HOT 5
- Instance Level Load Balancing of a Subcriber in a PUB SUB Mode on the event bus HOT 1
- Export vert.x specific metrics when using Opentelemetry java agent. HOT 2
- Persistent HTTP/1.1 connection doesn't get upgraded to HTTP/2
- Missing peer host and port info in SSLEngine for server SslHandler HOT 5
- Connection asynchronous operations should use the current context instead of the connection context
- Used VM option to config vert.x deployment option, But setter invoke method is Unreachable. HOT 2
- vert.x 4.4.9 and above, http server will not receive requests after deployment HOT 1
- Undeployed Virtual Thread verticles cause memory leaks and/or never stop HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vert.x.