toptal / jvm-monitoring-agent

Monitor JVM from within, detect thread blocks and automatically save threads dump

JVM Monitoring Agent

Monitor JVM threads and save a threads dump when threads get blocked for given time.

Usage

Download the jar from https://github.com/toptal/jvm-monitoring-agent/releases

Add it to the command line of the application you wish to monitor:

java -javaagent:jvm-monitoring-agent-0.9.0.jar=threshold=1000,debug ...rest of command

Configuration flags/options

debug

Enables debugging output - in cases you suspect something is wrong it might provide additional information. To debug options problems make it first passed parameter.

path=...

Specifies where to save threads dumps.

interval=...

Specifies how often to check thread list for blocked threads. (Milliseconds)

threshold=...

Specifies how long thread needs to be blocked before dumps will be saved. (Milliseconds)

delay=...

Specifies delay between saving consecutive threads dumps. (Milliseconds)

filterRegex=...

Specifies a regular expression to filter known blocked threads. (Milliseconds)

Examples

Examples in examples/ are meant to check this library manually.

Check the agent does not interrupt exit.

rm tmp/*
ant && (cd examples/empty/ && ant) &&
java -javaagent:dist/jvm-monitoring-agent.jar -jar examples/empty/dist/empty.jar
ls -l tmp/

There should be no output from the execution.

Check the agent does find blocked threads.

rm tmp/*
ant && (cd examples/SynchronizedThreads/ && ant) &&
java -javaagent:dist/jvm-monitoring-agent.jar=debug,threshold=1000,delay=5000 -jar examples/SynchronizedThreads/dist/SynchronizedThreads.jar
ls -l tmp/

Execution is supposed to show arguments and times it took each second.

License

JVM Monitoring Agent is licensed under GNU Affero General Public License (GNU AGPLv3).

jvm-monitoring-agent's Issues

Limit how offten a threads dump will be saved

Currently in each iteration when threads a re blocked we will save a dump, it should be possible to limit it with extra threshold, and it probably could be different then the one to check for how long thread was blocked.

Relevant lines:

• jvm-monitoring-agent/src/com/toptal/jvm/monitoring/Agent.java

  Line 109 in e22e410

  saveThreadsDump(threads);

Monitor threads spending time on IO

It is possible that threads are not blocked but are spending time on IO and preventing things from happening, we could write an example to simulate that and catch such situations.

Question: Should it be separate agent or parameter?

Related code:

• jvm-monitoring-agent/src/com/toptal/jvm/monitoring/Agent.java

  Line 115 in e22e410

  (thread.getState() != Thread.State.BLOCKED)

Send JMX notifications on stuck threads

In addition, or even instead of saving a thread dump we could also send JMX notifications that threads are stuck.

Extra idea - associate threads_dump file with a thread and when it get's unblocked - trigger a notification with the file name.

Use milliseconds instead of Date for timestamp

To limit strain on GC we can simplify blockedThreads to include long of milliseconds(System.currentTimeMillis()) instead of Date.

Relevant lines:

• jvm-monitoring-agent/src/com/toptal/jvm/monitoring/Agent.java

  Line 33 in e22e410

  private final WeakHashMap<Thread, Date> blockedThreads = new WeakHashMap<>();

• jvm-monitoring-agent/src/com/toptal/jvm/monitoring/Agent.java

  Line 124 in e22e410

  blockedThreads.putIfAbsent(thread, new Date())

• jvm-monitoring-agent/src/com/toptal/jvm/monitoring/Agent.java

  Line 130 in e22e410

  long now = new Date().getTime();