aogburn / yatda Goto Github PK

This results in overlaps of matching nids so more threads are checked than expected. So for example, if a thread has a short nid like 0x529, it can match and compare with 0x529* (0x529b, 0x529c, etc.) for bad math from the improper thread comparisons:

"non-blocking-thread--p9-t3" #181 daemon prio=5 os_prio=0 cpu=0.31ms elapsed=18650.74s tid=0x0000563107d3c800 nid=0x529 waiting on condition [0x00007f50d839e000] "thread-1757,ee,sso-43-675hj" #6403 prio=5 os_prio=0 cpu=1312.12ms elapsed=300.70s tid=0x0000563109591000 nid=0x529b waiting on condition [0x00007f50d4c75000] "thread-1758,ee,sso-43-675hj" #6404 prio=5 os_prio=0 cpu=2001.58ms elapsed=300.70s tid=0x000056310c5ed000 nid=0x529c waiting on condition [0x00007f50e040c000] /home/aogburn/code/yatda/yatda.sh: line 549: 10**4 * (2001 - 1312) / (300700 - 300700): division by 0 (error token is "(300700 - 300700)") /home/aogburn/code/yatda/yatda.sh: line 550: [: -gt: unary operator expected

Find and summarize state of threads that don't show progression.

It could be handy to have clickable file links of files summarized by yatda

I believe both of these threads are "busy", but the remoting thread is not identified. Is there a way to account for busy remoting threads?

"default task-196" #27004 prio=5 os_prio=0 cpu=1097,08ms elapsed=8743,38s tid=0x000055da04611800 nid=0x1b56 waiting on condition [0x00007fbab2936000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000000a50dd568> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:194)
at java.util.concurrent.CompletableFuture$Signaller.block([email protected]/CompletableFuture.java:1796)
at java.util.concurrent.ForkJoinPool.managedBlock([email protected]/ForkJoinPool.java:3128)
at java.util.concurrent.CompletableFuture.waitingGet([email protected]/CompletableFuture.java:1823)
at java.util.concurrent.CompletableFuture.get([email protected]/CompletableFuture.java:1998)
at jdk.internal.net.http.HttpClientImpl.send([email protected]/HttpClientImpl.java:541)
at jdk.internal.net.http.HttpClientFacade.send([email protected]/HttpClientFacade.java:119)
...
at jdk.internal.reflect.GeneratedMethodAccessor384.invoke(Unknown Source)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke([email protected]/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke([email protected]/Method.java:566)
at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:254)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:416)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:162)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3.security.AuthorizationInterceptor.processInvocation(AuthorizationInterceptor.java:138)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
at framework.security.interceptor.ServerSecurityInterceptor.aroundInvoke(ServerSecurityInterceptor.java:60)
at jdk.internal.reflect.GeneratedMethodAccessor50.invoke(Unknown Source)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke([email protected]/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke([email protected]/Method.java:566)
at org.jboss.as.ejb3.component.ContainerInterceptorMethodInterceptorFactory$ContainerInterceptorMethodInterceptor.processInvocation(ContainerInterceptorMethodInterceptorFactory.java:91)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:50)
at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:57)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:633)
at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
at org.jboss.as.ejb3.remote.AssociationImpl$$Lambda$2857/0x0000000102245440.apply(Unknown Source)
at org.wildfly.security.auth.server.SecurityIdentity.runAsFunctionEx(SecurityIdentity.java:421)
at org.jboss.as.ejb3.remote.AssociationImpl.invokeWithIdentity(AssociationImpl.java:674)
at org.jboss.as.ejb3.remote.AssociationImpl.invokeMethod(AssociationImpl.java:655)
at org.jboss.as.ejb3.remote.AssociationImpl.lambda$receiveInvocationRequest$0(AssociationImpl.java:251)
at org.jboss.as.ejb3.remote.AssociationImpl$$Lambda$2856/0x0000000102244440.run(Unknown Source)
at org.jboss.as.ejb3.remote.AssociationImpl.execute(AssociationImpl.java:344)
at org.jboss.as.ejb3.remote.AssociationImpl.receiveInvocationRequest(AssociationImpl.java:297)
at org.jboss.ejb.protocol.remote.EJBServerChannel$ReceiverImpl.handleInvocationRequest(EJBServerChannel.java:473)
at org.jboss.ejb.protocol.remote.EJBServerChannel$ReceiverImpl.handleMessage(EJBServerChannel.java:208)
at org.jboss.remoting3.remote.RemoteConnectionChannel.lambda$receiveMessage$2(RemoteConnectionChannel.java:363)
at org.jboss.remoting3.remote.RemoteConnectionChannel$$Lambda$2745/0x0000000101fd2840.run(Unknown Source)
at org.jboss.remoting3.EndpointImpl$TrackingExecutor.lambda$execute$0(EndpointImpl.java:991)
at org.jboss.remoting3.EndpointImpl$TrackingExecutor$$Lambda$2717/0x0000000101f8e840.run(Unknown Source)
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1363)
at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1280)
at java.lang.Thread.run([email protected]/Thread.java:834)
Locked ownable synchronizers:
- None

"default task-165" #25444 prio=5 os_prio=0 cpu=668,76ms elapsed=11479,75s tid=0x000055da041f4000 nid=0x78bb runnable [0x00007fba91329000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPoll.wait([email protected]/Native Method)
at sun.nio.ch.EPollSelectorImpl.doSelect([email protected]/EPollSelectorImpl.java:120)
at sun.nio.ch.SelectorImpl.lockAndDoSelect([email protected]/SelectorImpl.java:124)
- locked <0x00000000d354bb30> (a sun.nio.ch.Util$2)
- locked <0x00000000be6a5600> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select([email protected]/SelectorImpl.java:141)
at org.xnio.nio.SelectorUtils.await(SelectorUtils.java:51)
at org.xnio.nio.NioSocketConduit.awaitWritable(NioSocketConduit.java:233)
at org.xnio.conduits.AbstractSinkConduit.awaitWritable(AbstractSinkConduit.java:66)
at io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.awaitWritable(AbstractFixedLengthStreamSinkConduit.java:282)
at org.xnio.conduits.ConduitStreamSinkChannel.awaitWritable(ConduitStreamSinkChannel.java:134)
at io.undertow.channels.DetachableStreamSinkChannel.awaitWritable(DetachableStreamSinkChannel.java:87)
at io.undertow.server.HttpServerExchange$WriteDispatchChannel.awaitWritable(HttpServerExchange.java:2126)
at io.undertow.servlet.spec.ServletOutputStreamImpl.writeBufferBlocking(ServletOutputStreamImpl.java:587)
at io.undertow.servlet.spec.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:151)
...
at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117)
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
at io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:275)
at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:79)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:134)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:131)
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction$$Lambda$1235/0x0000000100f4c040.call(Unknown Source)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1551)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$1236/0x0000000100f4c440.call(Unknown Source)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1551)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$1236/0x0000000100f4c440.call(Unknown Source)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1551)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$1236/0x0000000100f4c440.call(Unknown Source)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1551)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$1236/0x0000000100f4c440.call(Unknown Source)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:255)
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:79)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:100)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:387)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:852)
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1280)
at java.lang.Thread.run([email protected]/Thread.java:834)
Locked ownable synchronizers:
- None

Stalls when executing in a directory that has a space and dash:

$ pwd
/path/to/ThreadDumps are - here
[user@localhost ThreadDumps are - here]$ ./yatda.sh jstack1.out
Checking script update. Use option '-u never' to skip the update check
md5sum: /path/to/ThreadDumps: Is a directory
md5sum: are: No such file or directory

In MacOS it can't check upstream version because there's no md5sum:

aferreir@aferreir-mac 2023-08-16 % sh ~/yatda.sh jstack_threaddump_03aug2023.out
Checking script update. Use option '-u never' to skip the update check
/Users/aferreir/yatda.sh: line 157: md5sum: command not found

To check md5sum in MacOS, we have to run md5 -r filename.ext

Ignoring the message about md5sum (that's on another issue already created) it also complains about the binary operator

aferreir@aferreir-mac 2023-08-16 % sh ~/yatda.sh jstack_threaddump_03aug2023.out
Checking script update. Use option '-u never' to skip the update check
/Users/aferreir/yatda.sh: line 157: md5sum: command not found
/Users/aferreir/yatda.sh: line 161: [: eaf590d3c184e6f1c931898f0ec1fe8b: binary operator expected

This seems to be related to the $NEWSUM variable.

Can you remove the -f option and just make the last argument the file path/name? I think that would be the most consistent w/ other command line commands? Or is there some precedence for the -f option?

Total number of pool-30-thread threads: 300
Total number of in process pool-30-thread threads: 151
Percent of present pool-30-thread threads in use: 50.33
expr: syntax error: unexpected argument ‘6’
Average number of in process pool-30-thread threads per thread dump:
Average number of 300 threads per thread dump: 50

03607445

It could be helpful to have an option to run yatda to split a multi-dump file into their own individual dumps that are separately summarized.

Shouldn't this remoting thread be identified as an in-process thread?

"default task-38" #440 prio=5 os_prio=0 tid=0x0000558f0f9e8800 nid=0x865e runnable [0x00007f3f11b1f000]
java.lang.Thread.State: RUNNABLE
at org.jboss.remoting3._private.IntIndexHashMap.doGet(IntIndexHashMap.java:521)
at org.jboss.remoting3._private.IntIndexHashMap.containsKey(IntIndexHashMap.java:178)
at org.jboss.remoting3.util.InvocationTracker.addInvocation(InvocationTracker.java:122)
at org.jboss.ejb.protocol.remote.EJBClientChannel.processInvocation(EJBClientChannel.java:307)
at org.jboss.ejb.protocol.remote.RemoteEJBReceiver$1.lambda$handleDone$0(RemoteEJBReceiver.java:91)
at org.jboss.ejb.protocol.remote.RemoteEJBReceiver$1$$Lambda$1190/929558111.notify(Unknown Source)
at org.xnio.FinishedIoFuture.addNotifier(FinishedIoFuture.java:79)
at org.jboss.ejb.protocol.remote.RemoteEJBReceiver$1.handleDone(RemoteEJBReceiver.java:76)
at org.jboss.ejb.protocol.remote.RemoteEJBReceiver$1.handleDone(RemoteEJBReceiver.java:74)
at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:208)
at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:720)
at org.jboss.remoting3.EndpointImpl$TrackingExecutor.lambda$execute$0(EndpointImpl.java:926)
at org.jboss.remoting3.EndpointImpl$TrackingExecutor$$Lambda$119/570714649.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- <0x00000006979527e8> (a java.util.concurrent.ThreadPoolExecutor$Worker)

There is a typo in yatda.sh tool for example:

## Specific findings ##

1: The number of present request threads is a multple of 16 so this may be a default thread pool size fitting 125 CPU cores.

Here multple should be multiple.

summarize the states of threads (BLOCKED, WAITING, RUNNING, etc.)

Add support for Eclipse MAT thread and jstack -F output.

For example:
$ sh ./yatda.sh --mat -f heap.hprof.threads
$ sh ./yatda.sh --forced -f jstack-F.out

I didn't add any logic for --mat or --forced, but it looks like both could be handled with these changes, at least for the basic "Top lines of all threads" and "Most common from first 10 lines of all threads":

$ diff yatda.sh yatda-mat-jstack-F.sh 
15,16c15,16
< DUMP_NAME="Full thread dump "
< ALL_THREAD_NAME=" nid=0x"
---
> DUMP_NAME="(Full thread dump |Attaching to process ID )"
> ALL_THREAD_NAME="( nid=0x|Thread [[:digit:]])"
45c45
< DUMP_COUNT=`grep "$DUMP_NAME" $FILE_NAME | wc -l`
< echo "Number of thread dumps: " $DUMP_COUNT > $FILE_NAME.yatda
< 
< THREAD_COUNT=`grep "$ALL_THREAD_NAME" $FILE_NAME | wc -l`
---
> DUMP_COUNT=`grep -E "$DUMP_NAME" $FILE_NAME | wc -l`
> THREAD_COUNT=`grep -E "$ALL_THREAD_NAME" $FILE_NAME | wc -l`
> if [ `expr $DUMP_COUNT` == 0 ]; then
>     if [ $THREAD_COUNT -gt 0 ]; then
>         # MAT thread output 
>         echo "Number of thread dumps: 1" > $FILE_NAME.yatda
>     fi
> else
>     echo "Number of thread dumps: " $DUMP_COUNT > $FILE_NAME.yatda
> fi
211c217,218
< grep "$ALL_THREAD_NAME" -A 2 $FILE_NAME | grep "at " | sort | uniq -c | sort -nr >> $FILE_NAME.yatda
---
> #grep "$ALL_THREAD_NAME" -A 2 $FILE_NAME | grep "at " | sort | uniq -c | sort -nr >> $FILE_NAME.yatda
> grep -E "$ALL_THREAD_NAME" -A 1 $FILE_NAME | grep -E "(at |- )" | sort | uniq -c | sort -nr >> $FILE_NAME.yatda
216c223,224
< grep "$ALL_THREAD_NAME" -A `expr $ALL_LINE_COUNT + 1` $FILE_NAME | grep "at " | sort | uniq -c | sort -nr >> $FILE_NAME.yatda
---
> #grep "$ALL_THREAD_NAME" -A `expr $ALL_LINE_COUNT + 1` $FILE_NAME | grep "at " | sort | uniq -c | sort -nr >> $FILE_NAME.yatda
> grep -E "$ALL_THREAD_NAME" -A `expr $ALL_LINE_COUNT + 1` $FILE_NAME | grep -E "(at |- )" | sort | uniq -c | sort -nr >> $FILE_NAME.yatda

It'd have idle threads instead like:
"default task-96" #277 prio=5 os_prio=0 tid=0x0000000004c18000 nid=0x1b8e1 waiting on condition [0x00007f333dffa000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000005c2e617f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

Report any deadlock

Summarize thread quantities blocked on monitors, point out a high contention point and lock owner

A summary of the timestamps in the dumps could be helpful