com.alipay.remoting.rpc.exception.InvokeTimeoutException: Rpc invocation timeout[responseCommand TIMEOUT] about sofa-bolt

[iohao]

比较奇怪的是少量请求次数是可以的；
请求次数过多，就会出现超时；
可以帮忙分析一下原因吗；

控制台

19:36:45.387 [Bolt-default-executor-10-thread-14] ERROR b.p.BoltServerUserIdRequestSyncProcessor.handleRequest (BoltServerUserIdRequestSyncProcessor.java:57) - Rpc invocation timeout[responseCommand TIMEOUT]! the address is 127.0.0.1:63558
com.alipay.remoting.rpc.exception.InvokeTimeoutException: Rpc invocation timeout[responseCommand TIMEOUT]! the address is 127.0.0.1:63558
	at com.alipay.remoting.rpc.RpcResponseResolver.preProcess(RpcResponseResolver.java:83)
	at com.alipay.remoting.rpc.RpcResponseResolver.resolveResponseObject(RpcResponseResolver.java:54)
	at com.alipay.remoting.rpc.RpcRemoting.invokeSync(RpcRemoting.java:186)
	at com.alipay.remoting.rpc.RpcServerRemoting.invokeSync(RpcServerRemoting.java:67)
	at com.alipay.remoting.rpc.RpcRemoting.invokeSync(RpcRemoting.java:143)
	at com.alipay.remoting.rpc.RpcServer.invokeSync(RpcServer.java:638)
	at com.my.bolt.processor.BoltServerUserIdRequestSyncProcessor.handleRequest(BoltServerUserIdRequestSyncProcessor.java:46)
	at com.my.bolt.processor.BoltServerUserIdRequestSyncProcessor.handleRequest(BoltServerUserIdRequestSyncProcessor.java:32)
	at com.alipay.remoting.rpc.protocol.RpcRequestProcessor.dispatchToUserProcessor(RpcRequestProcessor.java:252)
	at com.alipay.remoting.rpc.protocol.RpcRequestProcessor.doProcess(RpcRequestProcessor.java:146)
	at com.alipay.remoting.rpc.protocol.RpcRequestProcessor$ProcessTask.run(RpcRequestProcessor.java:393)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

common-default.log

2022-11-03 19:36:29,356 WARN  [AbstractConnectionFactory#259] [Thread-3] [client side] bolt netty low water mark is 32768 bytes, high water mark is 65536 bytes
2022-11-03 19:36:29,356 WARN  [AbstractConnectionFactory#259] [Thread-4] [client side] bolt netty low water mark is 32768 bytes, high water mark is 65536 bytes
2022-11-03 19:36:29,377 WARN  [AbstractRemotingServer#95] [Thread-2] Prepare to start server on port 50056 
2022-11-03 19:36:29,430 WARN  [AbstractRemotingServer#97] [Thread-2] Server started on port 50056
2022-11-03 19:36:35,359 WARN  [BaseRemoting#113] [Bolt-default-executor-10-thread-8] Wait response, request id=105 timeout!
2022-11-03 19:36:35,359 WARN  [BaseRemoting#113] [Bolt-default-executor-10-thread-3] Wait response, request id=111 timeout!
2022-11-03 19:36:35,359 WARN  [BaseRemoting#113] [Bolt-default-executor-10-thread-5] Wait response, request id=133 timeout!
2022-11-03 19:36:35,359 WARN  [BaseRemoting#113] [Bolt-default-executor-10-thread-17] Wait response, request id=104 timeout!
2022-11-03 19:36:35,359 WARN  [BaseRemoting#113] [Bolt-default-executor-10-thread-9] Wait response, request id=136 timeout!
2022-11-03 19:36:35,359 WARN  [BaseRemoting#113] [Bolt-default-executor-10-thread-19] Wait response, request id=141 timeout!
2022-11-03 19:36:35,359 WARN  [BaseRemoting#113] [Bolt-default-executor-10-thread-11] Wait response, request id=135 timeout!
2022-11-03 19:36:35,359 WARN  [BaseRemoting#113] [Bolt-default-executor-10-thread-7] Wait response, request id=138 timeout!
2022-11-03 19:36:35,359 WARN  [BaseRemoting#113] [Bolt-default-executor-10-thread-16] Wait response, request id=49 timeout!

common-error.log

2022-11-03 19:36:35,371 ERROR [RpcRequestProcessor#246] [Bolt-default-executor-10-thread-6] AYSNC process rpc request failed in RpcRequestProcessor, id=54
java.lang.RuntimeException: com.alipay.remoting.rpc.exception.InvokeTimeoutException: Rpc invocation timeout[responseCommand TIMEOUT]! the address is 127.0.0.1:50056
	at com.my.bolt.client.MyBoltClient.invokeSync(MyBoltClient.java:66)
	at com.my.bolt.processor.InternalClientMyRequestProcessor.handleRequest(InternalClientMyRequestProcessor.java:55)
	at com.my.bolt.processor.InternalClientMyRequestProcessor.handleRequest(InternalClientMyRequestProcessor.java:34)
	at com.alipay.remoting.rpc.protocol.RpcRequestProcessor.dispatchToUserProcessor(RpcRequestProcessor.java:235)
	at com.alipay.remoting.rpc.protocol.RpcRequestProcessor.doProcess(RpcRequestProcessor.java:146)
	at com.alipay.remoting.rpc.protocol.RpcRequestProcessor$ProcessTask.run(RpcRequestProcessor.java:393)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Caused by: com.alipay.remoting.rpc.exception.InvokeTimeoutException: Rpc invocation timeout[responseCommand TIMEOUT]! the address is 127.0.0.1:50056
	at com.alipay.remoting.rpc.RpcResponseResolver.preProcess(RpcResponseResolver.java:83)
	at com.alipay.remoting.rpc.RpcResponseResolver.resolveResponseObject(RpcResponseResolver.java:54)
	at com.alipay.remoting.rpc.RpcRemoting.invokeSync(RpcRemoting.java:186)
	at com.alipay.remoting.rpc.RpcClientRemoting.invokeSync(RpcClientRemoting.java:72)
	at com.alipay.remoting.rpc.RpcRemoting.invokeSync(RpcRemoting.java:143)
	at com.alipay.remoting.rpc.RpcClient.invokeSync(RpcClient.java:219)
	at com.my.bolt.client.MyBoltClient.invokeSync(MyBoltClient.java:64)
	... 8 common frames omitted

[iohao]

此问题会在发送端发送消息密集时触发，大约连续发送 50 条

发送端伪代码如下，往 boltClient 发送

        byte[] bytes = ...；
        for (int i = 0; i < 50; i++) {
            webSocketClient.send(bytes);
        }

[JervyShi]

You could override XXServerProcessor#getExecutor() to set the user thread pool. It can separate IO ThreadPool and User ThreadPool. Then your server can handle more requests in the same time.

[iohao]

好的，感谢！