Blocked thread in RedisShardBackplane about bazel-buildfarm

[DarkMatterV]

because I am not familiar with java, I asked chatgpt about RedisShardBackplane getStorageWorkers，and i want to know if it's right，this is it‘s answer：

Based on the error log, it seems that there is a thread blocking issue occurring within the synchronized block of the getStorageWorkers() method. Specifically, the thread "grpc-default-executor-7603" is being blocked while waiting to acquire the lock on the RedisShardBackplane object.

To address this issue, you can consider the following:

Reduce the execution time of the synchronized block: Ensure that operations within the synchronized block are completed quickly. Avoid performing time-consuming operations within the synchronized block, especially avoid calling methods that may cause blocking.

Avoid calling potentially blocking methods within the synchronized block: The call to client.call(this::fetchAndExpireStorageWorkers) within the synchronized block may cause blocking. You may consider moving this call outside the synchronized block or executing it asynchronously to prevent blocking other threads.

Optimize synchronization of data structures: While the storageWorkerSet is being operated on within the synchronized block to prevent concurrent modifications, there may still be issues with concurrent modification. You can use thread-safe data structures or implement finer-grained synchronization when modifying the collection.

Based on these suggestions, you can try modifying the code as follows:

@SuppressWarnings("ConstantConditions")
@Override
public synchronized Set<String> getStorageWorkers() throws IOException {
    if (storageWorkersDeadline == null || storageWorkersDeadline.isExpired()) {
        Set<String> newWorkerSet = client.call(this::fetchAndExpireStorageWorkers);
        synchronized (storageWorkerSet) {
            storageWorkerSet.clear();
            storageWorkerSet.addAll(newWorkerSet);
        }
        storageWorkersDeadline = Deadline.after(workerSetMaxAge, SECONDS);
    }
    synchronized (storageWorkerSet) {
        return new HashSet<>(storageWorkerSet);
    }
}

In this modification, the call to client.call(this::fetchAndExpireStorageWorkers) has been moved outside the synchronized block to avoid potential blocking issues within the synchronized block. Additionally, synchronization has been added when returning the storageWorkerSet to ensure that there are no concurrent modification issues when returning the collection.

[DarkMatterV]

It's similar to issue 1426: #1426

[werkt]

You're running at least 6 month old code, as ShardInstance.java no longer exists. Please update and see if this hang reproduces.

There are no longer any synchronizations in getStorageWorkers that could cause this.

[DarkMatterV]

You're running at least 6 month old code, as ShardInstance.java no longer exists. Please update and see if this hang reproduces.您正在运行至少 6 个月前的代码，因为 ShardInstance.java 已不存在。请更新并查看此挂起是否会重现。

There are no longer any synchronizations in getStorageWorkers that could cause this.getStorageWorkers 中不再有任何可能导致此问题的同步。

Okay, thanks. I'll upgrade and see how it goes.

[werkt]

Going to assume this has been corrected. Closing for now, feel free to reopen if there's anything to add here.