The iperf3 server hangs while printing the report about iperf

[davidBar-On]

@RizziMau, it may be that the problem is when the server is waiting for the "Done" from the client. However, since the client already terminated, the control socket should not be available already between the processes and the server should have failed / timeout. Therefore, it is not clear why the server did not end.

In any case, few questions:

1. Just to make sure. The failed log is reverse test (-R) and the success log non-reverse. Is this because that all the reverse tests have this error?
2. Again, to make sure, is the client version also 3.16?
3. Can you recreate a failed log (using --debug=3), but now try running the client few time when the server is stuck (before terminating it). When the client display the the server is busy running a test message, the server should display successfully sent ACCESS_DENIED to an unsolicited connection request during active test? Does the server display thiese messages? (Running the client few times is to make sure that some of the server messages will be displayed, as they are not flushed.
4. When the test is running there should be 5 server threads - the main thread and one for each of the 4 streams. When the server got stuck:
   • Are all the 5 threads still running?
   • How much CPU each of them consumes?
5. Are you able to build iperf3 executable for the server (Linux)? I am asking in case it will be useful to create special versions that may help the analysis (usually with more debug messages).

[davidBar-On]

@RizziMau, I believe I understand what is the problem ("DONE" state sent by the client is not received by the server and the server is waiting for it forever). I have a proposed fix for the problem, using the --rcv-timeout value as the timeout for waiting. Before I submit a PR for the fix, it would be very helpful if you can test at least the server side of it, to confirm that indeed this is a fix for the problem.

Can build and run iperf3 for the server (at least) from branch "issue-1735-timeout-select-when-not-in-running-state" in "https://github.com/davidBar-On/iperf.git" (git clone https://github.com/davidBar-On/iperf.git -b issue-1735-timeout-select-when-not-in-running-state)?

[RizziMau]

@davidBar-On, I'm testing your iperf3 version, I will keep you update.

Just a remark:
for this issue it's better to use the --rcv-timeout parameter, or it's better to add a new parameter, e.g. --done-timeout or --close-timeout?

Iperf3 has different timeouts for different aspects:

• --idle-timeout to restart iperf server when in idle
• --connect-timeout for establishing the control connection
• --rcv-timeout for data reception during data transfers

I think it would be preferable to use a specific timeout parameter for the closure of the control connection.

[davidBar-On]

@RizziMau, thanks for testing the change. It would help validating it in general (and of course finding if it solves your issue).

Regarding the use of --rcv-timeout. The iperf3 teams does not like to add new options. Therefore, for having a better chance that the change will be merged into mainline, I try to reuse existing options when possible.

Currently the --rcv-timeout value is used only when test data is sent (TEST_RUNNING state). Usually, its value is probably related to the length of "network stuck" periods (except for very low test bandwidth, e.g. sending a packet once in every 10 seconds). Therefore, I believe that this value can also be used for the timeout of receiving control messages when test data is not sent.

Note that regardless of the option name, the change implements timeout for most of the state-change control messages. I believe it solves a general issue in iperf3 that server/client are getting stuck if a control message is not received (your issue is one example).