Performance tests broken (always paste a result from 3rd Jan, actual run fails) about swift-nio

[FranzBusch]

cc @yim-lee can you help us here. The performance tests seem to broken.

[weissi]

CC @tomerd

[gmilos]

I just noticed that my manual tests occasionally failed. I reproduced it by running it in tight loop and eventually tracked it down to an apparent race in TCPThroughtputBenchmark:

self.context.writeAndFlush is called here[1], before context is set here[2].

This may be the reason we don't get perf test results. Not the fact that perf testing infra is broken exactly.

[1]

swift-nio/Sources/NIOPerformanceTester/TCPThroughputBenchmark.swift

Line 49 in 82fd942

_ = self.context.writeAndFlush(self.wrapOutboundOut(message.slice()))

[2]

swift-nio/Sources/NIOPerformanceTester/TCPThroughputBenchmark.swift

Line 44 in 82fd942

self.context = context

[weissi]

CC @FranzBusch for flakey test (we should track that separately)

@gmilos good catch! However whatever happens in the performance tests should not make the harness post wrong and outdated data. Also, it's posting is so quickly that I doubt it ran anything at all

[gmilos]

Agreed re the harness being immune to failures in the test binary.

For reference, I filed the issue about the specific test here: #2649
//cc @FranzBusch

[yim-lee]

It posts bogus results because the build doesn't get far enough to the cleanup stage to delete previous test result files, and the job is configured to post contents from the report file. I manually wiped the workspace this time. Updating the job to wipe workspace before build didn't help because it fails very early on (some Git operation), which I will look into now.

[yim-lee]

@tomerd fixed the issue (thanks!) and perf job is green again: #2652