Is this a threading issue? about bacalhau

[binocarlos]

Ahhh this old chestnut :-)

So - me and Luke lost a day to this also, it turns out that attempting to write directly to os.Std{out,err} from a go-routine does very strange things with locking the process - we came across this when trying to close devstack after a test was running

Fixed this issue by removing TeeCommand and just printing the std{out,err} buffers - the command is now called RunCommandWithBuffer

It would be nice to know exactly why directly writing to os.Std{err,out} causes these deadlocks but let's bump that to a "deep dive to understand go concurrent stream writing" issue

[binocarlos]

So - it turns out that there is a more fundamental problem here: golang/go#23019

In summary:

• exec a command
• it spawns sub-commands which spawns sub-commands (bash job.sh -> grand-children)
• stdout / stderr pipes from grand-children are linked to main go process
• kill the top level child process

This will cause the deadlock because the grand-child is still thinking it's connected to the stdout/stderr pipes

The solution it seems is to always write to temporary files instead of os.Std{out,err} - this does not trigger the blocking code path....

So - let's update utils.go to handle running commands in this way so we don't get bitten by this again

[lukemarsden]

This is how the Android team worked around the problem: https://go-review.googlesource.com/c/go/+/42271/3/misc/android/go_android_exec.go

We are doing something similar.

[aronchick]

Fixed with this - a60f080