GCC trunk not updating / NFS problems about infra

[mattgodbolt]

Thanks @partouf - as best as I can tell the piping to tar is slow because writing to the EFS drive it backs on is slow.

I wonder if sometimes the untarring is so slow that it backpressures curl into timing out (or the upstream server times us out!). I hadn't thought of that...but it would make the most sense! Perhaps all the installs should be done locally then rsync'd (or similar) onto /opt/compiler-explorer/...

Good thinking!

[RubenRBS]

Maybe something to do with the permissions issue we had a couple of days ago?

[partouf]

or the rust bash error

[mattgodbolt]

Yes, could be something like that! Just spotted it and thought I'd file to remind us, haven't had a chance to investigate

…

On Sat, 6 Oct 2018, 15:34 Patrick Quist, ***@***.***> wrote: or the rust bash error — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#117 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAmsdX9peAfmTGx8mH3H7PRkZY2jnxctks5uiRPfgaJpZM4XLho4> .

[mattgodbolt]

We're seeing weird EFS/NFS problems too now :/

[mattgodbolt]

Most recently:

Cronic detected failure or error output for the command: bash -c sudo /home/ubuntu/compiler-explorer-image/update_compilers/install_compilers.sh nightly RESULT CODE: 1 ERROR OUTPUT: rm: cannot remove '/opt/compiler-explorer/rust-nightly/lib/rustlib/mips64-unknown-linux-gnuabi64/lib/libparking_lot_core-29d209a4a1c65bcb.rlib': Stale file handle

The admin node has been complaining for months that it needs a reboot...I'm going to do that now.

[partouf]

Strange events continue with the nightly's. Not only has the Clang build failed according to the buildlogs through a peculiar error;

Status: Downloaded newer image for mattgodbolt/clang-builder:latest
+ mkdir -p /opt/compiler-explorer
+ pushd /opt/compiler-explorer
+ curl -sL https://s3.amazonaws.com/compiler-explorer/opt/gcc-7.2.0.tar.xz
+ tar Jxf -
xz: (stdin): Unexpected end of input
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

The admin node has some issues untarring the downloaded trunk builds, without error.
I managed to provide a workaround by temporarily saving the files first before untarring, that seem to have worked, but that doesn't seem like a good solution.

The downloads of the trunk xz files seem to go pretty fast on the admin node, but somehow the piping to tar slows down the process to infinity. Could there some kind of weird memory/paging issue going on?

[mattgodbolt]

Closing as this seems...er...better, I guess?