compiler-explorer / infra

Infrastructure to set up the public Compiler Explorer instances and compilers

Compiler Explorer

Is an interactive compiler exploration website. Edit code in C, C++, C#, F#, Rust, Go, D, Haskell, Swift, Pascal, ispc, Python, Java, or any of the other 30+ supported languages components, and see how that code looks after being compiled in real time.

Bug Report · Compiler Request · Feature Request · Language Request · Library Request · Report Vulnerability

Overview

Multiple compilers are supported for each language, many different tools and visualizations are available, and the UI layout is configurable (thanks to GoldenLayout).

Try out at godbolt.org, or run your own local instance. An overview of what the site lets you achieve, why it's useful, and how to use it is available here.

Compiler Explorer follows a Code of Conduct which aims to foster an open and welcoming environment.

Compiler Explorer was started in 2012 to show how C++ constructs are translated to assembly code. It started as a tmux session with vi running in one pane and watch gcc -S foo.cc -o - running in the other.

Since then, it has become a public website serving over 3,000,000 compilations per week.

You can financially support this project on Patreon, GitHub, Paypal, or by buying cool gear on the Compiler Explorer store.

Using Compiler Explorer

FAQ

There is now a FAQ section in the repository wiki. If your question is not present, please contact us as described below, so we can help you. If you find that the FAQ is lacking some important point, please feel free to contribute to it and/or ask us to clarify it.

Videos

Several videos showcase some features of Compiler Explorer:

• Compiler Explorer 2023: What's New?: Presentation for CppNorth 2023.
• Presentation for CppCon 2019 about the project
• Older 2 part series of videos which go into a bit more detail into the more obscure features.
• Just Enough Assembly for Compiler Explorer: Practical introduction to Assembly with a focus on the usage of Compiler Explorer, from CppCon 2021.
• Playlist: Compiler Explorer: A collection of videos discussing Compiler Explorer; using it, installing it, what it's for, etc.

A Road map is available which gives a little insight into the future plans for Compiler Explorer.

Developing

Compiler Explorer is written in TypeScript, on Node.js.

Assuming you have a compatible version of node installed, on Linux simply running make ought to get you up and running with an Explorer running on port 10240 on your local machine: http://localhost:10240/. If this doesn't work for you, please contact us, as we consider it important you can quickly and easily get running. Currently, Compiler Explorer requires node 20 installed, either on the path or at NODE_DIR (an environment variable or make parameter).

Running with make EXTRA_ARGS='--language LANG' will allow you to load LANG exclusively, where LANG is one for the language ids/aliases defined in lib/languages.ts. For example, to only run Compiler Explorer with C++ support, you'd run make EXTRA_ARGS='--language c++'. The Makefile will automatically install all the third-party libraries needed to run; using npm to install server-side and client-side components.

For development, we suggest using make dev to enable some useful features, such as automatic reloading on file changes and shorter startup times.

You can also use npm run dev to run if make dev doesn't work on your machine.

Some languages need extra tools to demangle them, e.g. rust, d, or haskell. Such tools are kept separately in the tools repo.

Configuring compiler explorer is achieved via configuration files in the etc/config directory. Values are key=value. Options in a {type}.local.properties file (where {type} is c++ or similar) override anything in the {type}.defaults.properties file. There is a .gitignore file to ignore *.local.* files, so these won't be checked into git, and you won't find yourself fighting with updated versions when you git pull. For more information see Adding a Compiler.

Check CONTRIBUTING.md for detailed information about how you can contribute to Compiler Explorer, and the docs folder for specific details regarding various things you might want to do, such as how to add new compilers or languages to the site.

Running a local instance

If you want to point it at your own GCC or similar binaries, either edit the etc/config/LANG.defaults.properties or else make a new one with the name LANG.local.properties, substituting LANG as needed. *.local.properties files have the highest priority when loading properties.

If you want to support multiple compilers and languages like godbolt.org, you can use the bin/ce_install install compilers command in the infra project to install all or some of the compilers. Compilers installed in this way can be loaded through the configuration in etc/config/*.amazon.properties. If you need to deploy in a completely offline environment, you may need to remove some parts of the configuration that are pulled from www.godbolt.ms@443.

When running in a corporate setting the URL shortening service can be replaced by an internal one if the default storage driver isn't appropriate for your environment. To do this, add a new module in lib/shortener/myservice.js and set the urlShortenService variable in configuration. This module should export a single function, see the tinyurl module for an example.

RESTful API

There's a simple restful API that can be used to do compiles to asm and to list compilers.

You can find the API documentation here.

Contact us

We run a Compiler Explorer Discord, which is a place to discuss using or developing Compiler Explorer. We also have a presence on the cpplang Slack channel #compiler_explorer and we have a public mailing list.

There's a development channel on the discord, and also a development mailing list.

Feel free to raise an issue on github or email Matt directly for more help.

Official domains

Following are the official domains for Compiler Explorer:

• https://godbolt.org/
• https://godbo.lt/
• https://compiler-explorer.com/

The domains allow arbitrary subdomains, e.g., https://foo.godbolt.org/, which is convenient since each subdomain has an independent local state. Also, language subdomains such as https://rust.compiler-explorer.com/ will load with that language already selected.

Credits

Compiler Explorer is maintained by the awesome people listed in the AUTHORS file.

We would like to thank the contributors listed in the CONTRIBUTORS file, who have helped shape Compiler Explorer.

We would also like to specially thank these people for their contributions to Compiler Explorer:

• Gabriel Devillers (while working for Kalray)
• Johan Engelen
• Joshua Sheard
• Andrew Pardoe

Many amazing sponsors, both individuals and companies, have helped fund and promote Compiler Explorer.

infra's Issues

Zig master compiler does not get any updates after first installation

After the first installation, the master branch is not updated ever again, because the check at https://github.com/mattgodbolt/compiler-explorer-image/blob/master/update_compilers/install_zig_compilers.sh#L11 stops it from re-downloading it.
That's ok for fixed versions (for example, once 0.3.0 releases, we don't want to download it more than necessary), but for master, it should not be happening.

@tiehuis can you modify the installation script to allow for a fresher master to replace the old one? (We're short on time around here, sorry to bother you with such small things!)

install_binaries.sh didn't seem to update

seems like the daily job didn't run install_binaries, and so only when I ran it manually did node get updated (change checked in Jan 7th)

Clang builds dying

https://godbolt.org/admin/builds.html --

Cloning into 'llvm'...
error: RPC failed; curl 56 GnuTLS recv error (-9): A TLS packet with unexpected length was received.
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed

looks like the git clone isn't working...

set up a "root" node to log into the others

and close off the network for the rest of the nodes

Blaze library unusable in online version

Just including the library seems to exceed the compilation time allowed.

#include<blaze/Blaze.h>
int square(int num) {
    return num * num;
}

Compilation output:

Killed - processing time exceeded
Compiler returned: -1

After ~7500ms, tested with clang 6.0 and gcc 8.1

Move building of trunk compilers to EC2

Currently kirk (my home fileserver) has this in its crontab:

[email protected]
  0  0 * * * cronic bash -c "cd /home/matthew/dev/compiler-explorer-image && ./build_and_upload_latest_gcc.sh > /home/matthew/build_logs/gcc"
  0  2 * * * cronic bash -c "cd /home/matthew/dev/compiler-explorer-image && ./build_and_upload_latest_clang.sh > /home/matthew/build_logs/clang"
  0  4 * * * cronic bash -c "cd /home/matthew/dev/compiler-explorer-image && ./build_and_upload_latest_clang_concepts.sh > /home/matthew/build_logs/clang_concepts"
#  0  4 * * * cronic bash -c "cd /home/matthew/dev/compiler-explorer-image && ./build_and_upload_latest_clang_cppx.sh"
  #0  6 * * * cronic bash -c "cd /home/matthew/dev/compiler-explorer-image && make update-compilers"
#  0  8 * * * cronic bash -c "cd /home/matthew/dev/compiler-explorer-image && ./remove_old_compilers.sh"

Some of this is being done on the admin node, but the builds are still done on poor kirk (a 4-core 2.5GHz Yorkfield with 12G of ram, running a bunch of minecraft servers for my kids too...)

Enable ena enhanced networking in ec2 images

Suggested by @cread

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking-ena.html

seems to work on all instance types.

Make docker image public

Clang builds failing?

Recent set of daily builds ran but seemingly didn't upload:

 ls -l 
total 812
-rw-rw-r-- 1 ubuntu ubuntu  57005 Aug 14 00:54 clang
-rw-rw-r-- 1 ubuntu ubuntu  17703 Aug 14 02:02 clang_autonsdmi
-rw-rw-r-- 1 ubuntu ubuntu  16751 Aug 14 01:10 clang_concepts
-rw-rw-r-- 1 ubuntu ubuntu 183983 Aug 14 01:26 clang_cppx
-rw-rw-r-- 1 ubuntu ubuntu  17043 Aug 14 01:44 clang_relocatable

ubuntu@admin-node ~/build_logs> tail -10 clang
Creating clang++
Creating clang-cl
Creating clang-cpp
Creating llvm-strip
Creating llvm-readelf
+ export 'XZ_DEFAULTS=-T 0'
+ XZ_DEFAULTS='-T 0'
+ tar Jcf /root/clang-trunk-20180814.tar.xz --transform 's,^./,./clang-trunk-20180814/,' -C /root/staging .
+ [[ ! -z s3://compiler-explorer/opt/ ]]
+ s3cmd put --rr /root/clang-trunk-20180814.tar.xz s3://compiler-explorer/opt/
ubuntu@admin-node ~/build_logs> tail -10 clang_concepts 
/root/llvm/tools/gold/gold-plugin.cpp:411:3: note: here
   case DS_Warning:
   ^~~~
Creating llvm-ranlib
Creating llvm-lib
Creating llvm-dlltool
Creating clang++
Creating clang-cl
Creating clang-cpp
Creating llvm-readelf

not sure why they got to the end and then didn't upload. seems clang_cppx did

Upgrade docker

Docker > 1.11.0 would be great to fix some issues.

Better network isolation for non-admin nodes

As part of general hardening pre-execution support:

• Update ce tools to use internal IPs
• Update settings to not allocate pubilc IPs for nodes
• Teach ce to either warn/error if not on admin node, or ping-pong via admin node.

Install autojump in the admin instance

Now I can't fathom using the terminal without it. Thank @mattgodbolt!

Can't download clang-3.9.1

Running update_compilers/install_compilers.sh stops when it reaches clang 3.9.1 as it hits a 505 return code.
Relevant console output:

+ compiler=clang-3.9.1.tar.xz
+ fetch https://s3.amazonaws.com/compiler-explorer/opt/clang-3.9.1.tar.xz /opt/compiler-explorer/clang-3.9.1.tar.xz
+ curl -v -L 'https://s3.amazonaws.com/compiler-explorer/opt/clang-3.9.1.tar.xz /opt/compiler-explorer/clang-3.9.1.tar.xz'
*   Trying 52.216.230.85...
* TCP_NODELAY set
* Connected to s3.amazonaws.com (52.216.230.85) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=Washington; L=Seattle; O=Amazon.com Inc.; CN=s3.amazonaws.com
*  start date: Sep 26 00:00:00 2017 GMT
*  expire date: Sep 20 12:00:00 2018 GMT
*  subjectAltName: host "s3.amazonaws.com" matched cert's "s3.amazonaws.com"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert Baltimore CA-2 G2
*  SSL certificate verify ok.
> GET /compiler-explorer/opt/clang-3.9.1.tar.xz /opt/compiler-explorer/clang-3.9.1.tar.xz HTTP/1.1
> Host: s3.amazonaws.com
> User-Agent: curl/7.55.1
> Accept: */*
> 
< HTTP/1.1 505 HTTP Version not supported
< Date: Mon, 06 Nov 17 01:51:32 GMT
< Connection: close
< Transfer-Encoding: chunked
< x-amz-id-2: RLs1FbWTt1ppn9gfPkM9h6olkC0NiVHFM3ldHfORzvwJ2sA3v9L8i80+yI1bgaifOy251jvplRORwku1bGZ3Cv2ChmOJJRZV
< x-amz-request-id: F1EB1DCF9F2F556A
< Content-Type: application/xml
< 
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
<?xml version="1.0" encoding="UTF-8"?><Error><Code>HttpVersionNotSupported</Code><Message>The HTTP version specified is not supported.</Message><RequestId>F1EB1DCF9F2F556A</RequestId><HostId>RLs1FbWTt1ppn9gfPkM9h6olkC0NiVHFM3ldHfORzvwJ2sA3v9L8i80+yI1bgaifOy251jvplRORwku1bGZ3Cv2ChmOJJRZV</HostId></Error>+ tar axf clang-3.9.1.tar.xz
tar: clang-3.9.1.tar.xz: No se puede efectuar open: No existe el archivo o el directorio
tar: Error is not recoverable: exiting now

After some time seems we lose docker IPs

Just checking out the beta site and weirdly it's "down". But all the containers are up:

ubuntu@ip-172-30-1-29:~$ sudo docker ps
CONTAINER ID        IMAGE                                   COMMAND                  CREATED             STATUS              PORTS                      NAMES
a117a2471238        mattgodbolt/compiler-explorer:swift     "/run.sh"                7 days ago          Up 7 days           0.0.0.0:20483->20483/tcp   swift
c889787e9d8d        mattgodbolt/compiler-explorer:haskell   "/run.sh"                7 days ago          Up 7 days           0.0.0.0:20482->20482/tcp   haskell
77f01e26f3bc        mattgodbolt/compiler-explorer:ispc      "/run.sh"                7 days ago          Up 7 days           0.0.0.0:20481->20481/tcp   ispc
5c8c85e22283        mattgodbolt/compiler-explorer:cppx      "/run.sh"                7 days ago          Up 7 days           0.0.0.0:20480->20480/tcp   cppx
9dc62aac9867        mattgodbolt/compiler-explorer:rust      "/run.sh"                7 days ago          Up 7 days           0.0.0.0:10242->10242/tcp   rust
e18b5b235a81        mattgodbolt/compiler-explorer:d         "/run.sh"                7 days ago          Up 7 days           0.0.0.0:10241->10241/tcp   dx
075ddfdce5a1        gliderlabs/logspout                     "/bin/logspout sys..."   7 days ago          Up 7 days           80/tcp                     logspout

Just trying to fetch / using telnet localhost fails with a bad gateway, and in the logs:

2017/11/14 13:45:34 [error] 5#5: *54864 connect() failed (113: No route to host) while connecting to upstream, client: 172.17.0.1, server: gcc.godbolt.org, request: "GET / HTTP/1.1", upstream: "http://172.17.0.10:10240/", host: "gcc.godbolt.org"

interfaces:

ubuntu@ip-172-30-1-29:~$ ifconfig | grep 'inet '
          inet addr:172.17.0.1  Bcast:0.0.0.0  Mask:255.255.0.0
          inet addr:172.30.1.29  Bcast:172.30.1.255  Mask:255.255.255.0
          inet addr:127.0.0.1  Mask:255.0.0.0

Seems we've lost the 172.17.0.* docker bridge interface IPs. There's a ton of docker bridge interfaces but none with IP addresses.

As we've seen issues on the main site, I wonder if something funny's going on here.

Replace auto-upgrade scripts

The hook.py fails terrible with back-to-back release branch checkins. Really it would be better to have hook.py (or something else) build the release tarballs and then have a different process entirely for updating the nodes with a "current" image.

newrelic thing

Jan 01 19:53:27 1226fb711065 unified: { Error: EROFS: read-only file system, open '/compiler-explorer/newrelic_agent.log'
Jan 01 19:53:27 1226fb711065 unified: code: 'EROFS',

due to it running on docker. Might go away if we drop newrelic.

GCC 5.5 and 7.3 fail to build

Have asked on the mailing list for help: https://gcc.gnu.org/ml/gcc-help/2018-01/msg00109.html

Papertrail integration of docker containers is broken

Email authors of branches on success

e.g. @Quuxplusone would love an email when the build for his clang branch completes. Then of course there's the time it updates and actually goes live

RISC-V support

With a crosstool config and an appropriate version of crosstool, we should add risc-v support.

See also compiler-explorer/compiler-explorer#964, compiler-explorer/compiler-explorer#965 and #85

Prevent compile/execute nodes from having write access to /opt

• Mount /opt read-only
  • beta
  • prod
• Only give read-only permissions : https://docs.aws.amazon.com/efs/latest/ug/access-control-managing-permissions.html

ce deploy bugs

Had a catastrophic pain:

$ ce --env prod restart
Running in prod
Confirm restart version Release(1610, release, dist/travis/release/1610.tar.xz, 37396964, 4fff87..ca7df2) in env prod by typing the name of the environment: prod
INFO:ce:Restarting [email protected]...
INFO:ce:Enabling instance protection for [email protected]
INFO:ce:Putting [email protected] into standby
INFO:ce:Waiting for [email protected] to reach autoscale lifecycle 'Standby'...
INFO:ce:...done
INFO:ce:Restarting service on [email protected]
Job for compiler-explorer.service failed because the control process exited with error code. See "systemctl status compiler-explorer.service" and "journalctl -xe" for details.
ERROR:ce:Failed restarting [email protected] - skipping: Command '['ssh', '-o', 'ConnectTimeout=5', '-o', 'UserKnownHostsFile=/dev/null', '-o', 'StrictHostKeyChecking=no', '-o', 'LogLevel=ERROR', '[email protected]', '--', "'sudo'", "'systemctl'", "'restart'", "'compiler-explorer'"]' returned non-zero exit status 1
INFO:ce:Restarting [email protected]...
INFO:ce:Enabling instance protection for [email protected]
INFO:ce:Putting [email protected] into standby
INFO:ce:Waiting for [email protected] to reach autoscale lifecycle 'Standby'...
INFO:ce:...done
INFO:ce:Restarting service on [email protected]
Job for compiler-explorer.service failed because the control process exited with error code. See "systemctl status compiler-explorer.service" and "journalctl -xe" for details.
ERROR:ce:Failed restarting [email protected] - skipping: Command '['ssh', '-o', 'ConnectTimeout=5', '-o', 'UserKnownHostsFile=/dev/null', '-o', 'StrictHostKeyChecking=no', '-o', 'LogLevel=ERROR', '[email protected]', '--', "'sudo'", "'systemctl'", "'restart'", "'compiler-explorer'"]' returned non-zero exit status 1
INFO:ce:Restarting [email protected]...
INFO:ce:Enabling instance protection for [email protected]
INFO:ce:Group 'Gcc Explorer' needs to be adjusted to keep enough nodes
INFO:ce:Putting [email protected] into standby
INFO:ce:Waiting for [email protected] to reach autoscale lifecycle 'Standby'...
INFO:ce:...done
INFO:ce:Restarting service on [email protected]
Job for compiler-explorer.service failed because the control process exited with error code. See "systemctl status compiler-explorer.service" and "journalctl -xe" for details.
ERROR:ce:Failed restarting [email protected] - skipping: Command '['ssh', '-o', 'ConnectTimeout=5', '-o', 'UserKnownHostsFile=/dev/null', '-o', 'StrictHostKeyChecking=no', '-o', 'LogLevel=ERROR', '[email protected]', '--', "'sudo'", "'systemctl'", "'restart'", "'compiler-explorer'"]' returned non-zero exit status 1
INFO:ce:Restarting [email protected]...
INFO:ce:Enabling instance protection for [email protected]
INFO:ce:Group 'Gcc Explorer' needs to be adjusted to keep enough nodes
INFO:ce:Putting [email protected] into standby
INFO:ce:Waiting for [email protected] to reach autoscale lifecycle 'Standby'...
INFO:ce:...done
INFO:ce:Restarting service on [email protected]
Job for compiler-explorer.service failed because the control process exited with error code. See "systemctl status compiler-explorer.service" and "journalctl -xe" for details.
ERROR:ce:Failed restarting [email protected] - skipping: Command '['ssh', '-o', 'ConnectTimeout=5', '-o', 'UserKnownHostsFile=/dev/null', '-o', 'StrictHostKeyChecking=no', '-o', 'LogLevel=ERROR', '[email protected]', '--', "'sudo'", "'systemctl'", "'restart'", "'compiler-explorer'"]' returned non-zero exit status 1
INFO:ce:Putting desired instances for Gcc Explorer back to 2

Looked ok but nearly all nodes were down...and half were in standby, and had scale-in protection on them still. Robustness needed!

Windows compiler install issue

See compiler-explorer/compiler-explorer#1075 - looks like there's another DLL needed

Cloudwatch stats logging doesn't work

Oct 08 07:40:03 godbolt CloudWatch-PutInstanceData: Cannot obtain instance id from EC2 meta-data.

Install newer ICC

"2019 Initial Release" is now out

Find new download source for msp430

The page we got them from http://software-dl.ti.com/ seems to have died, so we need a new source for them

"ops" like script

would be nice to have an ops like script (ce maybe?):

$ ce login prod
 [ list of prod nodes] , pick one to ssh to it

$ ce list prod

list nodes

etc

Rather than farting about with aws cmd line or the aws website.

Same tool could be used to deploy and upgrade nodes.

Trunk Compiles have been broken for a number of days

Looks like compilers have not built since Aug 24. Also seems.deploys are failing with tar xz erroring out somewhere along the line.

Move site to running on non-privileged port

Script to anonymise old papertrail logs

compiler-explorer.com down

Probably related to changes in the terraforming stuff.

Daily GCC build broken

Daily build is broken. Seems the gcc-go part doesn't like either multipath or multilib.

EFS too slow

So, EFS is too slow after all. During startup the huge rush of compiler --version is enough to cause >7.5s delay in running the compilers. Once they're cached, it's much better. For now we're working around things with no timeouts for --version etc, but that's pretty nasty.

I experimented in beta with a 100GB EBS drive. Things are much better using that, understandably. Syncing /opt/ to the drive took two hours, but after that, accesses are super fast.

Clearly we can't wait two hours for a node to come up. But we can use snapshotting to keep a base image up to date and then generate EBS drives from that, or something. Running a periodic rsync to keep the local drive up to date with the EFS drive is a possibility too.

Needs further thought, and would take a fair amount of effort.

Split s3 into free and non-free

make update_compilers work by default for the (publically-available) public https: endpoint to S3, with only the non-free compilers in the other section.

Basically, make it so anyone can clone and run install_compilers.sh and get all the "free" CE compilers in /opt/compiler-explorer

m68k support

(Cross linking from compiler-explorer/compiler-explorer#271)

Hi,

So after spending a bit of time fixing up my build script for m68k compilers I thought I'd take a look and see how it could be added to this repository. I have a few initial questions:

1. From what I've seen from the other cross compiler build scripts (like for example gcc-cross\build\msp430-gcc453) only the compiler is being built without libraries. My script also builds quite a few libs - are those not needed? I'm asking this because if only the barebones compilers are required then the build script gets significantly smaller.
2. I also asked this in the compiler-explorer issue - from what I understand the compilers are being installed in /opt - what would an accepted naming scheme be? @RabsRincon suggested gcc-x-x-x-68k, by default the compilers are named m68k-x-x-x-elf fwiw.
3. Can anyone help me figuring out how docker works? For my build I require a few versions of gcc to be present (4 through 7), the usual GMP/MPFR/MPC and a couple of others like flex and bison (if the libraries are being built). From what I understand there's a repository of docker packages one can get and use the built-in tools but this is just dark arts for me right now :)

Feedback welcome!
George

build_compiler.py failure

-- Installing: /root/staging/lib/cmake/llvm/./TableGen.cmake
-- Installing: /root/staging/lib/cmake/llvm/./GetSVN.cmake
-- Installing: /root/staging/lib/cmake/llvm/./HandleLLVMOptions.cmake
-- Installing: /root/staging/lib/cmake/llvm/./DetermineGCCCompatible.cmake
-- Installing: /root/staging/lib/cmake/llvm/./HandleLLVMStdlib.cmake
-- Installing: /root/staging/lib/cmake/llvm/./CheckAtomic.cmake
-- Installing: /root/staging/lib/cmake/llvm/./CheckLinkerFlag.cmake
-- Installing: /root/staging/lib/cmake/llvm/./AddSphinxTarget.cmake
-- Installing: /root/staging/lib/cmake/llvm/./GenerateVersionFromCVS.cmake
ERROR: /root/.s3cfg: None
ERROR: Configuration file not available.
ERROR: Consider using --configure parameter to create one.
2018-06-04 09:50:24.434889 : Command returned exit code 78
2018-06-04 09:50:24.434953 : Shutting down instance
2018-06-04 09:50:26.254858 : Command returned exit code -1
2018-06-04 09:50:36.262643 : NOT terminating instance as build failed. Stopping instead
Traceback (most recent call last):
  File "./build_compiler.py", line 165, in <module>
    instance.stop()
  File "/usr/local/lib/python2.7/dist-packages/boto3/resources/factory.py", line 520, in do_action
    response = action(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/boto3/resources/action.py", line 83, in __call__
    response = getattr(parent.meta.client, operation_name)(**params)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 253, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 557, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (UnsupportedOperation) when calling the StopInstances operation: The instance 'i-07f6222c1325d835b' is a spot instance and may not be stopped.

Any chance at pushing a non Amazon Image to dockerhub?

The last few days I've been having some fun with gcc explorer and thought I'd give a go at building a local docker image when I saw this repo, but it didn't go so well because of the s3cfg stuff. Perhaps I should just build from the main repo in a dockerfile?

Intel license has expired

As above, for icc16 and icc17

Would be nice to rid myself of gcc1204 container

Have slowly been moving compilers off of the old gcc1204 docker image. Killing that image would be so nice. Remaining:

• GCC 4.4.7 - doesn't compile out of the box with my scripts; missing download_prerequisites
• GCC 4.5.3 - as above
• ARM GCC 4.5.3 - scripts don't support cross compilers
• ARM GCC 4.6.3 - as above
• AVR 4.5.3 - as above
• MSP430 4.5.3 - as above

Unify all containers

Get rid of any extraneous containers

Can't list compilers on remote site

Permissions on s3 are broken again:

+ S3URL=https://s3.amazonaws.com/compiler-explorer/opt
++ python //list_compilers.py --s3url https://s3.amazonaws.com/compiler-explorer --prefix opt/
Traceback (most recent call last):
  File "//list_compilers.py", line 21, in <module>
    sys.exit(main(parser.parse_args()))
  File "//list_compilers.py", line 10, in main
    result = urlopen(args.s3url + "?prefix=" + args.prefix).read()
urllib2.HTTPError: HTTP Error 403: Forbidden

clang should build with compiler-rt

in order to get the sanitizers. See compiler-explorer/compiler-explorer#920

Single-header, header only libs

Creating this for collecting requests.
There's quite an extensive list here.
Also of course nothings/stb.

s3 cleanup command

s3 gets very full very quickly. Should have an s3 cleanup that deletes not-in-use images, keeping only the last X versions.

Needs to scan all environments' current deployed version, not just the current env's.

(Related, perhaps it should be an error to run ce with a --env for non-env-specific commands like admin and s3-cleanup (or whatever we call this).

Fix crosstool-ng builds

Since the move to 18.04 the crosstool-ng docker image won't build. Plus we (probably) need to upgrade crosstool-ng too.

papertrail log can be hijacked

Anyone configuring the papertrail config the same way as us can spam our logs. Not quite sure anyone would do this...but it just happened from a "stratoserver.net" machine. So we should think of a better way...

No way to distinguish compiles by language

Existing metrics went off the container name. Now we need to bust the URL somehow to find each language, or have some otehr way to know. Or drop the metrics.

GCC trunk not updating / NFS problems

https://godbolt.org/z/0gRVSa indicates the latest build of GCC trunk was 20181002, but https://godbolt.org/admin/builds.html suggests 20181006 should be available.

clang-8 not executable from trunk install

(may be a clang issue, but):

ubuntu@admin-node ~/parallel_studio_xe_2019_composer_edition_online> ls -l /opt/compiler-explorer/clang-trunk/bin/clang*
lrwxrwxrwx 1 root root        7 Oct 25 02:27 /opt/compiler-explorer/clang-trunk/bin/clang -> clang-8*
-rwx------ 1 root root 56497152 Oct 25 03:07 /opt/compiler-explorer/clang-trunk/bin/clang-8*
-rwxr-xr-x 1 root root 81668328 Oct 25 02:27 /opt/compiler-explorer/clang-trunk/bin/clang-check*
-rwxr-xr-x 1 root root 30152768 Oct 25 02:26 /opt/compiler-explorer/clang-trunk/bin/clang-import-test*
-rwxr-xr-x 1 root root 24903800 Oct 25 02:25 /opt/compiler-explorer/clang-trunk/bin/clang-include-fixer*
-rwxr-xr-x 1 root root  3738352 Oct 25 02:20 /opt/compiler-explorer/clang-trunk/bin/clang-offload-bundler*
-rwxr-xr-x 1 root root 24948688 Oct 25 02:24 /opt/compiler-explorer/clang-trunk/bin/clang-rename*
-rwxr-xr-x 1 root root 54242136 Oct 25 02:27 /opt/compiler-explorer/clang-trunk/bin/clang-tidy*

manually chmodding for now, but needs a look.

ce puts instance in standby even if no healthy instances remain

If there's just one instance around in (say) live, then things go wrong:

ubuntu@ip-172-30-0-97 ~> ce --env prod restart
Running in prod
Confirm restart version Release(1982, master, dist/travis/master/1982.tar.xz, 40276304, 3851d6..0247b4) in env prod
Type the name of the environment: prod
INFO:ce:Restarting [email protected]...
INFO:ce:Enabling instance protection for [email protected]
INFO:ce:Group 'Gcc Explorer' needs to be adjusted to keep enough nodes
INFO:ce:Putting [email protected] into standby
INFO:ce:Waiting for [email protected] to reach autoscale lifecycle 'Standby'...
^TINFO:ce:...done
INFO:ce:Restarting service on [email protected]
INFO:ce:Moving [email protected] out of standby
INFO:ce:Waiting for [email protected] to reach autoscale lifecycle 'InService'...
INFO:ce:...done
INFO:ce:Waiting for [email protected] to reach ELB state 'healthy'...
ssh: connect to host 54.172.93.75 port 22: Connection refused
WARNING:ssh:Failed to execute on remote host: Command '['ssh', '-o', 'ConnectTimeout=5', '-o', 'UserKnownHostsFile=/dev/null', '-o', 'StrictHostKeyChecking=no', '-o', 'LogLevel=ERROR', '[email protected]', '--', "'sudo'", "'systemctl'", "'show'", "'compiler-explorer'"]' returned non-zero exit status 255
ERROR:ce:Failed restarting [email protected] - skipping: Instance no longer running (state shutting-down)
INFO:ce:Putting desired instances for Gcc Explorer back to 1

Main thing is putting the one good instance into standby before the node count reaches two healthy.