f0rmiga / gcc-toolchain

A fully-hermetic Bazel GCC toolchain for Linux.

Bazel GCC toolchain

This is a fully-hermetic Bazel GCC toolchain for Linux. It supports the glibc variants of https://toolchains.bootlin.com. You can find the documentation under docs.

Why would someone want or need a hermetic toolchain?

Reproducibility and portability.

Developers want their code to compile correctly, be reproducible on CI systems and other developers machines. The way C++ toolchains function usually rely on the system libraries to work.

Reproducibility

If you don't have the exact same headers and libraries, the output produced by Bazel will differ from systems. This is particularly expensive with C++ as caches cannot be shared between machines. More importantly, bugs may exist in one machine but not in others, making the development lifecycle and build system unreliable.

Portability

When it comes to building portable ELF binaries, the libc plays an important role. The linker will try to link against new symbols, and since the GNU libc has symbol versioning, linking against a newer glibc will prevent that program from running on a system using an older glibc. To solve this, we ship glibc 2.26 with the sysroots, which should be old enough by now to make all programs compiled with this toolchain portable.

Use cases

• Your repository contains first-party C/C++/Fortran code; or
• You need to run sanitizers (asan, lsan, tsan, ubsan) on your code; or
• You need to cross-compile from Linux x86_64 to Linux armv7 or aarch64; or
• You want to make your program portable to other Linux distros (our default sysroot ships with glibc 2.26); or
• You want reproducibility.

gcc-toolchain's Issues

[Bug]: libstdc++ is not bundled

What happened?

While the rules make sure to link against a rather old glibc, this is not true for libstdc++, which comes as part of gcc. The default build mode links against libstdc++ dynamically. When running on a host that has a version that is too old, things break. We can of course work around this by linking statically. But if dynamic linking is not reliable from a portability perspective, IMHO it would be cleaner to drop support for dynamically linking libstdc++ completely.

Bazel can handle dynamic libraries well: put the appropriate symlink into the runfiles dir and set RPATH properly, so that the library is found at runtime. I think using that in the gcc-toolchain rules should be possible.

Version

Development (host) and target OS/architectures:
Host: Ubuntu 22.04
Target: Debian Buster
Architecture: both x86_64

Output of bazel --version:
bazel 5.3.1

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:

Language(s) and/or frameworks involved:

How to reproduce

# checkout branch where the C++ examples uses std::filesystem
git clone https://github.com/ahans/gcc-toolchain.git
cd gcc-toolchain
git switch use-filesystem
bazel build //examples/hello_world_cpp

# now try to run in a Debian Buster container
docker run -v $HOME:$HOME --workdir $(pwd) -ti debian:buster bazel-bin/examples/hello_world_cpp/hello_world_cpp
# results in:
# bazel-bin/examples/hello_world_cpp/hello_world_cpp: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by bazel-bin/examples/hello_world_cpp/hello_world_cpp)

# Replacing buster with bullseye makes it work. Bullseye ships with gcc 10.2, so that does support 
# std::filesystem; gcc 8.x from buster does not

Any other information?

No response

Fund our work

• Sponsor our open source work by donating a bug bounty

Rename master branch to main

Moving from aspect-build to f0rmiga broke builds

This move affected the stripped prefix of the automatically generated archive files, which therefore also changes the SHA. End users will need to migrate their builds by updating both the sha256 and strip_prefix fields (and updating urls too couldn't hurt).

Here's an example.

[Bug]: incompatible_sandbox_hermetic_tmp (on by default in bazel 7.0.0) breaks the gcc wrapper

What happened?

incompatible_sandbox_hermetic_tmp causes some relative paths to change, and results in this logic no longer pointing to the correct location (on my machine, it just ends up being /tmp)

This produces an error like:

external/gcc_toolchain_armv7/bin/gcc: line 45: /tmp/external/gcc_toolchain_armv7_files/bin/arm-linux-gcc: No such file or directory

Version

Development (host) and target OS/architectures: linux/amd64

Output of bazel --version: 7.0.0

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:

Language(s) and/or frameworks involved:

How to reproduce

from aspect/gcc-toolchain:

echo 7.0.0 > .bazelversion
bazel build --noenable_bzlmod //...
ERROR: $ROOT/aspect/gcc-toolchain/examples/hello_world_cpp/BUILD.bazel:19:10: Compiling examples/hello_world_cpp/main.cpp failed: (Exit 127): gcc failed: error executing CppCompile command (from target //examples/hello_world_cpp:hello_world_cpp) external/gcc_toolchain_armv7/bin/gcc -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -MD -MF ... (remaining 32 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
external/gcc_toolchain_armv7/bin/gcc: line 45: /tmp/external/gcc_toolchain_armv7_files/bin/arm-linux-gcc: No such file or directory

Any other information?

What seems to work is to pass the previously calculated execroot into _render_tool_paths and then further into the wrapper via a new __EXECROOT__ substitution but i'm not sure if that's an approach that would backfire in some way.

Why build sysroot from source code

I'm just wondering if there are any concerns about building all packages from source code when constructing the sysroot. It would be significantly faster to directly use apt-get for the required packages and then extract them from /lib and /usr/lib directories. In terms of reproducibility, we can specify the versions of these packages during the apt-get process. To support different architectures, we can use QEMU as an emulator.

macOS/Windows support

Hello!

I was just wondering if you had any plans on supporting macOS and/or Windows in the future?

[Bug]: `gcc_toolchain` parameters like `includes` replaces all items instead of being extras as documented

What happened?

Tried registering toolchain with includes to add something like /usr/include/linux so <stddef.h> could be found. Using the includes parameter did include that directory for includes, but removed all the defaults.

Version

Development (host) and target OS/architectures:

host: linux/amd64
targets: linux/amd64, linux/aarch64

Output of bazel --version: bazel 6.1.1

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:

workspace(name = "my-code")

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")

http_archive(
    name = "rules_cc",
    sha256 = "3d9e271e2876ba42e114c9b9bc51454e379cbf0ec9ef9d40e2ae4cec61a31b40",
    strip_prefix = "rules_cc-0.0.6",
    urls = ["https://github.com/bazelbuild/rules_cc/releases/download/0.0.6/rules_cc-0.0.6.tar.gz"],
)

http_archive(
    name = "rules_pkg",
    sha256 = "8c20f74bca25d2d442b327ae26768c02cf3c99e93fad0381f32be9aab1967675",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/rules_pkg/releases/download/0.8.1/rules_pkg-0.8.1.tar.gz",
        "https://github.com/bazelbuild/rules_pkg/releases/download/0.8.1/rules_pkg-0.8.1.tar.gz",
    ],
)

load("@rules_pkg//:deps.bzl", "rules_pkg_dependencies")

rules_pkg_dependencies()

git_repository(
    name = "aspect_gcc_toolchain",
    commit = "4bd1f94536ee92b7c49673931773038d923ee86e",
    remote = "https://github.com/aspect-build/gcc-toolchain",
)

#local_repository(
#    name = "aspect_gcc_toolchain",
#    path = "/home/kris/code/kriswuollett/gcc-toolchain",
#)

load("@aspect_gcc_toolchain//toolchain:repositories.bzl", "gcc_toolchain_dependencies")

gcc_toolchain_dependencies()

load("@aspect_gcc_toolchain//toolchain:defs.bzl", "ARCHS", "gcc_register_toolchain")

gcc_register_toolchain(
    name = "gcc_toolchain_aarch64",
    target_arch = ARCHS.aarch64,
    includes = [
        "%sysroot%/usr/include/linux",
    ],
)

gcc_register_toolchain(
    name = "gcc_toolchain_armv7",
    target_arch = ARCHS.armv7,
    includes = [
        "%sysroot%/usr/include/linux",
    ],
)

gcc_register_toolchain(
    name = "gcc_toolchain_x86_64",
    sysroot_variant = "x86_64-X11",
    target_arch = ARCHS.x86_64,
    includes = [
        "%sysroot%/usr/include/linux",
    ],
)

http_archive(
    name = "io_bazel_rules_go",
    sha256 = "dd926a88a564a9246713a9c00b35315f54cbd46b31a26d5d8fb264c07045f05d",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/rules_go/releases/download/v0.38.1/rules_go-v0.38.1.zip",
        "https://github.com/bazelbuild/rules_go/releases/download/v0.38.1/rules_go-v0.38.1.zip",
    ],
)

load("@io_bazel_rules_go//go:deps.bzl", "go_download_sdk", "go_register_toolchains", "go_rules_dependencies")

go_rules_dependencies()

go_download_sdk(
    name = "go_1_20_2_host",
    version = "1.20.2",
)

go_download_sdk(
    name = "go_1_20_2_linux_amd64",
    goarch = "amd64",
    goos = "linux",
    version = "1.20.2",
)

go_download_sdk(
    name = "go_1_20_2_linux_arm64",
    goarch = "arm64",
    goos = "linux",
    version = "1.20.2",
)

go_register_toolchains()

http_archive(
    name = "bazel_gazelle",
    sha256 = "ecba0f04f96b4960a5b250c8e8eeec42281035970aa8852dda73098274d14a1d",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/bazel-gazelle/releases/download/v0.29.0/bazel-gazelle-v0.29.0.tar.gz",
        "https://github.com/bazelbuild/bazel-gazelle/releases/download/v0.29.0/bazel-gazelle-v0.29.0.tar.gz",
    ],
)

load("@bazel_gazelle//:deps.bzl", "gazelle_dependencies")
load("//:go.bzl", "go_dependencies")

# gazelle:repository_macro go.bzl%go_dependencies
go_dependencies()

gazelle_dependencies(go_sdk = "go_1_20_2_host")

http_archive(
    name = "rules_rust",
    sha256 = "b4e622a36904b5dd2d2211e40008fc473421c8b51c9efca746ab2ecf11dca08e",
    urls = ["https://github.com/bazelbuild/rules_rust/releases/download/0.19.1/rules_rust-v0.19.1.tar.gz"],
)

load("@rules_rust//rust:repositories.bzl", "rules_rust_dependencies", "rust_register_toolchains")

rules_rust_dependencies()

rust_register_toolchains(
    edition = "2021",
    versions = [
        "nightly/2023-03-24",
    ],
    extra_target_triples = [
        "aarch64-unknown-linux-gnu",
        "x86_64-unknown-linux-gnu",
        "wasm32-unknown-unknown",
        "wasm32-wasi",
    ],
)


load("@rules_rust//crate_universe:repositories.bzl", "crate_universe_dependencies")

crate_universe_dependencies()

load("@rules_rust//crate_universe:defs.bzl", "crates_repository", "render_config")
load("//bazel:rust.bzl", "RUST_PACKAGES")

crates_repository(
    name = "crate_index",
    cargo_lockfile = "//:Cargo.lock",
    generate_binaries = True,
    lockfile = "//:Cargo.Bazel.lock",
    packages = RUST_PACKAGES,
    render_config = render_config(
        default_package_name = "",
    ),
    rust_version = "nightly/2023-03-05",
)

load("@crate_index//:defs.bzl", "crate_repositories")

crate_repositories()

load("@rules_rust//tools/rust_analyzer:deps.bzl", "rust_analyzer_dependencies")

rust_analyzer_dependencies()

http_archive(
    name = "rules_oci",
    patch_args = ["-p1"],
    patches = ["//bazel:rules_oci_registry.patch"],
    sha256 = "48642588e91e992772b94de06234da6601854fda0ee32a91ce8ef303cf5e5837",
    strip_prefix = "rules_oci-0.3.7",
    url = "https://github.com/bazel-contrib/rules_oci/releases/download/v0.3.7/rules_oci-v0.3.7.tar.gz",
)

load("@rules_oci//oci:dependencies.bzl", "rules_oci_dependencies")

rules_oci_dependencies()

load("@rules_oci//oci:repositories.bzl", "LATEST_CRANE_VERSION", "LATEST_ZOT_VERSION", "oci_register_toolchains")

register_toolchains("//registry:registry_toolchain")

oci_register_toolchains(
    name = "oci",
    crane_version = LATEST_CRANE_VERSION,
    zot_version = LATEST_ZOT_VERSION,
)

load("@rules_oci//oci:pull.bzl", "oci_pull")

oci_pull(
    name = "distroless_base",
    digest = "sha256:5812871f5d87d6d4c226c536be70f7a8232a77230675b4b574c9866c8dc982fa",
    image = "gcr.io/distroless/base",
    platforms = [
        "linux/amd64",
        "linux/arm64",
    ],
)

oci_pull(
    name = "distroless_cc",
    digest = "sha256:f252d3ca44b7f2c718c67c2020feee7b7fd4ee6bff8a192dfea6e599b7d820ad",
    # TODO(https://github.com/bazel-contrib/rules_oci/issues/74) Use latest images
    # digest = "sha256:fb402c45f3ef485ccd56ca2af2a58615fc47c4978bb3004e8663a83456791f48",
    image = "gcr.io/distroless/cc",
    platforms = [
        "linux/amd64",
        "linux/arm64",
    ],
)

Language(s) and/or frameworks involved:

Rust

How to reproduce

load("@aspect_gcc_toolchain//toolchain:defs.bzl", "ARCHS", "gcc_register_toolchain")

gcc_register_toolchain(
    name = "gcc_toolchain_aarch64",
    target_arch = ARCHS.aarch64,
    includes = [
        "%sysroot%/usr/include/linux",
    ],
)

Any other information?

The documentation says that parameters like includes are supposed to be extra values:

https://github.com/aspect-build/gcc-toolchain/blob/4bd1f94536ee92b7c49673931773038d923ee86e/docs/defs.md?plain=1#L25-L28

The invocation of gcc-toolchain uses dict::pop that has the base values as the default which is only used if the key like "includes" is not in the dict.

https://github.com/aspect-build/gcc-toolchain/blob/4bd1f94536ee92b7c49673931773038d923ee86e/toolchain/defs.bzl#L286-L298

So it is not possible to add extra include directories without copy and pasting the directories from the following code:

https://github.com/aspect-build/gcc-toolchain/blob/4bd1f94536ee92b7c49673931773038d923ee86e/toolchain/defs.bzl#L418-L430

Original error without specifying includes:

error occurred: Command "/home/kris/.cache/bazel/_bazel_kris/ebd20a16b5bb6af41a7a481997cc3984/sandbox/linux-sandbox/556/execroot/my-code/external/gcc_toolchain_aarch64/bin/gcc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-gdwarf-4" "-fno-omit-frame-pointer" "--sysroot" "external/sysroot_aarch64" "-no-canonical-prefixes" "-fno-canonical-system-headers" "-Wno-builtin-macro-redefined" "-D__DATE__=\"redacted\"" "-D__TIMESTAMP__=\"redacted\"" "-D__TIME__=\"redacted\"" "-fdiagnostics-color=always" "-nostdinc" "-Bexternal/gcc_toolchain_aarch64/bin" "-isystemexternal/sysroot_aarch64//aarch64-linux/include/c++/10.3.0" "-isystemexternal/sysroot_aarch64//aarch64-linux/include/c++/10.3.0/aarch64-linux" "-isystemexternal/sysroot_aarch64//lib/gcc/aarch64-linux/10.3.0/include-fixed" "-isystemexternal/sysroot_aarch64//lib/gcc/aarch64-linux/10.3.0/include" "-isystemexternal/sysroot_aarch64//usr/include" "-I" "zstd/lib/" "-I" "zstd/lib/common" "-fvisibility=hidden" "-ffunction-sections" "-fdata-sections" "-fmerge-all-constants" "-DZSTD_LIB_DEPRECATED=0" "-DXXH_PRIVATE_API=" "-DZSTDLIB_VISIBILITY=" "-DZSTDERRORLIB_VISIBILITY=" "-o" "/home/kris/.cache/bazel/_bazel_kris/ebd20a16b5bb6af41a7a481997cc3984/sandbox/linux-sandbox/556/execroot/my-code/bazel-out/k8-fastbuild-ST-87f845362927/bin/external/crate_index__zstd-sys-2.0.7-zstd.1.5.4/zstd-sys_build_script.out_dir/zstd/lib/common/ent

Error with specifying includes, the original base system includes are missing:

error occurred: Command "/home/kris/.cache/bazel/_bazel_kris/ebd20a16b5bb6af41a7a481997cc3984/sandbox/linux-sandbox/612/execroot/my-code/external/gcc_toolchain_aarch64/bin/gcc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-gdwarf-4" "-fno-omit-frame-pointer" "--sysroot" "external/sysroot_aarch64" "-no-canonical-prefixes" "-fno-canonical-system-headers" "-Wno-builtin-macro-redefined" "-D__DATE__=\"redacted\"" "-D__TIMESTAMP__=\"redacted\"" "-D__TIME__=\"redacted\"" "-fdiagnostics-color=always" "-nostdinc" "-Bexternal/gcc_toolchain_aarch64/bin" "-isystemexternal/sysroot_aarch64//usr/include/linux" "-I" "zstd/lib/" "-I" "zstd/lib/common" "-fvisibility=hidden" "-ffunction-sections" "-fdata-sections" "-fmerge-all-constants" "-DZSTD_LIB_DEPRECATED=0" "-DXXH_PRIVATE_API=" "-DZSTDLIB_VISIBILITY=" "-DZSTDERRORLIB_VISIBILITY=" "-o" "/home/kris/.cache/bazel/_bazel_kris/ebd20a16b5bb6af41a7a481997cc3984/sandbox/linux-sandbox/612/execroot/my-code/bazel-out/k8-fastbuild-ST-87f845362927/bin/external/crate_index__zstd-sys-2.0.7-zstd.1.5.4/zstd-sys_build_script.out_dir/zstd/lib/common/entropy_common.o" "-c" "zstd/lib/common/entropy_common.c" with args "gcc" did not execute successfully (status code exit status: 1).

Fill in readme and user documentation

Currently it's a mystery what is the value prop, and how to adopt and use this repo.

rules_js and rules_py READMEs are a place to start - and we need high-level documentation as well.

[Bug]: Keeping `--cpu=k8` on macos throws `does not contain a toolchain for cpu 'k8'`. Removing the cpu flag breaks cross-platform caching for remote builds with the same remote_host architecture.

What happened?

Since switching over from the native cpp toolchain to this hermetic one we have noticed that the execution root found under bazel-out changed from k8-fastbuild to darwin_arm64-fastbuild on local macos machines. This may not seem like an issue at the surface, but we also have CI which runs Bazel and in that case the host is still k8.

The effect of this is that CI & local runs no longer share cache hits because the pathing changes are included in cache keys. Additionally our remote workers see a change in the JDK from this pathing and that change causes workers to restart for no reason leading to churn and delays.

We suspect the root cause of this is from needing to remove --cpu=k8 --host_cpu=k8 when switching over to the new cpp toolchain. These flags appear to also impact the bazel-out pathing. Setting these flags with the new toolchain produces this error:

/private/var/tmp/_bazel_ted_tenedorio/544c8a90b96ef2c7562b61f1f903de27/external/local_config_cc/BUILD:28:19: in cc_toolchain_suite rule @local_config_cc//:toolchain: cc_toolchain_suite '@local_config_cc//:toolchain' does not contain a toolchain for cpu 'k8'.

Ideally we could make changes to this toolchain so that those flags can be used again.

Version

Development (host) and target OS/architectures:
Host: Macos - cpu = darwin_arm64
Target: Linux - cpu = k8

Output of bazel --version: 6.0.0

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:
GCC Toolchain: aspect-build/gcc-toolchain/archive/refs/tags/0.4.2.tar.gz
rules_java: bazelbuild/rules_java/releases/download/0.1.1/rules_java-0.1.1.tar.gz
rules_kotlin: bazelbuild/rules_kotlin/releases/download/v1.7.1/rules_kotlin_release.tgz
JDK 11:
https://corretto.aws/downloads/resources/8.275.01.1/amazon-corretto-8.275.01.1-linux-x64.tar.gz
https://corretto.aws/downloads/resources/8.275.01.1/amazon-corretto-8.275.01.1-macosx-x64.tar.gz

Language(s) and/or frameworks involved:
CPP & Java/Kotlin. The issue is impacting all of caching though as it changes the bazel-out relative pathing.

How to reproduce

Run this on a macOS machine to see the bazel-out pathing of concern:
`bazel aquery //path/to/some/java_library/package --config remote`

And observe the `--output` field for the sourcejar:

action 'Building source jar path/to/some/java_library/package/package_lib-sources.jar'
  Mnemonic: JavaSourceJar
  Target: //path/to/some/java_library/package:package_lib
  Configuration: darwin_arm64-fastbuild
  Execution platform: @aspect_gcc_toolchain//platforms:x86_64_linux_remote
  ActionKey: 29ca9f15b7392f2feddf3804b63bae3341fa053f43482c2384bf630347b62682
  Inputs: [path/to/some/java_library/package/PackageSourceFile.kt, external/remote_java_tools_linux/java_tools/src/tools/singlejar/singlejar_local]
  Outputs: [bazel-out/darwin_arm64-fastbuild/bin/path/to/some/java_library/package/package_lib-sources.jar]
  ExecutionInfo: {OSFamily: Linux, container-image: REDACTED}
  Command Line: (exec external/remote_java_tools_linux/java_tools/src/tools/singlejar/singlejar_local \
    --output \
    bazel-out/darwin_arm64-fastbuild/bin/bazel/path/to/some/java_library/package/package_lib-sources.jar \
    --compression \
    --normalize \
    --exclude_build_data \
    --warn_duplicate_resources \
    --resources \

Notice how darwin_arm64-fastbuild is in the root:
bazel-out/darwin_arm64-fastbuild/bin/bazel/path/to/some/java_library/package/package_lib-sources.jar

Running the same command on a Linux machine produces:
bazel-out/k8-fastbuild/bin/bazel/path/to/some/java_library/package/package_lib-sources.jar

You can also run this to see the cpu flag error:
bazel build //path/to/some/java_library/package --config remote --cpu=k8 --host_cpu=k8

The remote config here is essentially:

--java_language_version=8
--java_runtime_version=remotejdk_11
--tool_java_language_version=11
--tool_java_runtime_version=remotejdk_11
--action_env=BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1
--incompatible_strict_action_env=true
--incompatible_enable_cc_toolchain_resolution
--extra_execution_platforms=@aspect_gcc_toolchain//platforms:x86_64_linux_remote
--host_platform=@aspect_gcc_toolchain//platforms:x86_64_linux_remote
--symlink_prefix=dist/
--remote_timeout=3600
--jobs=1024
--remote_local_fallback=false
--compiler_warnings=off
--define=EXECUTOR=remote
--disk_cache=
--remote_download_minimal
--remote_default_exec_properties="container-image=REDACTED"
--remote_default_exec_properties="OSFamily=Linux"
--experimental_remote_mark_tool_inputs=true
--remote_default_exec_properties="dockerReuse=True"
--remote_executor=grpcs:REMOTE_URL
--remote_cache=grpcs:REMOTE_URL

### Any other information?

_No response_

[FR]: can copts come after extra_cxxflags?

What is the current behavior?

We are adding a large set of warning flags with extra_cxxflags and extra_cflags. I just tried to disable a warning locally in a cc_library with with copts = ["-Wno-foo"], and it surprisingly didn't work. I built with bazel build -s and saw that the compiler invocation adds the extra_cxxflags after the flags from copts. This means that one cannot disable any flag from extra_cxxflags locally in a cc_library.

Was it intentional to do it this way or was that just how it was set up?

Describe the feature

If it makes sense, then the feature would be to swap the order of extra_cxxflags and copts. (Same for extra_cflags).

Unable to use gcc-toolchain with configure_make

First of all: Thanks for the great work of enabling an hermetic gcc toolchain in Bazel! It helped us already a lot for our understanding.

Description

At the moment we're trying to integrate your toolchain (with a bumped gcc version) in our build but I am already failing to compile a third party project (which we're bundling/shipping/linking against).

The configure_make step is failing during configure with a lot of errors, where some of them seem to be toolchain related. e.g.:

configure:3724: /home/timotheus/.cache/bazel/_bazel_timotheus/17bdf0994d16870d211e03bc3a9d26dd/sandbox/linux-sandbox/21/execroot/_main/external/gcc_toolchain_x86_64/bin/gcc  -c -Dredacted="redacted"  conftest.c >&5
conftest.c:10:10: fatal error: stdio.h: No such file or directory
   10 | #include <stdio.h>
      |          ^~~~~~~~~
compilation terminated.

This looks to me as configure is expecting gcc to find the system include paths by its own.
But those paths are explicitly registered (as I assume the pre-compiled/relocated gcc won't be able to do that on his own)

        gcc_toolchain_workspace_name = rctx.attr.gcc_toolchain_workspace_name,
        target_compatible_with = str(target_compatible_with),
        toolchain_files_repository_name = rctx.attr.toolchain_files_repository_name,

        # Sysroot
        sysroot = sysroot,

        # Includes
        cxx_builtin_include_directories = str(cxx_builtin_include_directories),

This may all be an issue of the external rrdtool project but you guys may have more insights / tips regarding that?

Repro

The issue should be fully reproducible with: Checkmk@3be3a14

bazel build --sandbox_add_mount_pair=/tmp  --sandbox_debug --verbose_failures --platforms=//platforms:x86_64_linux //examples/rrdtool:rrdtool

WORKSPACE snippet to use gcc-toolchain

It seems a WORKSPACE snippet to set up gcc-toolchain is missing from the current documentation. It would be nice to include the WORKSPACE snippet in documentation or the release page so that users can start using gcc-toolchain without troubles by copying the WORKSPACE snippet into their WORKSPACE files.

Based on this repo's WORKSPACE, below is an example of the WORKSPACE snippet and .bazelrc to use gcc-toolchain v0.4.2 to build binaries for x86_64. It would be great to confirm that there is something wrong or redundant with the snippet. I mentioned this because unusual configs, --action_env=BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1 and --incompatible_enable_cc_toolchain_resolution were required to add to .bazelrc so that Bazel can use the C/C++ toolchain managed by gcc-toolchain. If these configs are required indeed, I think it worth noting it in the documentation because these configs not so obvious for non-Bazel experts.

WORKSPACE:

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

# Skylib
# gcc-toolchain depends on bazel-skylib
http_archive(
    name = "bazel_skylib",
    sha256 = "f7be3474d42aae265405a592bb7da8e171919d74c16f082a5457840f06054728",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/1.2.1/bazel-skylib-1.2.1.tar.gz",
        "https://github.com/bazelbuild/bazel-skylib/releases/download/1.2.1/bazel-skylib-1.2.1.tar.gz",
    ],
)

load("@bazel_skylib//:workspace.bzl", "bazel_skylib_workspace")

bazel_skylib_workspace()

http_archive(
    name = "aspect_gcc_toolchain",
    sha256 = "3341394b1376fb96a87ac3ca01c582f7f18e7dc5e16e8cf40880a31dd7ac0e1e",
    strip_prefix = "gcc-toolchain-0.4.2",
    url = "https://github.com/aspect-build/gcc-toolchain/archive/refs/tags/0.4.2.tar.gz",
)

load("@aspect_gcc_toolchain//toolchain:defs.bzl", "ARCHS", "gcc_register_toolchain")

gcc_register_toolchain(
    name = "gcc_toolchain_x86_64",
    gcc_version = "10.3.0",
    sysroot_variant = "x86_64",
    target_arch = ARCHS.x86_64,
)

.bazelrc:

# Required to use the C/C++ toolchain managed by gcc-toolchain
build --action_env=BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1
build --incompatible_enable_cc_toolchain_resolution

# optional, but preferred for performance as per https://github.com/aspect-build/gcc-toolchain/issues/85
build --experimental_reuse_sandbox_directories

[Bug]: aarch64 target failing

What happened?

Compilation failed when targeting aarch64. See below error:

SUBCOMMAND: # @com_google_absl//absl/strings:cord_internal [action 'Compiling absl/strings/internal/cord_rep_btree_reader.cc', configuration: 176b5d5df169c5e83a68b4875b22e319078ba78a34afd48f69d1f7629630226d, execution platform: @local_config_platform//:host]
(cd /home/gfrankliu/.cache/bazel/_bazel_gfrankliu/48efe75e9cdaaaf5b7f7393802f01694/execroot/com_github_grpc_grpc && \
  exec env - \
    BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1 \
    GRPC_BAZEL_RUNTIME=1 \
    PATH=/home/gfrankliu/.local/bin:/home/gfrankliu/bin:/usr/local/buildtools/java/jdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/gfrankliu/go/bin \
    PWD=/proc/self/cwd \
  external/gcc_toolchain_aarch64/bin/gcc -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -MD -MF bazel-out/k8-fastbuild/bin/external/com_google_absl/absl/strings/_objs/cord_internal/cord_rep_btree_reader.pic.d '-frandom-seed=bazel-out/k8-fastbuild/bin/external/com_google_absl/absl/strings/_objs/cord_internal/cord_rep_btree_reader.pic.o' -fPIC '-DBAZEL_CURRENT_REPOSITORY="com_google_absl"' -iquote external/com_google_absl -iquote bazel-out/k8-fastbuild/bin/external/com_google_absl -Wno-typedef-redefinition -DGRPC_BAZEL_BUILD '-std=c++14' -Wall -Wextra -Wcast-qual -Wconversion-null -Wformat-security -Wmissing-declarations -Woverlength-strings -Wpointer-arith -Wundef -Wunused-local-typedefs -Wunused-result -Wvarargs -Wvla -Wwrite-strings -DNOMINMAX --sysroot external/sysroot_aarch64 -no-canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' '-fdiagnostics-color=always' -nostdinc -nostdinc++ -Bexternal/gcc_toolchain_aarch64/bin -isystemexternal/sysroot_aarch64//aarch64-linux/include/c++/10.3.0 -isystemexternal/sysroot_aarch64//aarch64-linux/include/c++/10.3.0/aarch64-linux -isystemexternal/sysroot_aarch64//lib/gcc/aarch64-linux/10.3.0/include-fixed -isystemexternal/sysroot_aarch64//lib/gcc/aarch64-linux/10.3.0/include -isystemexternal/sysroot_aarch64//usr/include -c external/com_google_absl/absl/strings/internal/cord_rep_btree_reader.cc -o bazel-out/k8-fastbuild/bin/external/com_google_absl/absl/strings/_objs/cord_internal/cord_rep_btree_reader.pic.o)
# Configuration: 176b5d5df169c5e83a68b4875b22e319078ba78a34afd48f69d1f7629630226d
# Execution platform: @local_config_platform//:host
ERROR: /home/gfrankliu/.cache/bazel/_bazel_gfrankliu/48efe75e9cdaaaf5b7f7393802f01694/external/com_google_absl/absl/random/internal/BUILD.bazel:337:11: Compiling absl/random/internal/randen_hwaes.cc failed: (Exit 1): gcc failed: error executing command (from target @com_google_absl//absl/random/internal:randen_hwaes_impl) external/gcc_toolchain_aarch64/bin/gcc -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -MD -MF ... (remaining 50 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
aarch64-linux-gcc.br_real: error: unrecognized command-line option '-maes'
aarch64-linux-gcc.br_real: error: unrecognized command-line option '-msse4.1'
Target //:promise failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 52.470s, Critical Path: 2.08s
INFO: 146 processes: 24 internal, 122 linux-sandbox.
FAILED: Build did NOT complete successfully

Version

Development (host) and target OS/architectures:

Output of bazel --version:

INFO: Running bazel wrapper (see //tools/bazel for details), bazel version 6.3.2 will be used instead of system-wide bazel installation.
bazel 6.3.2

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:

See below "How to reproduce" section for the WORKSPACE

Language(s) and/or frameworks involved:

cc/gprc

How to reproduce

git clone https://github.com/grpc/grpc.git

cd grpc

cat >> WORKSPACE << EOF

# aspect-build
http_archive(
    name = "aspect_gcc_toolchain",
    sha256 = "3341394b1376fb96a87ac3ca01c582f7f18e7dc5e16e8cf40880a31dd7ac0e1e",
    strip_prefix = "gcc-toolchain-0.4.2",
    url = "https://github.com/aspect-build/gcc-toolchain/archive/refs/tags/0.4.2.tar.gz",
)

load("@aspect_gcc_toolchain//toolchain:defs.bzl", "gcc_register_toolchain", "ARCHS")

gcc_register_toolchain(
    name = "gcc_toolchain_aarch64",
    sysroot_variant = "aarch64",
    target_arch = ARCHS.aarch64,
)

gcc_register_toolchain(
    name = "gcc_toolchain_x86_64",
    sysroot_variant = "x86_64",
    target_arch = ARCHS.x86_64,
)
EOF

cat >> .bazelrc << EOF
build --action_env=BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1
build --incompatible_enable_cc_toolchain_resolution
build --reuse_sandbox_directories
EOF

bazel build //:promise -s --platforms=@aspect_gcc_toolchain//platforms:aarch64_linux

### Any other information?

_No response_

[Bug]: Build fails for .s assembly files

What happened?

• gcc-toolchain with GCC 11.3.0 fails to assemble .s files
• However, when renaming the .s file to .S, the build works

Version

Development (host) and target OS/architectures:

Output of bazel --version: bazel 6.0.0-pre.20220909.2

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:

• gcc-toolchain-0.4.2
• GCC 11.3.0 (x86-64-v2)

Language(s) and/or frameworks involved:

• x86_64 assembly

How to reproduce

Build rule


load("//bazel:fd_build_system.bzl", "fd_cc_library", "fd_cc_test")

package(default_visibility = ["//src/ballet:__subpackages__"])

fd_cc_library(
    name = "sha512",
    srcs = [
        "fd_sha512.c",
        "fd_sha512_core_avx2.s",
    ],
    hdrs = [
        "fd_sha512.h",
    ],
    deps = [
        "//src/ballet:base_lib",
    ],
)

When changing fd_sha512_core_avx2.s to fd_sha512_core_avx2.S, the build succeeds.

Any other information?

Build with `.s` file (fail)

Loading: 
Loading: 0 packages loaded
Analyzing: target //src/ballet/sha512:sha512 (0 packages loaded, 0 targets configured)
INFO: Analyzed target //src/ballet/sha512:sha512 (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
[0 / 4] [Prepa] BazelWorkspaceStatusAction stable-status.txt
SUBCOMMAND: # //src/ballet/sha512:sha512 [action 'Compiling src/ballet/sha512/fd_sha512_core_avx2.s', configuration: 462f7a8353785ed4ab21eca57082c09d0458027d8309ce41bb7dd96b8ad0bff4, execution platform: @local_config_platform//:host]
(cd /home/ripatel/.cache/bazel/_bazel_ripatel/c4b4531357e10c25e070592873e21602/execroot/firedancer && \
  exec env - \
    PATH=/bin:/usr/bin:/usr/local/bin \
    PWD=/proc/self/cwd \
  external/gcc11_x86-64-v2/bin/gcc -fPIC -Wno-misleading-indentation -Wno-ignored-attributes '-DFD_HAS_DOUBLE=1' '-DFD_HAS_ALLOCA=1' '-DFD_HAS_X86=1' '-D_XOPEN_SOURCE=700' '-DFD_HAS_HOSTED=1' -pthread '-DFD_HAS_THREADS=1' '-DFD_HAS_ATOMIC=1' -no-canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -isystemexternal/sysroot_x86_64//include/c++/10.3.0 -isystemexternal/sysroot_x86_64//include/c++/10.3.0/x86_64-linux -isystemexternal/sysroot_x86_64//lib/gcc/x86_64-linux/10.3.0/include-fixed -isystemexternal/sysroot_x86_64//lib/gcc/x86_64-linux/10.3.0/include -isystemexternal/sysroot_x86_64//usr/include -c src/ballet/sha512/fd_sha512_core_avx2.s -o bazel-out/k8-fastbuild/bin/src/ballet/sha512/_objs/sha512/fd_sha512_core_avx2.pic.o)
# Configuration: 462f7a8353785ed4ab21eca57082c09d0458027d8309ce41bb7dd96b8ad0bff4
# Execution platform: @local_config_platform//:host
ERROR: /data/prj/firedancer/src/ballet/sha512/BUILD:5:14: Compiling src/ballet/sha512/fd_sha512_core_avx2.s failed: (Exit 1): gcc failed: error executing command (from target //src/ballet/sha512:sha512) external/gcc11_x86-64-v2/bin/gcc -fPIC -Wno-misleading-indentation -Wno-ignored-attributes '-DFD_HAS_DOUBLE=1' '-DFD_HAS_ALLOCA=1' '-DFD_HAS_X86=1' '-D_XOPEN_SOURCE=700' '-DFD_HAS_HOSTED=1' -pthread ... (remaining 17 arguments skipped)
Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging

src/main/tools/linux-sandbox-pid1.cc:490: "execvp(external/gcc11_x86-64-v2/bin/gcc, 0x1149cf0)": No such file or directory

Target //src/ballet/sha512:sha512 failed to build

Use --verbose_failures to see the command lines of failed build steps.

INFO: Elapsed time: 0.132s, Critical Path: 0.04s

INFO: 2 processes: 2 internal.

FAILED: Build did NOT complete successfully

FAILED: Build did NOT complete successfully

Build with `.S` file (success)

Loading: 
Loading: 0 packages loaded
Analyzing: target //src/ballet/sha512:sha512 (1 packages loaded, 0 targets configured)
INFO: Analyzed target //src/ballet/sha512:sha512 (1 packages loaded, 4 targets configured).
INFO: Found 1 target...
[0 / 4] [Prepa] BazelWorkspaceStatusAction stable-status.txt
SUBCOMMAND: # //src/ballet/sha512:sha512 [action 'Compiling src/ballet/sha512/fd_sha512_core_avx2.S', configuration: 462f7a8353785ed4ab21eca57082c09d0458027d8309ce41bb7dd96b8ad0bff4, execution platform: @local_config_platform//:host]
(cd /home/ripatel/.cache/bazel/_bazel_ripatel/c4b4531357e10c25e070592873e21602/execroot/firedancer && \
  exec env - \
    PATH=/bin:/usr/bin:/usr/local/bin \
    PWD=/proc/self/cwd \
  external/gcc11_x86-64-v2/bin/gcc -MD -MF bazel-out/k8-fastbuild/bin/src/ballet/sha512/_objs/sha512/fd_sha512_core_avx2.pic.d -fPIC -iquote . -iquote bazel-out/k8-fastbuild/bin -isystem bazel-out/k8-fastbuild/bin/external/numa/libnuma_headers/include -isystem bazel-out/k8-fastbuild/bin/external/numa/libnuma/include -Wno-misleading-indentation -Wno-ignored-attributes '-DFD_HAS_DOUBLE=1' '-DFD_HAS_ALLOCA=1' '-DFD_HAS_X86=1' '-D_XOPEN_SOURCE=700' '-DFD_HAS_HOSTED=1' -pthread '-DFD_HAS_THREADS=1' '-DFD_HAS_ATOMIC=1' --sysroot external/sysroot_x86_64 -no-canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -isystemexternal/sysroot_x86_64//include/c++/10.3.0 -isystemexternal/sysroot_x86_64//include/c++/10.3.0/x86_64-linux -isystemexternal/sysroot_x86_64//lib/gcc/x86_64-linux/10.3.0/include-fixed -isystemexternal/sysroot_x86_64//lib/gcc/x86_64-linux/10.3.0/include -isystemexternal/sysroot_x86_64//usr/include -c src/ballet/sha512/fd_sha512_core_avx2.S -o bazel-out/k8-fastbuild/bin/src/ballet/sha512/_objs/sha512/fd_sha512_core_avx2.pic.o)
# Configuration: 462f7a8353785ed4ab21eca57082c09d0458027d8309ce41bb7dd96b8ad0bff4
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # //src/ballet/sha512:sha512 [action 'Linking src/ballet/sha512/libsha512.a', configuration: 462f7a8353785ed4ab21eca57082c09d0458027d8309ce41bb7dd96b8ad0bff4, execution platform: @local_config_platform//:host]
(cd /home/ripatel/.cache/bazel/_bazel_ripatel/c4b4531357e10c25e070592873e21602/execroot/firedancer && \
  exec env - \
    PATH=/bin:/usr/bin:/usr/local/bin \
    PWD=/proc/self/cwd \
  external/gcc11_x86-64-v2/bin/ar rcsD bazel-out/k8-fastbuild/bin/src/ballet/sha512/libsha512.a bazel-out/k8-fastbuild/bin/src/ballet/sha512/_objs/sha512/fd_sha512.pic.o bazel-out/k8-fastbuild/bin/src/ballet/sha512/_objs/sha512/fd_sha512_core_avx2.pic.o)
# Configuration: 462f7a8353785ed4ab21eca57082c09d0458027d8309ce41bb7dd96b8ad0bff4
# Execution platform: @local_config_platform//:host
SUBCOMMAND: # //src/ballet/sha512:sha512 [action 'Linking src/ballet/sha512/libsha512.so', configuration: 462f7a8353785ed4ab21eca57082c09d0458027d8309ce41bb7dd96b8ad0bff4, execution platform: @local_config_platform//:host]
(cd /home/ripatel/.cache/bazel/_bazel_ripatel/c4b4531357e10c25e070592873e21602/execroot/firedancer && \
  exec env - \
    PATH=/bin:/usr/bin:/usr/local/bin \
    PWD=/proc/self/cwd \
  external/gcc11_x86-64-v2/bin/gcc -shared -o bazel-out/k8-fastbuild/bin/src/ballet/sha512/libsha512.so bazel-out/k8-fastbuild/bin/src/ballet/sha512/_objs/sha512/fd_sha512.pic.o bazel-out/k8-fastbuild/bin/src/ballet/sha512/_objs/sha512/fd_sha512_core_avx2.pic.o -pthread -Wl,-S -Wl,-z,relro,-z,now -pass-exit-codes -lm -lstdc++ --sysroot external/sysroot_x86_64 -Bexternal/gcc11_x86-64-v2/bin -Bexternal/sysroot_x86_64//usr/lib -Bexternal/sysroot_x86_64//lib64 -Lexternal/sysroot_x86_64//lib64 -Lexternal/sysroot_x86_64//usr/lib -Lexternal/sysroot_x86_64//lib/gcc/x86_64-linux/10.3.0)
# Configuration: 462f7a8353785ed4ab21eca57082c09d0458027d8309ce41bb7dd96b8ad0bff4
# Execution platform: @local_config_platform//:host
Target //src/ballet/sha512:sha512 up-to-date:
  bazel-bin/src/ballet/sha512/libsha512.a
  bazel-bin/src/ballet/sha512/libsha512.so
INFO: Elapsed time: 0.775s, Critical Path: 0.64s
INFO: 4 processes: 1 internal, 3 linux-sandbox.
INFO: Build completed successfully, 4 total actions
INFO: Build completed successfully, 4 total actions

Build our own toolchains

Since we already compile GCC for the sysroot, we might as well just setup a separate pipeline to also publish a toolchain. It will require extra tools like binutils.

This will solve the problem we have with default RBE images that don't have libgmp, libmpc and libmpfr shared objects installed that the bootlin binaries are dynamically linked to. Since we statically link those libraries into the GCC binaries, our toolchain will play nicer with RBE.

How to achieve better build performance?

It seems build performance needs to be improved compared to the build performance using the host compiler. I realized the issue in a x86-64 Linux VM on GCP when I built C++ libraries (e.g., protobuf, gRPC) with this toolchain. It seems the build using gcc-toolchain takes longer than when building C++ libraries using the host compiler. The reason seems to be costs for sandbox setup and deletion for every compile action, as far as I investigated. I noticed this performance issue can be mitigated by adding the --experimental_reuse_sandbox_directories option to CLI or .bazelrc. Since that option is experimental, I'm wondering there is any way to improve the build performance using gcc-toolchain.

For the record, below is the result of performance profiling on that VM. I measured the build time of the //examples/protobuf:hello_world_proto target in this repo in three different settings:

1. build with host compiler (by turning off --action_env=BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1, --incompatible_strict_action_env=true, --incompatible_enable_cc_toolchain_resolution in .bazelrc)
2. build with gcc-toolchain
3. build with gcc-toolchain, enabling --experimental_reuse_sandbox_directories in .bazelrc.

Environment Information

$ uname -a
Linux ubuntu-0 5.15.0-1016-gcp #21~20.04.1-Ubuntu SMP Fri Aug 5 12:53:07 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/9/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:hsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 9.4.0-1ubuntu1~20.04.1' --with-bugurl=file:///usr/share/doc/gcc-9/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,gm2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-9 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-9-Av3uEd/gcc-9-9.4.0/debian/tmp-nvptx/usr,hsa --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
$ bazel version
Build label: 5.2.0
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Jun 7 16:02:26 2022 (1654617746)
Build timestamp: 1654617746
Build timestamp as int: 1654617746

Repro steps

git clone https://github.com/aspect-build/gcc-toolchain.git
cd gcc-toolchain
git checkout 381975950d0909e1a1608c8a90858536562e4b1d # latest as of 2022-08-16
bazel build --profile=/tmp/prof //examples/protobuf:hello_world_proto
bazel analyze-profile /tmp/prof

Results

The following table shows the outputs of the above bazel analyze-profile /tmp/prof command for three settings. Here are few observations:

• Total run time of gcc-toolchain takes 1.7 times longer than host compiler
• gcc-toolchain spends more time (1.8 times longer) on execution phase than host compiler.
• Reusing sandbox helps. Total run time of gcc-toolchain w/ reuse sandbox is close to that of host compiler.

	host compiler	gcc-toolchain	gcc-toolchain w/ reuse sandbox
Total launch phase time	1.418 s (0.68%)	1.415 s (0.40%)	1.308 s (0.57%)
Total init phase time	0.671 s (0.32%)	0.673 s (0.19%)	0.748 s (0.33%)
Total target pattern evaluation phase time	1.817 s (0.87%)	5.008 s (1.40%)	1.821 s (0.80%)
Total interleaved loading-and-analysis phase time	71.166 s (34.11%)	107.429 s (30.10%)	71.944 s (31.43%
Total preparation phase time	0.026 s (0.01%)	0.017 s (0.00%)	0.025 s (0.01%)
Total execution phase time	133.468 s (63.98%)	242.359 s (67.90%)	152.983 s (66.84%)
Total finish phase time	0.045 s (0.02%)	0.054 s (0.02%)	0.045 s (0.02%)
Total run time	208.613 s (100.00%)	356.958 s (100.00%)	228.876 s (100.00%)

Tracing

Analyzed collected files by bazel analyze-profile in chrome://tracing for each setting, following https://bazel.build/rules/performance. The following screenshots show the results (execution phase) of gcc-toolchain and gcc-toolchain w/ reuse sandbox, respectively (corrected order of the screenshots).

Observations

• CPU usage (total) of gcc-toolchain is "jagged" compared to that of gcc-toolchain w/ reuse sandbox which keeps CPU busy.
• When CPU usage of gcc-toolchain goes down, Bazel spends on sandbox creation and deletion, showing green colored boxes before and after purple colored "subprocess.run" box.

[Bug]: C compilation breaks when using gcc 12

What happened?

I'm working on integrating support for gcc 12.2.0. I added the relevant entries to _TOOLCHAINS in toolchain/defs.bzl and built an updated sysroot, also updating _SYSROOTS in the same file to point to my local archive. Things work mostly as expected, however, there is one issue when compiling a C source file that includes stdatomic.h. Since gcc 12.2.0 adds (some) support for C++23, there is a file stdatomic.h as part of the C++ headers. Since all directories are added to the gcc call via -isystem, a C-only built finds that header as well, and due to the preprocessor guards in there it does nothing. Digging a bit further and asking on the libstdc++ mailinglist, I found out that a C compilation is not supposed to see that header in the first place. When using only --sysroot and dropping the -nostdinc and -isystem arguments, gcc actually does the right thing. So it looks like the natural fix for this is to do exactly that: use --sysroot and drop the -nostdinc and -isystem arguments. Unfortunately, it's not entirely straightforward, because gcc uses stddef.h and stdarg.h from the toolchain archive and then those paths are made absolute ones in the .d file, making Bazel complain about undeclared includes.

The easiest workaround is patching the C++23 stdatomic.h, so it does #include_next when included from C code. However, it looks like the better solution here is to properly use --sysroot. I'm not sure why gcc makes some paths absolute in the .d file, but I can imagine that using a single archive with compiler and sysroot would make things much easier.

Why do you have the custom sysroot and explicitly remove the one from the Bootlin toolchain? Also, when you build gcc yourself as part of the sysroot, why do you even use the Bootlin toolchain?

Version

Development (host) and target OS/architectures:

Output of bazel --version:
bazel 5.3.1

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:
patched main

Language(s) and/or frameworks involved:
C, C++

How to reproduce

I can put up an example to repro. But understanding the rationale behind the custom sysroot would be very helpful as a start.

Any other information?

No response

Fund our work

• Sponsor our open source work by donating a bug bounty

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

• chore(deps): update dependency rules_pkg to v0.10.1
• chore(deps): update actions/github-script action to v7
• chore(deps): update bazelbuild/setup-bazelisk action to v3
• chore(deps): update dependency aspect_bazel_lib to v2
• chore(deps): update dependency bazel to v7
• 🔐 Create all rate-limited PRs at once 🔐

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update dependency rules_cc to v0.0.9
• chore(deps): update dependency aspect_bazel_lib to v1.40.2
• chore(deps): update dependency bazel to v6.5.0
• chore(deps): update dependency bazel_skylib to v1.5.0
• chore(deps): update dependency com_google_protobuf to v3.25.3
• chore(deps): update dependency io_bazel_stardoc to v0.6.2
• chore(deps): update dependency lapack to v3.12.0
• chore(deps): update dependency rules_foreign_cc to v0.10.1
• chore(deps): update dependency com_google_protobuf to v25
• chore(deps): update dependency ubuntu to v22
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

bazel

internal.bzl

• io_bazel_stardoc 0.5.6
• aspect_bazel_lib v1.33.0
• rules_foreign_cc 0.9.0
• lapack v3.11.0
• com_google_protobuf v3.21.12
• rules_pkg 0.9.1

toolchain/repositories.bzl

• bazel_skylib 1.4.2
• aspect_bazel_lib v1.33.0
• rules_cc 0.0.8

bazelisk

.bazelversion

• bazel 6.3.1

dockerfile

sysroot/Dockerfile

• ubuntu 22.04

github-actions

.github/workflows/default.yaml

• actions/checkout v4
• bazelbuild/setup-bazelisk v2
• actions/checkout v4
• bazelbuild/setup-bazelisk v2
• actions/checkout v4
• bazelbuild/setup-bazelisk v2
• actions/checkout v4
• bazelbuild/setup-bazelisk v2
• actions/checkout v4
• bazelbuild/setup-bazelisk v2
• ubuntu 20.04
• ubuntu 20.04
• ubuntu 20.04

.github/workflows/new_issue.yaml

• actions/github-script v6

---

• Check this box to trigger a request for Renovate to run again on this repository

crosstools-ng support

Can I replace the current docker-based sysroots with the ones I generate using crosstools? I am looking for more control over glibc and libstdc++ versions.

Error during bazel build: cannot find /usr/lib/libc.so.6

I've been running into an issue where I get the following error when trying to use this toolchain to build zlib:

/home/atobaccowalla/.cache/bazel/_bazel_atobaccowalla/49400f0e8b5fdbc2448f09cd587fc4d3/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: cannot find /usr/lib/libc.so.6
/home/atobaccowalla/.cache/bazel/_bazel_atobaccowalla/49400f0e8b5fdbc2448f09cd587fc4d3/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: cannot find /usr/lib/libc_nonshared.a
/home/atobaccowalla/.cache/bazel/_bazel_atobaccowalla/49400f0e8b5fdbc2448f09cd587fc4d3/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: cannot find /usr/lib/ld-linux-x86-64.so.2

None of those files exist on my machine in /usr/lib, but they do exist in /usr/lib64 (ex /usr/lib64/libc.so.6):

➜ find /usr -name 'libc.so.6'
/usr/lib64/libc.so.6

That being said, I'm not sure if this toolchain is (or should be) checking paths on my machine rather than only looking inside the downloaded sysroot which contains those files in the expected location:

➜ find /home/atobaccowalla/.cache/bazel/_bazel_atobaccowalla/49400f0e8b5fdbc2448f09cd587fc4d3/execroot/gcc-toolchain-sample/external/sysroot_x86_64/ -name 'libc.so.6'
/home/atobaccowalla/.cache/bazel/_bazel_atobaccowalla/49400f0e8b5fdbc2448f09cd587fc4d3/execroot/gcc-toolchain-sample/external/sysroot_x86_64/usr/lib/libc.so.6

I'm including the full output I get when trying to build here and I've also created a minimal reproduction repo here.

[Bug]: Hermetic GCC toolchain and @io_bazel_rules_go produce warnings building debug-mode shared libraries

What happened?

What happened: linking a Cgo library into a Go binary produces a warning about incorrect use of DT_TEXTREL in a PIE target.
What I expected: no warnings, because consistent link strategies are used for intermediate and final results.

A Go package that uses cgo, compiled as a library in debug mode, will normally produce a .gopclntab that requires relocations. When using the standard Go rules with a system C toolchain on various flavors of Ubuntu, cgo=True libraries are built such that .gopclntab and other relocated read-only sections are prefixed with .data.relro. It appears that some interaction between the Aspect GCC toolchain and the Go rules is leading to a situation where intermediate .o files for Go builds are using internal linkage, while the final link step is using GNU ld as an external linker.

That conflict causes warnings and has in the past been a sign of binaries with executable data sections that need to be specially handled by the system loader. Notably, various musl-based systems are not able to correctly load such binaries.

One possible cause here is that the PIE-by-default nature of the GCC toolchain is not correctly detected by the Go rules.

See golang/go issue 10914, 17847, and 59866 for related discussions.

Version

Development (host) and target OS/architectures:
Host is Ubuntu 22.04.3 LTS, uname -srv Linux 6.2.0-1018-gcp #20~22.04.1-Ubuntu SMP Mon Oct 23 12:29:43 UTC 2023
Target is Linux x86-64, with similar kernel revisions

Output of bazel --version: bazel 6.4.0

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file: @io_bazel_rules_go = v0.43.0, Go toolchain = 1.20.2, Aspect GCC toolchain = 0.4.2

Complete repo available at https://github.com/rbosetti/aspect-repro

Language(s) and/or frameworks involved: Go, C

How to reproduce

`bazel build //:demobin` from the workspace in https://github.com/rbosetti/aspect-repro

Any other information?

Output from the repro step.

INFO: Analyzed target //:demobin (55 packages loaded, 16855 targets configured).
INFO: Found 1 target...
INFO: From GoLink demobin_/demobin:
/home/rlb/.cache/bazel/_bazel_rlb/5d655e0d5670b9e1307c236b8abb2e79/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: /tmp/go-link-2297262172/go.o: warning: relocation in read-only section `.gopclntab'
/home/rlb/.cache/bazel/_bazel_rlb/5d655e0d5670b9e1307c236b8abb2e79/external/gcc_toolchain_x86_64_files/bin/x86_64-linux-ld: warning: creating DT_TEXTREL in a PIE
Target //:demobin up-to-date:
  bazel-bin/demobin_/demobin
INFO: Elapsed time: 191.332s, Critical Path: 166.08s
INFO: 13 processes: 7 internal, 6 linux-sandbox.
INFO: Build completed successfully, 13 total actions

[Bug]: Headers present in the sysroot but not available in the bazel sandbox

What happened?

Some headers that are definitely present in the default toolchain's sysroot are not copied in the bazel sandbox.

For example bits/types/struct___jmp_buf_tag.h.

See https://gist.github.com/0x2Adr1/51635421f64d6de8cbb14dbf40a27a4e for a minimal code that reproduces the issue.

  exec env - \
    BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1 \
    PATH=/bin:/usr/bin:/usr/local/bin \
    PWD=/proc/self/cwd \
  external/gcc_toolchain_x86_64/bin/gcc -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -MD -MF bazel-out/k8-fastbuild/bin/_objs/main/main.pic.d '-frandom-seed=bazel-out/k8-fastbuild/bin/_objs/main/main.pic.o' -fPIC '-DBAZEL_CURRENT_REPOSITORY=""' -iquote . -iquote bazel-out/k8-fastbuild/bin -iquote external/bazel_tools -iquote bazel-out/k8-fastbuild/bin/external/bazel_tools --sysroot external/sysroot_x86_64 -no-canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' '-std=c++17' -isystemexternal/sysroot_x86_64//include/c++/10.3.0 -isystemexternal/sysroot_x86_64//include/c++/10.3.0/x86_64-linux -isystemexternal/sysroot_x86_64//lib/gcc/x86_64-linux/10.3.0/include-fixed -isystemexternal/sysroot_x86_64//lib/gcc/x86_64-linux/10.3.0/include -isystemexternal/sysroot_x86_64//usr/include -c main.cpp -o bazel-out/k8-fastbuild/bin/_objs/main/main.pic.o)
# Configuration: f68de507e19f0236b7b3947a923f4eddf8a133099bf7d87ad0db3cf41a440b09
# Execution platform: @local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
main.cpp:3:10: fatal error: bits/types/struct___jmp_buf_tag.h: No such file or directory
    3 | #include <bits/types/struct___jmp_buf_tag.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
Target //:main failed to build
INFO: Elapsed time: 0.798s, Critical Path: 0.13s
INFO: 4 processes: 4 internal.
FAILED: Build did NOT complete successfully

But this header is definitely present if I download and look at the toolchain content manually:

$ ls x86-64--glibc--stable-2021.11-5/x86_64-buildroot-linux-gnu/sysroot/usr/include/bits/types/struct___jmp_buf_tag.h 
x86-64--glibc--stable-2021.11-5/x86_64-buildroot-linux-gnu/sysroot/usr/include/bits/types/struct___jmp_buf_tag.h

Version

Development (host) and target OS/architectures: ArchLinux x86_64

Output of bazel --version: bazel 6.0.0

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file: 22862ac

Language(s) and/or frameworks involved: C++

How to reproduce

Minimal repro code: https://gist.github.com/0x2Adr1/51635421f64d6de8cbb14dbf40a27a4e

Any other information?

No response

Fund our work

• Sponsor our open source work by donating a bug bounty

Upload tarballs with every release

This is to give users a stable archive whose SHA they can depend on. GitHub does not guarantee stability of the SHA over time for automatically generated archives. (Remember when they broke the world last January?)

The "righteous" thing to do here is that every time you cut a release, you should upload a .tar.gz archive and attach it to the release. Here's an example of a release that has an artifact like this: besides the two auto-generated ones, there's also au-0.3.4.tar.gz, whose contents are stable and will never change. Here's the section of our own release instructions that show how to obtain the .tar.gz file easily.