[msan/tsan] MemorySanitizer can not mmap the shadow memory // Disabled ASLR // non-PIE about sdk

[dcharkes]

TSAN errors started occuring in the same time span:

--- Command "vm_compile_to_kernel" (took 41.000588s):
DART_CONFIGURATION=ReleaseTSANX64 /b/s/w/ir/pkg/vm/tool/gen_kernel --aot --platform=out/ReleaseTSANX64/vm_platform_strong.dill -o /b/s/w/ir/out/ReleaseTSANX64/generated_compilations/vm-aot-tsan-linux-release-x64/runtime_tests_vm_dart_regress_46796_test.dart_1/out.dill /b/s/w/ir/runtime/tests/vm/dart/regress_46796_test.dart -Dtest_runner.configuration=vm-aot-tsan-linux-release-x64 --packages=/b/s/w/ir/.dart_tool/package_config.json -Ddart.vm.product=false --sound-null-safety

exit code:
0

--- Command "precompiler" (took 169ms):
DART_CONFIGURATION=ReleaseTSANX64 out/ReleaseTSANX64/gen_snapshot --snapshot-kind=app-aot-assembly --assembly=/b/s/w/ir/out/ReleaseTSANX64/generated_compilations/vm-aot-tsan-linux-release-x64/runtime_tests_vm_dart_regress_46796_test.dart_1/out.S --sound-null-safety --use_slow_path -Dtest_runner.configuration=vm-aot-tsan-linux-release-x64 --ignore-unrecognized-flags --packages=/b/s/w/ir/.dart_tool/package_config.json /b/s/w/ir/out/ReleaseTSANX64/generated_compilations/vm-aot-tsan-linux-release-x64/runtime_tests_vm_dart_regress_46796_test.dart_1/out.dill

exit code:
-6

stderr:
FATAL: ThreadSanitizer: unexpected memory mapping 0x5b222626f000-0x5b22265d1000

--- Re-run this test:
python3 tools/test.py -n vm-aot-tsan-linux-release-x64 vm/dart/regress_46796_test/1

https://dart-ci.appspot.com/log/vm-aot-tsan-linux-release-x64/vm-aot-tsan-linux-release-x64/411/vm/dart/regress_46796_test/1

[dcharkes]

And many many more failures:

There are new test failures on [ddc] do not fail during cleanup of expression_compiler tests....Remove additional --no-goma options since it's the new default..

The tests

ffi/callback_unwind_error_test Timeout (expected Pass)
standalone/io/raw_socket_test/2 Crash (expected Pass)
standalone/io/raw_socket_test/3 Crash (expected Pass)
standalone/io/raw_socket_typed_data_test/1 Crash (expected Pass)
standalone/io/raw_socket_typed_data_test/3 Crash (expected Pass)
standalone/io/regress_10026_test Crash (expected Pass)
standalone/io/regress_21987_test Crash (expected Pass)
standalone/io/regress_54516_test Crash (expected Pass)
standalone/io/resource_handle_test Timeout (expected Pass)
standalone/io/secure_server_client_certificate_test Crash (expected Pass)
standalone/io/secure_socket_alpn_test Crash (expected Pass)
standalone/io/secure_socket_test/0 Crash (expected Pass)
standalone/io/secure_socket_test/1 Crash (expected Pass)
standalone/io/secure_socket_test/3 Crash (expected Pass)
standalone/io/security_context_no_private_key_test Crash (expected Pass)
standalone/io/signals_exception_test/none Crash (expected Pass)
standalone/io/socket_close_test/1 Crash (expected Pass)
standalone/io/socket_close_test/2 Crash (expected Pass)
standalone/io/socket_close_test/3 Crash (expected Pass)
standalone/io/socket_connect_stream_data_close_cancel_test/0 Crash (expected Pass)
standalone/io/socket_connect_stream_data_close_cancel_test/2 Crash (expected Pass)
standalone/io/socket_connect_stream_data_close_test/0 Crash (expected Pass)
standalone/io/socket_connect_stream_data_close_test/2 Crash (expected Pass)
standalone/io/socket_connect_stream_data_close_test/3 Crash (expected Pass)
standalone/io/socket_cross_process_test RuntimeError (expected Pass)
standalone/io/socket_finalizer_test Crash (expected Pass)
standalone/io/socket_hang_test RuntimeError (expected Pass)
standalone/io/socket_ipv6_test/02 Crash (expected Pass)
standalone/io/socket_ipv6_test/05 Crash (expected Pass)
standalone/io/socket_ipv6_test/06 Crash (expected Pass)
    and 105 more tests

are failing on configurations

vm-aot-msan-linux-release-x64
vm-aot-tsan-linux-release-x64

[dcharkes]

The MSAN and TSAN errors do not reproduce locally on 2b2c6ec:

$ tools/build.py --sanitizer=msan -mrelease create_platform_sdk runtime_precompiled && tools/test.py -n vm-aot-msan-linux-release-x64 --repeat=20 vm/dart/regress_49424_test
warning: linux-intel.cfg is deprecated, please update your RBE_cfg variable to unix.cfg use RBE=1 instead per go/dart-rbe
Done. Made 444 targets from 121 files in 316ms
Proxy started successfully.
buildtools/ninja/ninja -C out/ReleaseMSANX64 -j1000 -l64 create_platform_sdk runtime_precompiled
ninja: Entering directory `out/ReleaseMSANX64'
ninja: no work to do.
buildtools/ninja/ninja -C out/ReleaseMSANX64 -j1000 -l64 create_platform_sdk runtime_precompiled done.
The build took 0.897 seconds
No build targets found.
Test configuration:
    vm-aot-msan-linux-release-x64(architecture: x64, compiler: dartkp, mode: release, runtime: dart_precompiled, system: linux)
Suites tested: vm
[02:36 | 100% | +   20 | -    0]

=== All 20 tests passed ===

$ tools/build.py -mrelease --sanitizer=tsan create_platform_sdk runtime_precompiled && tools/test.py -n vm-aot-tsan-linux-release-x64 --repeat=20 vm/dart/regress_46796_test
warning: linux-intel.cfg is deprecated, please update your RBE_cfg variable to unix.cfg use RBE=1 instead per go/dart-rbe
Done. Made 444 targets from 121 files in 363ms
Proxy started successfully.
buildtools/ninja/ninja -C out/ReleaseTSANX64 -j1000 -l64 create_platform_sdk runtime_precompiled
ninja: Entering directory `out/ReleaseTSANX64'
ninja: no work to do.
buildtools/ninja/ninja -C out/ReleaseTSANX64 -j1000 -l64 create_platform_sdk runtime_precompiled done.
The build took 0.919 seconds
No build targets found.
Test configuration:
    vm-aot-tsan-linux-release-x64(architecture: x64, compiler: dartkp, mode: release, runtime: dart_precompiled, system: linux)
Suites tested: vm
[07:07 |  95% | +   38 | -    0]

[dcharkes]

@sortie thinks it might be a kernel update to the bots.

Some googling around on the error message seems to verify this hypothesis:

• https://stackoverflow.com/questions/77850769/fatal-threadsanitizer-unexpected-memory-mapping-when-running-on-linux-kernels

The workaround mentions: llvm/llvm-project@7d039ef (2023 nov), we are already on llvm/llvm-project@c58bc24 (2024 jan), so we might be able to apply the workaround mentioned in the issue.

[dcharkes]

A clang roll to an even newer clang: https://dart-review.googlesource.com/c/sdk/+/359541

[dcharkes]

I believe the clang roll made everything green again:

• https://dart-ci.web.app/#showLatestFailures=false&configurations=vm-aot-msan-linux-release-x64
• https://dart-ci.web.app/#showLatestFailures=false&configurations=vm-aot-tsan-linux-release-x64

(The amount of tests turned green is equal to the amount turned red yesterday.)

Thanks @rmacnak-google!