Reproducible Builds about sdk

[srawlins]

Tentatively triaging to VM. Let me know if this is more of a language issue...

[alexmarkov]

This happens because Flutter generates .dart_tool/flutter_build/dart_plugin_registrant.dart file outside of a package. It has file:// URI which includes absolute path. This is a Flutter tooling issue, consider filing it in the Flutter repository.

Also, it is not clear why Flutter needs both:

--source .dart_tool/flutter_build/dart_plugin_registrant.dart 
--source package:flutter/src/dart_plugin_registrant.dart 

[jwinarske]

@alexmarkov I get the same behavior regardless of the combo passed; with or without file:// prefix. app.dill as source to gen_snapshot contains absolute paths for all of the dart files.

--source .dart_tool/flutter_build/dart_plugin_registrant.dart 
--source package:flutter/src/dart_plugin_registrant.dart 

These two files are not the same. The local folder one is what shows up in libapp.so post gen_snapshot:

$ strings libapp.so |grep tmp
file:///mnt/raid10/yocto/master/raspberry-pi5/tmp/work/cortexa76-poky-linux/flutter-gallery/2.10.2/git/.dart_tool/flutter_build/dart_plugin_registrant.dart

The problem lies in gen_snapshot.

[alexmarkov]

Compiler retains URIs as they may be needed at runtime (for example to show stack traces). This is not a problem for package:// URIs, but file:// URIs include absolute paths. In this case, Flutter tooling added .dart_tool/flutter_build/dart_plugin_registrant.dart to the program being compiled. This file lives outside of any package, so compiler includes its file:// URI.

Maybe you can try using --obfuscate and --split-debug-info Flutter options (see https://docs.flutter.dev/deployment/obfuscate) to avoid those absolute paths.

[jwinarske]

I see. Needing the path for debugging makes sense, only the file is transient. It would only resolve if the image was built. The other scenario is there are no debug tools in the case of AOT.

Neither of these (--obfuscate --strip / --obfuscate --save-obfuscation-map=obfuscation.map --strip) impact the issue:

$ gen_snapshot --deterministic --obfuscate --strip --snapshot_kind=app-aot-elf --elf=libapp.so --strip .dart_tool/flutter_build/*/app.dill
$ strings libapp.so |grep tmp
file:///mnt/raid10/yocto/master/raspberry-pi5/tmp/work/cortexa76-poky-linux/flutter-gallery/2.10.2/git/.dart_tool/flutter_build/dart_plugin_registrant.dart
$ gen_snapshot --deterministic --obfuscate --save-obfuscation-map=obfuscation.map --strip --snapshot_kind=app-aot-elf --elf=libapp.so --strip .dart_tool/flutter_build/*/app.dill
$ strings libapp.so |grep tmp
file:///mnt/raid10/yocto/master/raspberry-pi5/tmp/work/cortexa76-poky-linux/flutter-gallery/2.10.2/git/.dart_tool/flutter_build/dart_plugin_registrant.dart

I'm open to maintaining a patch. Just not clear what to patch in gen_snapshot.

[jwinarske]

This combo looks promising:

$ dart --disable-analytics --disable-dart-dev /mnt/raid10/yocto/master/raspberry-pi5/tmp/work/cortexa76-poky-linux/flutter-gallery/2.10.2/recipe-sysroot-native/usr/share/flutter/sdk/bin/cache/artifacts/engine/linux-x64/frontend_server.dart.snapshot --sdk-root /mnt/raid10/yocto/master/raspberry-pi5/tmp/work/cortexa76-poky-linux/flutter-gallery/2.10.2/recipe-sysroot-native/usr/share/flutter/sdk/bin/cache/artifacts/engine/common/flutter_patched_sdk_product/ --target=flutter --no-print-incremental-dependencies -Ddart.vm.profile=false -Ddart.vm.product=true --aot --tfa --target-os linux --packages .dart_tool/package_config.json  --output-dill .dart_tool/flutter_build/*/app.dill  --depfile .dart_tool/flutter_build/*/kernel_snapshot.d --source package:flutter/src/dart_plugin_registrant.dart  -Dflutter.dart_plugin_registrant=.dart_tool/flutter_build/dart_plugin_registrant.dart --verbosity=error package:gallery/main.dart
result c38bebdc-5ba9-4c1d-b1ab-8ebc1b318c9e
c38bebdc-5ba9-4c1d-b1ab-8ebc1b318c9e
c38bebdc-5ba9-4c1d-b1ab-8ebc1b318c9e .dart_tool/flutter_build/b7af51db6b4c9bfe7a0b7902f9856b16/app.dill 0
$ ../git/engine_sdk/sdk/clang_x64/gen_snapshot --deterministic --obfuscate --save-obfuscation-map=obfuscation.map --strip --snapshot_kind=app-aot-elf --elf=libapp.so --strip .dart_tool/flutter_build/*/app.dill
Warning: Generating ELF library without DWARF debugging information.
joel@air:/mnt/raid10/yocto/master/raspberry-pi5/tmp/work/cortexa76-poky-linux/flutter-gallery/2.10.2/git$ strings libapp.so |grep tmp

Contents of packages/flutter/lib/src/dart_plugin_registrant.dart

$ cat ./recipe-sysroot-native/usr/share/flutter/sdk/packages/flutter/lib/src/dart_plugin_registrant.dart
// Copyright 2014 The Flutter Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

/// The location of the Dart Plugin Registrant. This is used by the engine to
/// execute the Dart Plugin Registrant when the Isolate is started or
/// DartPluginRegistrant.ensureInitialized() is called from a background
/// Isolate.
@pragma('vm:entry-point')
const String dartPluginRegistrantLibrary = String.fromEnvironment('flutter.dart_plugin_registrant');

How does one confirm the entry point other than via a target test?

[alexmarkov]

If you're running gen_snapshot directly, then you can also try its --dwarf-stack-traces option. Flutter tool passes it when --split-debug-info is specified.

[jwinarske]

I see dartPluginRegistrantLibrary in the data.debug output but it's not clear it's the actual vm entry point. How do I determine the vm:entry-point?

Child 109170 (at offset 0x4ff649):
  Abbreviation code: 3):
    DW_AT_abstract_origin => 0x000fa89d (origin: dartPluginRegistrantLibrary)
    DW_AT_low_pc => 0x10dbf44
    DW_AT_high_pc => 0x10dbf4c
    DW_AT_artificial => false
...
Debug information entry at offset 0x4ff649:
Abbreviation code: 3):
  DW_AT_abstract_origin => 0x000fa89d (origin: dartPluginRegistrantLibrary)
  DW_AT_low_pc => 0x10dbf44
  DW_AT_high_pc => 0x10dbf4c
  DW_AT_artificial => false

[jwinarske]

With my last idea I stated looked promising, testing on device dartPluginRegistrantLibrary is not called. It would seem only the implicit file:// addition works, which leaves the problem as I originally documented.

[a-siva]

@jwinarske are you using 'flutter tools' to build the app or do you have your own set of scripts for generating the final app ?

[jwinarske]

I manually cross compile (Yocto) using this function:
https://github.com/meta-flutter/meta-flutter/blob/master/conf/include/common.inc#L283

I tried to setup a minimum viable repro for the flutter team this morning, only not cross compiling with standard flutter bits does not repro.

I custom build the flutter engine, and use the gen_snapshot from this. Linux_simarm, linux_simarm64, linux_simx86, linux_x64, linux_simriscv32, and linux_simriscv64 are used depending on the BSP/machine architecture. The buildroot flutter work is all based/derived from my work as well.

Not sure yet if it's isolated to linux_sim...

[a-siva]

@jwinarske without a minimal repro case for us to debug/investigate there is not much we can do here, please provide steps for a reproducing the problem.

[jwinarske]

@a-siva The initial post here has the steps to repro. The dart-sdk gen_snapshot would need to be linux_simarm64. Anything other than a straight host build; which is what the flutter tooling uses.