go-debos / debos Goto Github PK

For those of us not adjacent to a mirror, apt-cacher-ng in proxy mode is a useful way to speed up the test/debug/fix cycle. Setting http_proxy is probably the best way: that has the benefit that it can work for both debootstrap and apt, and the generated filesystem is left with a canonical mirror like deb.debian.org hard-coded into it (as opposed to having the proxy or a local mirror end up in /etc in the output).

I think this should be a command-line option or environment variable instead of part of the recipe, because it's really a fact about the machine/LAN where you're running debos rather than a fact about the recipe itself, and doesn't make a whole lot of sense to version-control.

no_proxy might also be useful to pass through, for the case where locally-served packages can't or shouldn't go through the proxy.

https_proxy and ftp_proxy are perhaps less useful but work the same way as http_proxy, so wouldn't need be significantly more code.

I'm using go-debos and I am getting build failure.
It does not always occur. I'm on ubuntu and have a couple patches.

fredo@X250:~/Documents/bos0009/apertis-image-recipes$ ../go/src/github.com/go-debos/debos/debos -t architecture:$ARCH --debug-shell apertis-image-$ARCH.yaml
2017/11/17 11:36:47 Ignored link: /etc/ld.so.conf.d//etc/ld.so.conf.d/i386-linux-gnu_GL.conf
2017/11/17 11:36:47 Ignored link: /etc/ld.so.conf.d//etc/ld.so.conf.d/x86_64-linux-gnu_EGL.conf
2017/11/17 11:36:47 Ignored link: /etc/ld.so.conf.d//etc/ld.so.conf.d/x86_64-linux-gnu_GL.conf
2017/11/17 11:36:47 Getting module from host kernel/drivers/virtio/virtio.ko
2017/11/17 11:36:47 Ignoring kernel/drivers/virtio/virtio.ko
2017/11/17 11:36:47 Getting module from host kernel/drivers/virtio/virtio_pci.ko
2017/11/17 11:36:47 Ignoring kernel/drivers/virtio/virtio_pci.ko
2017/11/17 11:36:47 Getting module from host kernel/net/9p/9pnet.ko
2017/11/17 11:36:47 Getting module from host kernel/drivers/virtio/virtio_ring.ko
2017/11/17 11:36:47 Ignoring kernel/drivers/virtio/virtio_ring.ko
2017/11/17 11:36:47 Getting module from host kernel/fs/9p/9p.ko
2017/11/17 11:36:47 Getting module from host kernel/net/9p/9pnet_virtio.ko
2017/11/17 11:36:47 Getting module from host kernel/fs/fscache/fscache.ko
2017/11/17 11:36:47 Getting module from host modules.order
2017/11/17 11:36:47 Getting module from host modules.builtin
2017/11/17 11:36:47 Getting module from host modules.dep
2017/11/17 11:36:47 Getting module from host modules.dep.bin
2017/11/17 11:36:47 Getting module from host modules.alias
2017/11/17 11:36:47 Getting module from host modules.alias.bin
2017/11/17 11:36:47 Getting module from host modules.softdep
2017/11/17 11:36:47 Getting module from host modules.symbols
2017/11/17 11:36:47 Getting module from host modules.symbols.bin
2017/11/17 11:36:47 Getting module from host modules.builtin.bin
2017/11/17 11:36:47 Getting module from host modules.devname
[    1.341739] systemd-fstab-generator[137]: Failed to create mount unit file /run/systemd/generator/home-fredo-Documents-bos0009-apertis\x2dimage\x2drecipes.mount, as it already exists. Duplicate entry in /etc/fstab?
[    1.835466] Error: Driver 'pcspkr' is already registered, aborting...
Running /debos --artifactdir /home/fredo/Documents/bos0009/apertis-image-recipes --template-var architecture:"amd64" /home/fredo/Documents/bos0009/apertis-image-recipes/apertis-image-amd64.yaml --debug-shell --shell /bin/bash --internal-image /dev/disk/by-id/virtio-fakedisk-0
2017/11/17 10:36:51 ==== Unpack ospack_17.12-amd64-minimal_00000000.0.tar.gz ====
2017/11/17 10:36:54 ==== image-partition ====
2017/11/17 10:36:55 Formatting partition 1 | mkfs.fat 4.0 (2016-05-06)
2017/11/17 10:36:56 Formatting partition 2 | mkfs.fat: warning - lowercase labels might not work properly with DOS or Windows
2017/11/17 10:36:56 Formatting partition 2 | mkfs.fat 4.0 (2016-05-06)
2017/11/17 10:36:56 Formatting partition 4 | mkfs.fat: warning - lowercase labels might not work properly with DOS or Windows
2017/11/17 10:36:56 Formatting partition 4 | mkfs.fat 4.0 (2016-05-06)
2017/11/17 10:36:57 Formatting partition 5 | mkfs.fat: warning - lowercase labels might not work properly with DOS or Windows
2017/11/17 10:36:57 Formatting partition 5 | mkfs.fat 4.0 (2016-05-06)
2017/11/17 10:36:58 Formatting partition 6 | btrfs-progs v4.9.1
2017/11/17 10:36:58 Formatting partition 6 | See http://btrfs.wiki.kernel.org for more information.
2017/11/17 10:36:58 Formatting partition 6 | 
2017/11/17 10:36:58 Formatting partition 6 | Label:              system
2017/11/17 10:36:58 Formatting partition 6 | UUID:               61db6be5-758c-4153-8d28-12ad22f7048d
2017/11/17 10:36:58 Formatting partition 6 | Node size:          16384
2017/11/17 10:36:58 Formatting partition 6 | Sector size:        4096
2017/11/17 10:36:58 Formatting partition 6 | Filesystem size:    3.81GiB
2017/11/17 10:36:58 Formatting partition 6 | Block group profiles:
2017/11/17 10:36:58 Formatting partition 6 |   Data:             single            8.00MiB
2017/11/17 10:36:58 Formatting partition 6 |   Metadata:         DUP             195.31MiB
2017/11/17 10:36:58 Formatting partition 6 |   System:           DUP               8.00MiB
2017/11/17 10:36:58 Formatting partition 6 | SSD detected:       no
2017/11/17 10:36:58 Formatting partition 6 | Incompat features:  extref, skinny-metadata
2017/11/17 10:36:58 Formatting partition 6 | Number of devices:  1
2017/11/17 10:36:58 Formatting partition 6 | Devices:
2017/11/17 10:36:58 Formatting partition 6 |    ID        SIZE  PATH
2017/11/17 10:36:58 Formatting partition 6 |     1     3.81GiB  /dev/disk/by-id/virtio-fakedisk-0-part6
2017/11/17 10:36:58 Formatting partition 6 | 
2017/11/17 10:36:58 Formatting partition 7 | btrfs-progs v4.9.1
2017/11/17 10:36:58 Formatting partition 7 | See http://btrfs.wiki.kernel.org for more information.
2017/11/17 10:36:58 Formatting partition 7 | 
2017/11/17 10:36:58 Formatting partition 7 | Label:              general_storage
2017/11/17 10:36:58 Formatting partition 7 | UUID:               3d24d634-dbd0-4779-b17e-51925beea726
2017/11/17 10:36:58 Formatting partition 7 | Node size:          16384
2017/11/17 10:36:58 Formatting partition 7 | Sector size:        4096
2017/11/17 10:36:58 Formatting partition 7 | Filesystem size:    2.16GiB
2017/11/17 10:36:58 Formatting partition 7 | Block group profiles:
2017/11/17 10:36:58 Formatting partition 7 |   Data:             single            8.00MiB
2017/11/17 10:36:58 Formatting partition 7 |   Metadata:         DUP             110.81MiB
2017/11/17 10:36:58 Formatting partition 7 |   System:           DUP               8.00MiB
2017/11/17 10:36:58 Formatting partition 7 | SSD detected:       no
2017/11/17 10:36:58 Formatting partition 7 | Incompat features:  extref, skinny-metadata
2017/11/17 10:36:58 Formatting partition 7 | Number of devices:  1
2017/11/17 10:36:58 Formatting partition 7 | Devices:
2017/11/17 10:36:58 Formatting partition 7 |    ID        SIZE  PATH
2017/11/17 10:36:58 Formatting partition 7 |     1     2.16GiB  /dev/disk/by-id/virtio-fakedisk-0-part7
2017/11/17 10:36:58 Formatting partition 7 | 
2017/11/17 10:36:58 Action `image-partition` failed at stage Run, error: system mount failed: no such file or directory
2017/11/17 10:36:58 >>> Starting a debug shell

Add correct property for debootstrap action.

Udevd have no enough time to start and create all needed partition symlinks for commands using direct access to device:

2017/11/20 09:29:33 ==== Prepare uInitrd ====
2017/11/20 09:29:33 DEBUG: [systemd-nspawn -q -D /scratch/mnt sh -c dd if=/dev/zero of=/dev/disk/by-id/virtio-fakedisk-0-part3 bs=512 count=1]
2017/11/20 09:29:33 dd if=/dev/zero of=/dev/disk/by-id/virtio-fakedisk-0-part3 bs=512 count=1 | host's /etc/localtime is not a symlink, not updating container timezone.
2017/11/20 09:29:33 dd if=/dev/zero of=/dev/disk/by-id/virtio-fakedisk-0-part3 bs=512 count=1 | /bin/dd: opening `/dev/disk/by-id/virtio-fakedisk-0-part3': No such file or 

Currently '/dev/vda' is hardcoded. So adding any other disk image prior to this action breaks normal work sequence.

Hi,

How to install debos?

Yes, I'm new to the go-lang world.

In the GCC world there is the good tradition of providing an INSTALL file.

Please provide an eqivalent of the

./configure
make
sudo make install

dance, preferable in INSTALL :-)

See also https://en.wikipedia.org/wiki/README#As_a_generic_term

When deboostrapping HTTPS-based repositories the apt-transport-https and ca-certificates should be installed right from the start, and that's usually handled with the --include option.

In my specific case their availability is probably better ensured through a custom debootstrap script (see #16), but I wonder if there's value in exposing more deboostrap options.

debos looks very similar to https://github.com/larswirzenius/vmdb2/ :)

Perhaps you could collaborate or merge the two projects.

cc @larswirzenius @d4s

Add option to install things with pip in yaml, I can imagine something like:

- action: pip
  version: 3
  packages:
      - pyserial
      - sarlanga

Add mountpoint which are mounted only in build time, e.g. do not need to add them into fstab and need to remove mount points on rootfs after umounting.

Ability to create OS directly on block devices.

In debootstrap between 1.0.85 (inclusive) and 1.0.87 (exclusive), and in debootstrap 1.0.102+, merged /usr is the default. debos correctly uses the --merged-usr option when merged-usr: true is in the YAML, but does not use --no-merged-usr when merged-usr: false is in the YAML.

This particularly matters when bootstrapping distributions that are so old they are not mentioned in debootstrap's own merged-usr blacklist, like Ubuntu 12.04 'precise'.

Or possibly merged-usr should be some sort of tristate: true, false, or use the debootstrap default?

Currently have 'Out of memory' for medium and large sized OS.

Need a recipes validator able to warn user about incorrect type passed to avoid issues similar to #74

https://github.com/go-debos/debos-recipes should be merged in here so examples, as essential part of documentation, are shipped together with Debos.

debos currently hardcodes /usr/share/debootstrap/scripts/unstable as the debootstrap script, which of course is not always appropriate.

It should be customizable, possibly with some magic to make local files available in the fakemachine even without installing the script in the host's /usr (it would be handy to ship the script in the same repository as the YAML for debos and directly use that).

Source or origin must be set to prevent copying of recipedir

If debootstrap fails, it often doesn't print full details, just "see debootstrap.log", which is not a whole lot of use if the sysroot you're bootstrapping is in a transient virtual machine's /scratch filesystem.

In other projects I've done things like this (argv is a debootstrap command-line):

            try:
                self.root_worker.check_call(argv)
            except:
                with suppress(Exception):
                    self.root_worker.check_call([
                        'cat',
                        '{}/debootstrap/debootstrap.log'.format(base_chroot),
                    ])
                raise

Need a preseed mechanism support.

Partition Type GUIDs are used to identify GPT partitions.

They are at the core of the Discoverable Partition Spec, which is implemented by systemd-auto-gpt-generator. This generator can't work without it.

Right now I set these Partition Type GUIDs through a post-process step such as:

  # EFI System Partition
  - action: run
    postprocess: true
    command: /sbin/sfdisk --part-type image.img 1 c12a7328-f81f-11d2-ba4b-00a0c93ec93b

It would be nice to have it supported in the image-partition action though, with something like:

  partitions:
    - name: label
    - part-type: c12a7328-f81f-11d2-ba4b-00a0c93ec93b
      ...

Is it something that you would like to support in debos?

It would be nice to have some usable end-to-end examples of how to make an image using Debos and how to run the image. Preferably for some popular architectures like amd64, arm, etc.

I've successfully built an image using Debos but I can not boot it...

Some old packages (libpng12-0_1.2.54-1ubuntu1, I'm looking at you) return puzzling errors with a merged /usr.

To make debos usable in those cases it should be possible to disable passing the --merged-user option to deboostrap.

Hi,

In https://github.com/go-debos/debos/blob/master/actions/debootstrap_action.go there is
for Run
cmdline := []string{"debootstrap", ... }
and for RunSecondStage
cmdline := []string{"/debootstrap/debootstrap", ... }

I would expect that both paths to the debootstrap executable would be the same.

Why isn't the path the same?

Cheers
Geert Stappers

Useful to run autopkgtests

From some discussions outside github, some ideas for a future documentation:

• move to rst/md all the non developer information about debos
• document the need to have rw permission to use /dev/kvm by the user
• document how to install debos and/or use public docker image ?
• explain that recipes are run though go templating

I have a local mirror, which I was attempting to run debos against it passing file:///srv/mirror as mirror, of course that was failing as debootstrap stage happens inside virtual fakemachine.

In case if a single file is needed the origin is set to the file name.
It is not possible to use the pair origin + filename in other actions.
Example:

  - action: download
    name: uboot
    url: http://linode.boundarydevices.com/u-boot-images/u-boot.nitrogen6q

Ths action sets origin named uboot to downloaded file /scratch/u-boot.nitrogen6q

• Expected raw action

  - action: raw
    description: Install U-Boot
    origin: uboot
    source: u-boot.nitrogen6q
    offset: {{ sector 2 }}

is trying to write file named /scratch/u-boot.nitrogen6q/u-boot.nitrogen6q

• Workaround

  - action: raw
    description: Install U-Boot
    origin: uboot
    source: ../u-boot.nitrogen6q
    offset: {{ sector 2 }}

Hi,

Debos is now in Debian and I have it installed in a chroot.

Testdriving debos with the rpi3 recipe reveals that some test on the host system are performed.
I don't understand why /lib/modules and Kernel are needed in early execution stage.

Screen shot:

root@yanur:/src/githubclones/debos-recipes/debian/arm64/image-rpi3# debos debimage-rpi3.yaml 
open /lib/modules: no such file or directory
root@yanur:/src/githubclones/debos-recipes/debian/arm64/image-rpi3# mkdir /lib/modules
root@yanur:/src/githubclones/debos-recipes/debian/arm64/image-rpi3# debos debimage-rpi3.yaml 
No kernel found
root@yanur:/src/githubclones/debos-recipes/debian/arm64/image-rpi3# 

Why the search for a kernel?

During call of any command/script in chroot environment 'systemd-nspawn' copies resolv.conf from host's environment.

Just discovered debos, and I love it! I copied the example out of the readme to get started:

{{- $image := or .image "debian.tgz" -}}

architecture: arm64

actions:
  - action: debootstrap
    suite: "buster"
    components:
      - main
      - non-free
    mirror: https://deb.debian.org/debian
    variant: minbase

  - action: apt
    packages: [ sudo, openssh-server, adduser, systemd-sysv, firmware-linux ]

  - action: run
    chroot: true
    command: echo debian > /etc/hostname

  - action: pack
    file: {{ $image }}
    compression: gz

When I run debos example.yaml, the scratch VM boots and starts building, but debootstrap fails near the end of base system installation:

<most debootstrap spam snipped>
2018/08/17 19:28:33 Debootstrap (stage 2) | I: Configuring the base system...
2018/08/17 19:28:33 Debootstrap (stage 2) | I: Configuring apt-transport-https...
2018/08/17 19:28:33 Debootstrap (stage 2) | I: Configuring libssl1.1:arm64...
2018/08/17 19:28:34 Debootstrap (stage 2) | I: Configuring openssl...
2018/08/17 19:28:34 Debootstrap (stage 2) | I: Configuring ca-certificates...
2018/08/17 19:29:09 Debootstrap (stage 2) | I: Configuring libc-bin...
2018/08/17 19:29:09 Debootstrap (stage 2) | I: Configuring ca-certificates...
2018/08/17 19:29:18 Debootstrap (stage 2) | I: Base system installed successfully.
2018/08/17 19:29:19 Action `debootstrap` failed at stage Run, error: exec: "systemd-nspawn": executable file not found in $PATH
Powering off.

I've only dug a tiny bit, but it looks like systemd-nspawn maybe got split out into its own package, systemd-container, and debootstrap isn't aware of this yet? Is there anything debos can do to work around this?

Option for mountpoint is ignored. For example:

  - action: image-partition
    imagename: {{ $image }}.img
    mountpoints:
      - mountpoint: /
        partition: system
        flags: [ boot ]
        options: ro

I'm hitting an issue when running the following recipe:

architecture: amd64

actions:

  - action: image-partition
    imagename: test.img
    imagesize: 20G
    partitiontype: gpt

    mountpoints:
      - mountpoint: /
        partition: system
        flags: [ boot ]
      - mountpoint: /boot/efi
        partition: EFI

    partitions:
      - name: EFI
        fs: vfat
        start: 6176s
        end: 256M
        flags: [ boot ]
      - name: system
        fs: btrfs
        start: 256M
        end: 100%

This fails on first run with a mkfs.vfat error (as shown below) but will succeed on subsequent attempts. Whether it succeeds or fails appears to be linked to the use of /dev/loop0:

user@apertis:~$ losetup
user@apertis:~$ sudo debos test.yaml
2018/05/23 14:00:56 fakemachine not supported, running on the host!
2018/05/23 14:00:56 ==== image-partition ====
2018/05/23 14:00:56 Formatting partition 1 | mkfs.vfat: Will not try to make filesystem on full-disk device '/dev/loop0p1' (use -I if wanted)
2018/05/23 14:00:56 Formatting partition 1 | mkfs.vfat 2.11 (12 Mar 2005)
2018/05/23 14:00:56 Action `image-partition` failed at stage Run, error: exit status 1
user@apertis:~$ losetup
user@apertis:~$ sudo debos test.yaml
2018/05/23 14:01:03 fakemachine not supported, running on the host!
2018/05/23 14:01:03 ==== image-partition ====
2018/05/23 14:01:03 Formatting partition 1 | mkfs.vfat 2.11 (12 Mar 2005)
2018/05/23 14:01:03 Formatting partition 1 | unable to get drive geometry, using default 255/63
2018/05/23 14:01:03 Formatting partition 2 | btrfs-progs v4.4
2018/05/23 14:01:03 Formatting partition 2 | See http://btrfs.wiki.kernel.org for more information.
2018/05/23 14:01:03 Formatting partition 2 | 
2018/05/23 14:01:03 Formatting partition 2 | Performing full device TRIM (18.39GiB) ...
2018/05/23 14:01:03 Formatting partition 2 | Label:              system
2018/05/23 14:01:03 Formatting partition 2 | UUID:               29114924-0aa2-47da-aaaf-3e59343371b0
2018/05/23 14:01:03 Formatting partition 2 | Node size:          16384
2018/05/23 14:01:03 Formatting partition 2 | Sector size:        4096
2018/05/23 14:01:03 Formatting partition 2 | Filesystem size:    18.39GiB
2018/05/23 14:01:03 Formatting partition 2 | Block group profiles:
2018/05/23 14:01:03 Formatting partition 2 |   Data:             single            8.00MiB
2018/05/23 14:01:03 Formatting partition 2 |   Metadata:         DUP               1.01GiB
2018/05/23 14:01:03 Formatting partition 2 |   System:           DUP              12.00MiB
2018/05/23 14:01:03 Formatting partition 2 | SSD detected:       no
2018/05/23 14:01:03 Formatting partition 2 | Incompat features:  extref, skinny-metadata
2018/05/23 14:01:03 Formatting partition 2 | Number of devices:  1
2018/05/23 14:01:03 Formatting partition 2 | Devices:
2018/05/23 14:01:03 Formatting partition 2 |    ID        SIZE  PATH
2018/05/23 14:01:03 Formatting partition 2 |     1    18.39GiB  /dev/loop1p2
2018/05/23 14:01:03 Formatting partition 2 | 
2018/05/23 14:01:04 ==== Recipe done ====
user@apertis:~$ losetup
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE
/dev/loop0         0      0         0  0 /home/user/test.img
/dev/loop1         0      0         0  0 /home/user/test.img

It also appears that debos is not clearing up loopback devices that it has created.

Tested on commit 8f2bc2a

Live-build has a fancy match-based overlay system, where it looks for the folders matching the current suite, arch and platform, even partially.

It basically expands the (suite,arch,platform) tuple and automatically overlays the contents of any existing folder that matches the expansion:

$ bash -c 'SUITE=buster; ARCH=amd64; PLATFORM=uefi; echo {suite,$SUITE}-{arch,$ARCH}-{platform,$PLATFORM}'
suite-arch-platform suite-arch-uefi suite-amd64-platform suite-amd64-uefi buster-arch-platform buster-arch-uefi buster-amd64-platform buster-amd64-uefi

Having some kind of folder matching for overlays in debos could be useful, doing that through templates becomes quite verbose very quickly.

The YAML parser returns line numbers when encountering errors, but that's useless as they don't necessarily match the original line numbers in the source file due to template expansion.

Ideally it would be great to print a few lines around the error but the yaml module does not provide the line number information in its error type (it's in the string, but let's not go that way). Missing that, I think the best viable option would be to just dump the whole expanded YAML on errors.

Sometimes it is needed to write directly to partition instead of the disk image.
Need additional parameter for Raw action for pointing to correct partition on image.

Add the same option 'append-kernel-cmdline' as we have for 'ostree-deploy' action.

I don't see any caveats that this tool wont work on 16.04 (xenial) however the installation steps in the README do not work on 16.04. Is there:

1. any reason this won't work on older ubuntu distributions
2. are there alternate installation steps for those distros (I can update the README if so).

Debootstrap | E: Failed getting release file http://apt-cacher-ng:3142/debian/dists/unstable/Release

Apparently using a mirror URL without a port (e.g. http://deb.debian.org/debian) works...

Consider the following snippet in a recipe:

{{ $kver := "4.9.0" }}
#{{ $kver := "4.16.0" }}

Currently the value of kver will be set to 4.16.0, since the Golang templating engine is not aware of the YAML syntax. But this is definitely not what an user expects, comments should be ignored.

Description of actions and properties.

#8 adds the ability to unpack .deb files which is quite useful, but to download a .deb on demand you'd have to know the exact filename. In the case wehre you want to get the .deb because you want to raw write the bootloader into an image file this is a bit annoying as the url will change on every package update. If there were an action similar to download but takes an apt repository as parameters that could nicely be avoided

Sometimes knowing exactly what is getting executed is extremely helpful when debugging an error. Printing the commandline before executing a program provides a good hint to reproduce it (just an hint because there's fakemachine and what not to make things more complex, but often that's enough). :)

This is not a bug, more a feature request.
We used systemd-nspawn in a script called from debos to use target specific tools.
While they could have been in debos itself, they were already available in the target rootfs and were usable by script.
On arm, we needed mkimage, and on amd64 bootctl. mkimage was used thanks to qemu-arm-static.
It would be useful to have bind as option to bind the required folders.
To use with OStree, it is needed to get the OStree commit id from the image.

When using a recent host distro (Debian >= stretch) to build images based on an older distro (Debian <= jessie), mkfs.ext4 will enable the metadata_csum feature by default, but jessie's e2fsck doesn't understand that feature. I'm not sure whether jessie kernels understand it either (I'm currently working with a jessie derivative that happens to have a newer kernel).

vmdebootstrap automatically disables metadata_csum for jessie, but that's probably too much magic for debos' design. It would be useful to have a way to turn off that flag, even if it's just adding ^metadata_csum or -O ^metadata_csum to the YAML somewhere.

debos knows how to pack up the entire $ROOTDIR into a tarball, but it can't pack up (for example) just /usr into a tarball.

Similarly, it can commit the entire $ROOTDIR to an ostree repository, but can't commit something other than the root. This would be useful for Flatpak runtimes, which are meant to contain:

• /files: a copy of a merged /usr
• /metadata: a metadata file

Workaround: use action: run, chroot: false to run tar or ostree yourself. The way I'm currently doing this is to create ostree/main and ostree/sources in the $ROOTDIR, move usr to ostree/main/files, fetch corresponding source code into ostree/sources, and pack up ostree/sources and ostree/main as though they were my root directory.

If/when I split out time zones and locales into Flatpak extensions, I'll need to commit those separately: for instance, a timezone info extension would contain /files (a copy of /usr/share/zoneinfo) and /metadata.

Similarly, at the moment I'm using a debos recipe for the Platform runtime and a separate debos recipe for the Sdk runtime, and duplicating the common preparation used for both. Ideally my process would go something like this:

• populate $ROOTDIR with a complete Debian system that is slightly larger than the Platform
• do the /usr merge and other common preparation
• copy /usr, /etc and /var twice, for example into /platform and /sdk (hard-link or reflink to avoid duplicating large files)
• chroot into /platform and delete Essential packages that are pointless in an immutable, non-bootable runtime, like login, init, apt and dpkg, then commit/pack
• chroot into /sdk, install extra -dev packages, and download corresponding source code for the union of Platform and Sdk, then commit/pack

Workaround: use chroot: false and make the scripts enter the appropriate chroot themselves, using chroot or systemd-nspawn.

When something breaks nothing beats opening a shell and poking things.

Ideally all the exec'ed programs should be (optionally) run as "$PROGRAM || sh". Of course that's a wild simplification as we want $PROGRAM output to be mangled by debos while the shell need access to the user's stdin/stdout, we need to print a nice header before firing up the shell, the return value should be the original error, etc. etc., but hopefully you get the point. :)

Note that to be useful the shell should be fired in the same fakemachine as the command that failed, otherwise developers won't be able to properly inspect what went wrong.